Opensearch 2.4.0 helm chart: Keystore generated from k8s secret doesn't allow updates

Hi All! Having issue with keystore changes. I configured Opensearch helm chart with keystore secret values, keystore was correctly updated during opensearch provisioning but now I want to update values I set during installation. I went into Opensearch shell and ran command:
echo -n XXXXXXXXXXXXXXXXX | bin/opensearch-keystore add --stdin --force s3.client.default.access_key and got error:
INFO: Response: Exception in thread “main” java.nio.file.FileAlreadyExistsException: /usr/share/opensearch/config/opensearch.keystore.tmp
at java.base/sun.nio.fs.UnixException.translateToIOException(
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(
at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(
at java.base/java.nio.file.Files.newOutputStream(
at org.opensearch.common.settings.AddStringKeyStoreCommand.executeCommand(
at org.opensearch.common.settings.BaseKeyStoreCommand.execute(
at org.opensearch.cli.EnvironmentAwareCommand.execute(
at org.opensearch.cli.Command.mainWithoutErrorHandling(
at org.opensearch.cli.MultiCommand.execute(
at org.opensearch.cli.Command.mainWithoutErrorHandling(
at org.opensearch.cli.Command.main(
at org.opensearch.common.settings.KeyStoreCli.main(
The same command running on Opensearch deployed without keystore settings through kubernetes secret will run successfully. Permissions on opensearch.keystore and opensearch.keystore.tmp allowing -rw- for opensearch user. Any clue about this issue?

@mmamaenko I don’t think this is related to the security plugin. The security plugin doesn’t manage the OpenSearch keystore.

Did you check if the reported file /usr/share/opensearch/config/opensearch.keystore.tmp exists in the filesystem?
I’ve reproduced your issue and once I removed opensearch.keystore.tmp file and I was able to add a new key:value to the opensearch.keystore using your command.

Actually this file is missing when opensearch is created. Only when I ran command:
[opensearch@master-0 ~]$ echo -n XXXXXXXXXXXXXXXXX | bin/opensearch-keystore add --stdin --force s3.client.default.access_key it gets created
This is error message:
Exception in thread “main” java.nio.file.FileSystemException: /usr/share/opensearch/config/opensearch.keystore.tmp → /usr/share/opensearch/config/opensearch.keystore: Device or resource busy

Permissions for files:
-rw-rw---- 1 opensearch opensearch 504 Jan 17 20:44 opensearch.keystore
-rw-rw-r-- 1 opensearch opensearch 501 Jan 17 20:48 opensearch.keystore.tmp

Changing permission for keystore to 777 didn’t help
Did you install opensearch with keystore mounted from k8s secret?

My secret looks like:

apiVersion: v1
kind: Secret
name: mysecret
type: Opaque
s3.client.default.access_key: XXXXXXXXXXXXXXXXXXXXXX

@mmamaenko I’ve used helm charts provided in OpenSearch documentation.

I understand that but did you set keystore to be mounted from k8s secret in chart values?

@mmamaenko I’ve used default settings and charts. I didn’t add any extra secrets in k8s.

The problems occurs only when set keystore.secretName. In this case secret will be mounted? and added to keystore file by init container