Opensearch 2.4.0 helm chart: Keystore generated from k8s secret doesn't allow updates

Hi All! Having issue with keystore changes. I configured Opensearch helm chart with keystore secret values, keystore was correctly updated during opensearch provisioning but now I want to update values I set during installation. I went into Opensearch shell and ran command:
echo -n XXXXXXXXXXXXXXXXX | bin/opensearch-keystore add --stdin --force s3.client.default.access_key and got error:
INFO: Response: Exception in thread “main” java.nio.file.FileAlreadyExistsException: /usr/share/opensearch/config/opensearch.keystore.tmp
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:484)
at java.base/java.nio.file.Files.newOutputStream(Files.java:228)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:392)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:385)
at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:220)
at org.apache.lucene.backward_codecs.store.EndiannessReverserUtil.createOutput(EndiannessReverserUtil.java:54)
at org.opensearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:528)
at org.opensearch.common.settings.AddStringKeyStoreCommand.executeCommand(AddStringKeyStoreCommand.java:124)
at org.opensearch.common.settings.BaseKeyStoreCommand.execute(BaseKeyStoreCommand.java:88)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:56)
The same command running on Opensearch deployed without keystore settings through kubernetes secret will run successfully. Permissions on opensearch.keystore and opensearch.keystore.tmp allowing -rw- for opensearch user. Any clue about this issue?

@mmamaenko I don’t think this is related to the security plugin. The security plugin doesn’t manage the OpenSearch keystore.

Did you check if the reported file /usr/share/opensearch/config/opensearch.keystore.tmp exists in the filesystem?
I’ve reproduced your issue and once I removed opensearch.keystore.tmp file and I was able to add a new key:value to the opensearch.keystore using your command.

Actually this file is missing when opensearch is created. Only when I ran command:
[opensearch@master-0 ~]$ echo -n XXXXXXXXXXXXXXXXX | bin/opensearch-keystore add --stdin --force s3.client.default.access_key it gets created
This is error message:
Exception in thread “main” java.nio.file.FileSystemException: /usr/share/opensearch/config/opensearch.keystore.tmp → /usr/share/opensearch/config/opensearch.keystore: Device or resource busy

Permissions for files:
-rw-rw---- 1 opensearch opensearch 504 Jan 17 20:44 opensearch.keystore
-rw-rw-r-- 1 opensearch opensearch 501 Jan 17 20:48 opensearch.keystore.tmp

Changing permission for keystore to 777 didn’t help
Did you install opensearch with keystore mounted from k8s secret?

My secret looks like:

apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
s3.client.default.access_key: XXXXXXXXXXXXXXXXXXXXXX
s3.client.default.secret_key: YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY

@mmamaenko I’ve used helm charts provided in OpenSearch documentation.

I understand that but did you set keystore to be mounted from k8s secret in chart values?

@mmamaenko I’ve used default settings and charts. I didn’t add any extra secrets in k8s.

The problems occurs only when set keystore.secretName. In this case secret will be mounted? and added to keystore file by init container