Hi All! Having issue with keystore changes. I configured Opensearch helm chart with keystore secret values, keystore was correctly updated during opensearch provisioning but now I want to update values I set during installation. I went into Opensearch shell and ran command:
echo -n XXXXXXXXXXXXXXXXX | bin/opensearch-keystore add --stdin --force s3.client.default.access_key and got error:
INFO: Response: Exception in thread “main” java.nio.file.FileAlreadyExistsException: /usr/share/opensearch/config/opensearch.keystore.tmp
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:484)
at java.base/java.nio.file.Files.newOutputStream(Files.java:228)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:392)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:385)
at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:220)
at org.apache.lucene.backward_codecs.store.EndiannessReverserUtil.createOutput(EndiannessReverserUtil.java:54)
at org.opensearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:528)
at org.opensearch.common.settings.AddStringKeyStoreCommand.executeCommand(AddStringKeyStoreCommand.java:124)
at org.opensearch.common.settings.BaseKeyStoreCommand.execute(BaseKeyStoreCommand.java:88)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:56)
The same command running on Opensearch deployed without keystore settings through kubernetes secret will run successfully. Permissions on opensearch.keystore and opensearch.keystore.tmp allowing -rw- for opensearch user. Any clue about this issue?
@mmamaenko I don’t think this is related to the security plugin. The security plugin doesn’t manage the OpenSearch keystore.
Did you check if the reported file /usr/share/opensearch/config/opensearch.keystore.tmp exists in the filesystem?
I’ve reproduced your issue and once I removed opensearch.keystore.tmp file and I was able to add a new key:value to the opensearch.keystore using your command.
Actually this file is missing when opensearch is created. Only when I ran command:
[opensearch@master-0 ~]$ echo -n XXXXXXXXXXXXXXXXX | bin/opensearch-keystore add --stdin --force s3.client.default.access_key it gets created
This is error message:
Exception in thread “main” java.nio.file.FileSystemException: /usr/share/opensearch/config/opensearch.keystore.tmp → /usr/share/opensearch/config/opensearch.keystore: Device or resource busy
Permissions for files:
-rw-rw---- 1 opensearch opensearch 504 Jan 17 20:44 opensearch.keystore
-rw-rw-r-- 1 opensearch opensearch 501 Jan 17 20:48 opensearch.keystore.tmp
Changing permission for keystore to 777 didn’t help
Did you install opensearch with keystore mounted from k8s secret?
My secret looks like:
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
s3.client.default.access_key: XXXXXXXXXXXXXXXXXXXXXX
s3.client.default.secret_key: YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
I understand that but did you set keystore to be mounted from k8s secret in chart values?
The problems occurs only when set keystore.secretName. In this case secret will be mounted? and added to keystore file by init container