Custom admin hash not working as expected

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch → 2.16.0
Dashboard → 2.16.0

Describe the issue:
I running opensearch with a custom admin password (so far hardcoded in a bash deployment script) that password was converted into a hash using the following python command (python -c ‘import bcrypt; print(bcrypt.hashpw(“password_string”.encode(“utf-8”), bcrypt.gensalt(12, prefix=b"2a")).decode(“utf-8”))’) from the github documentation (opensearch-k8s-operator/docs/userguide/main.md at main · opensearch-project/opensearch-k8s-operator · GitHub) and added into the internal_users_custom.yml, and its been working fine.

For obvious reasons we cannot continue using a hardcoded password therefore I try to do it using variables within the scripts and use sed to simply replace the hash in the internal_users_custom.yml file as we do for our kubernetes secret but for some reason I cannot understand its not working, the hash password is being converted as hash and then replaced in the internal_users_custom.yml but when the pods start to being deployed I get a “connection refused” message from the nods and master pods.

Configuration:
The lines below are those I’m having issues with. The created hash its being placed in the file but it seems it wasn’t correctly created, something is off. I saved all the python string in the hash variable and the variable is being replaced correctly (second line) so I guess the issue is during the creation or the sed replaced.

PS. If a do below process manually, I meant, pasting the password string (not the variable) in the command and then manually copy the hash into the file, it works perfectly fine, and opensearch is able to deploy completely.

adminpwd=$(date +%s | sha256sum | base64 | head -c 16)
#adhash="python3 -c 'import bcrypt; print(bcrypt.hashpw("${adminpwd}".encode("utf-8"), bcrypt.gensalt(12, prefix=b"2a")).decode("utf-8"))'"
adhash="$(python3 -c 'import bcrypt; print(bcrypt.hashpw("${adminpwd}".encode("utf-8"), bcrypt.gensalt(12, prefix=b"2a")).decode("utf-8"))')"
sed -i "s/XXXXX/${adhash}/g" /path-to-file/internal_users_custom.yml

Relevant Logs or Screenshots:

2

@GTGabaaron Could there be any previous volumes that have persisted data from before? The yaml files only seed the security index in a fresh cluster, if you have a cluster that has a .opendistro_security index then you need to use the securityadmin script to apply changes: Applying changes to configuration files - OpenSearch Documentation

3

Hi @cwperks I’m making sure to previously delete all PVC every time I deploy opensearch, but still the issue is the same.

4

Do you see any errors in the logs on node bootstrap? Just to make sure, have you verified that the output of the sed command is identical to copying-and-pasting the value of adHash into the internal_users_custom.yml? If copying-and-pasting works, then I’m not sure why the sed command with the same output would be failing.

5

Hi @cwperks yes, the output of the python command and the sed is the same that is being pasted inside the internal_users_custom.yaml file. I’ve compared both hash and they matched. I’ll re-deploy opensearch later today and get the logs from the bootstrap.

6

Hi @GTGabaaron,

If you have access to your admin certs you can run the following to check what has is applied currently on your cluster (might help with troubleshooting):

curl --insecure --cert <path/to/admin.pem> --key <path/to/admin-key.pem> -XGET https://<OpenSearch_node_FQDN_or_IP>:9200/.opendistro_security/_doc/internalusers?pretty

This will return JWT which you can decode and compare hashes.

You can always use plugins/opensearch-security/tools/hash.sh -p <new-password> to hash passwords too.

more here: Modifying the YAML files - OpenSearch Documentation

Best,
mj

7

Hi @cwperks this are the full logs of the bootstrap pod. I see a lot of errors about running the security admin but nothing in particular about the hash string, just something about the hash ring.

image
image1477×323 23.2 KB

image
image1619×339 8.92 KB

8

By any chance, do you have plugins.security.allow_default_init_securityindex set to false in opensearch.yml?

Are there any errors further up related to parsing internal_users_custom.yml?

If plugins.security.allow_default_init_securityindex is set to false, then you need to use Applying changes to configuration files - OpenSearch Documentation to source the security index.

9

I found the issue, it was actually related to the python command as I suspected. The issues was that python was not able to access the shell variables directly, I have to change the command to include “sys.argv[1]” in the part of the variable and the move the variable at the end of the command, that way it was taking the actual value of the variables instead of the name of the variable as value.

Correct form is as follow → 


adhash=$(python3 -c "import sys, bcrypt; print(bcrypt.hashpw(sys.argv[1].encode('utf-8'), bcrypt.gensalt(12, prefix=b'2a')).decode('utf-8'))" "$adminpwd")
2 Likes