Describe the issue:
I upgraded my Opensearch Instance, deployed via Helm on a Kubernetes Cluster to Version 2.18.0. After Upgrading I get an Error in the Log of the Pods “No custom admin password found. Please provide a password via the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD.” which Prevent them from starting.
So I set the Mentioned environment Variable with the Effect that I get now an other Error “Exception updating the admin password : Unable to update the internal users file with the hashed password.”, with the Pods still not Running.
The only Opportunity ist to delete my Custom securityConfig, then Opensearch come up an get running, but this is not the thing I want because I added my LDAP via this Config.
I have also encountered this problem whilst attempting to update to 2.18.0, via version 2.12.0 of the Helm chart.
The existing installation is configured for oidc using the securityConfigSecret. Here is the relevant part of our values.yaml, including the new, required OPENSEARCH_INITIAL_ADMIN_PASSWORD env var:
The security-config-secret Secret only contains two config files: config.yml and tenants.yml. i.e. there is no internal-users.yml configured.
Here is the entirety of the opensearch logs:
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.15.0-89-generic amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.12.0
Detected OpenSearch Security Version: 2.12.0.0
Admin password set successfully.
Exception updating the admin password : Unable to update the internal users file with the hashed password.
It sounds as if the new admin security process is incompatible with the way security config is set up. There is no internal-users.yml under /usr/share/opensearch/config/opensearch-security, and this folder is read-only because it is a secret volume mount.
I was able to proceed with the upgrade by disabling securityConfig in the helm values:
securityConfig:
enabled: false
The opensearch nodes could then start. I had then expected that I would have to re-enable this config in order to log in via oidc, but interestingly the oidc authentication module appears to remain configured, and so I was able to login to opensearch-dashboards.
If I then reenable the securityConfig the previous error reoccurs.
I have the same issue. I tried upgrading via helm to latest chart. Version 2.18.0. The PODs get stuck in “Error” an “Crash loop backoff”. In the Logs of the node I can see:
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.10.0-28-amd64 amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.12.0
Detected OpenSearch Security Version: 2.12.0.0
No custom admin password found. Please provide a password via the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD.
The cluster uses only internal users and authentication. What is the right way to upgrade to 2.12.0 using helm and keep the internal user configuration as is in place? I did some research but could not find a solution by now. thank you. I downgraded to Version 2.17.0 of the chart and it worked right away.
@danielt@garyhodgson@mbah My OpenSearch cluster was created with Charts version 2.10.0 and OpenSearch version 2.8.0.
helm install opensearch .
I’ve used the latest Charts 2.18 and OpenSearch 2.12.0 to upgrade the running cluster.
helm upgrade opensearch .
The first OpenSearch pod got terminated and started failing with the error.
Detected OpenSearch Version: 2.12.0
Detected OpenSearch Security Version: 2.12.0.0
No custom admin password found. Please provide a password via the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD.
I’ve edited values.yml and change the section
extraEnvs: []
# - name: MY_ENVIRONMENT_VAR
# value: the_value_goes_here
# Chart version 2.18.0 and App Version OpenSearch 2.12.0 onwards a custom strong password needs to be provided in order to setup demo admin user.
# Cluster will not spin-up without this unless demo config install is disabled.
# - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
# value: <password>
to
extraEnvs:
# - name: MY_ENVIRONMENT_VAR
# value: the_value_goes_here
# Chart version 2.18.0 and App Version OpenSearch 2.12.0 onwards a custom strong password needs to be provided in order to setup demo admin user.
# Cluster will not spin-up without this unless demo config install is disabled.
- name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
value: Password123!
Next, I upgraded the charts again with enabled extraEnvs.
helm upgrade opensearch .
After the successful upgrade, the admin password hasn’t changed and the old one was still working.
This is expected as the security index has been already initiated with the older charts and the only way to change the admin password is with securityadmin.sh script.
Please double-check if you didn’t duplicate extraEnvs: in your values.yml.
If you have:
extraEnvs:
# - name: MY_ENVIRONMENT_VAR
# value: the_value_goes_here
# Chart version 2.18.0 and App Version OpenSearch 2.12.0 onwards a custom strong password needs to be provided in order to setup demo admin user.
# Cluster will not spin-up without this unless demo config install is disabled.
- name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
value: Password123!
extraEnvs: []
The helm upgrade command won’t produce any error:
pablo@kube-1:~/helm-charts_old/charts/opensearch$ helm upgrade opensearch .
Release "opensearch" has been upgraded. Happy Helming!
NAME: opensearch
LAST DEPLOYED: Tue Mar 5 15:51:21 2024
NAMESPACE: default
STATUS: deployed
REVISION: 5
TEST SUITE: None
NOTES:
Watch all cluster members come up.
The OpenSearch pods will keep failing with the below error as the second occurrence of the extraEnvs will clear the variable.
No custom admin password found. Please provide a password via the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD.
Exception updating the admin password : Unable to update the internal users file with the hashed password.
I tried to comment out all entries in internal_users.yml keeping it to just:
internal_users.yml: |-
---
# This is the internal user database
# The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh
_meta:
type: "internalusers"
config_version: 2
Thank you. I added the extraEnvs to my vaules.yaml and retried deployment. Now I get a different error message:
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.10.0-28-amd64 amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.12.0
Detected OpenSearch Security Version: 2.12.0.0
Admin password set successfully.
Exception updating the admin password : /usr/share/opensearch/config/opensearch-security/internal_users.yml: Device or resource busy
@danielt@idemery I got the below error when I defined internal users in the values.yml file.
Exception updating the admin password : /usr/share/opensearch/config/opensearch-security/internal_users.yml: Device or resource busy
In my case, I’ve used secret instead of pasting the content of the internal_users.yml into the values.yml. I also got the same error in the docker deployment when I mapped internal_users.yml file to the OpenSearch docker container.
The deployment was completed successfully once I removed the mapping in the docker and values.yml to internal_users.yml.
I would consider this as a bug. The 2.12.0 initial deployment prevents the usage of the custom internal_users.yml file. It is possible to update internal_users after the deployment with securityadmin.sh. However, this shouldn’t happen.
Please create a bug report in the OpenSearch security GitHub and share the GitHub issue link here.
@idemery If this is a fresh installation, could you try removing the PVCs related to OpenSearch pods and try deployment with extraEnvs and without internal_users?
@pablo I got it working.
1- As you mentioned I set persistence.enabled to false and deleted all PVCs.
2- I configured all certificates as per documentation for all nodes, admin, and clients.
3- Uncommented the internal users back.
4- Tested everything with opensearch dashboards and get it working
5- Uninstalled everything, enabled persistence back, installed both charts, everything works.
When i applied the extraEnvs with name and value in helm values.yaml and installed, did not take effect. I will have to troubleshoot this one
As a work around, installed Opensearch and modified opensearch statefulset and added directly. Post update of statefulset, all pods started with running state