Describe the issue:
I upgraded my Opensearch Instance, deployed via Helm on a Kubernetes Cluster to Version 2.18.0. After Upgrading I get an Error in the Log of the Pods “No custom admin password found. Please provide a password via the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD.” which Prevent them from starting.
So I set the Mentioned environment Variable with the Effect that I get now an other Error “Exception updating the admin password : Unable to update the internal users file with the hashed password.”, with the Pods still not Running.
The only Opportunity ist to delete my Custom securityConfig, then Opensearch come up an get running, but this is not the thing I want because I added my LDAP via this Config.
I have also encountered this problem whilst attempting to update to 2.18.0, via version 2.12.0 of the Helm chart.
The existing installation is configured for oidc using the securityConfigSecret. Here is the relevant part of our values.yaml, including the new, required OPENSEARCH_INITIAL_ADMIN_PASSWORD env var:
The security-config-secret Secret only contains two config files: config.yml and tenants.yml. i.e. there is no internal-users.yml configured.
Here is the entirety of the opensearch logs:
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.15.0-89-generic amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.12.0
Detected OpenSearch Security Version: 2.12.0.0
Admin password set successfully.
Exception updating the admin password : Unable to update the internal users file with the hashed password.
It sounds as if the new admin security process is incompatible with the way security config is set up. There is no internal-users.yml under /usr/share/opensearch/config/opensearch-security, and this folder is read-only because it is a secret volume mount.
I was able to proceed with the upgrade by disabling securityConfig in the helm values:
securityConfig:
enabled: false
The opensearch nodes could then start. I had then expected that I would have to re-enable this config in order to log in via oidc, but interestingly the oidc authentication module appears to remain configured, and so I was able to login to opensearch-dashboards.
If I then reenable the securityConfig the previous error reoccurs.
I have the same issue. I tried upgrading via helm to latest chart. Version 2.18.0. The PODs get stuck in “Error” an “Crash loop backoff”. In the Logs of the node I can see:
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.10.0-28-amd64 amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.12.0
Detected OpenSearch Security Version: 2.12.0.0
No custom admin password found. Please provide a password via the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD.
The cluster uses only internal users and authentication. What is the right way to upgrade to 2.12.0 using helm and keep the internal user configuration as is in place? I did some research but could not find a solution by now. thank you. I downgraded to Version 2.17.0 of the chart and it worked right away.
@danielt@garyhodgson@mbah My OpenSearch cluster was created with Charts version 2.10.0 and OpenSearch version 2.8.0.
helm install opensearch .
I’ve used the latest Charts 2.18 and OpenSearch 2.12.0 to upgrade the running cluster.
helm upgrade opensearch .
The first OpenSearch pod got terminated and started failing with the error.
Detected OpenSearch Version: 2.12.0
Detected OpenSearch Security Version: 2.12.0.0
No custom admin password found. Please provide a password via the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD.
I’ve edited values.yml and change the section
extraEnvs: []
# - name: MY_ENVIRONMENT_VAR
# value: the_value_goes_here
# Chart version 2.18.0 and App Version OpenSearch 2.12.0 onwards a custom strong password needs to be provided in order to setup demo admin user.
# Cluster will not spin-up without this unless demo config install is disabled.
# - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
# value: <password>
to
extraEnvs:
# - name: MY_ENVIRONMENT_VAR
# value: the_value_goes_here
# Chart version 2.18.0 and App Version OpenSearch 2.12.0 onwards a custom strong password needs to be provided in order to setup demo admin user.
# Cluster will not spin-up without this unless demo config install is disabled.
- name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
value: Password123!
Next, I upgraded the charts again with enabled extraEnvs.
helm upgrade opensearch .
After the successful upgrade, the admin password hasn’t changed and the old one was still working.
This is expected as the security index has been already initiated with the older charts and the only way to change the admin password is with securityadmin.sh script.
Please double-check if you didn’t duplicate extraEnvs: in your values.yml.
If you have:
extraEnvs:
# - name: MY_ENVIRONMENT_VAR
# value: the_value_goes_here
# Chart version 2.18.0 and App Version OpenSearch 2.12.0 onwards a custom strong password needs to be provided in order to setup demo admin user.
# Cluster will not spin-up without this unless demo config install is disabled.
- name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
value: Password123!
extraEnvs: []
The helm upgrade command won’t produce any error:
pablo@kube-1:~/helm-charts_old/charts/opensearch$ helm upgrade opensearch .
Release "opensearch" has been upgraded. Happy Helming!
NAME: opensearch
LAST DEPLOYED: Tue Mar 5 15:51:21 2024
NAMESPACE: default
STATUS: deployed
REVISION: 5
TEST SUITE: None
NOTES:
Watch all cluster members come up.
The OpenSearch pods will keep failing with the below error as the second occurrence of the extraEnvs will clear the variable.
No custom admin password found. Please provide a password via the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD.
Exception updating the admin password : Unable to update the internal users file with the hashed password.
I tried to comment out all entries in internal_users.yml keeping it to just:
internal_users.yml: |-
---
# This is the internal user database
# The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh
_meta:
type: "internalusers"
config_version: 2
Thank you. I added the extraEnvs to my vaules.yaml and retried deployment. Now I get a different error message:
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.10.0-28-amd64 amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.12.0
Detected OpenSearch Security Version: 2.12.0.0
Admin password set successfully.
Exception updating the admin password : /usr/share/opensearch/config/opensearch-security/internal_users.yml: Device or resource busy
@danielt@idemery I got the below error when I defined internal users in the values.yml file.
Exception updating the admin password : /usr/share/opensearch/config/opensearch-security/internal_users.yml: Device or resource busy
In my case, I’ve used secret instead of pasting the content of the internal_users.yml into the values.yml. I also got the same error in the docker deployment when I mapped internal_users.yml file to the OpenSearch docker container.
The deployment was completed successfully once I removed the mapping in the docker and values.yml to internal_users.yml.
I would consider this as a bug. The 2.12.0 initial deployment prevents the usage of the custom internal_users.yml file. It is possible to update internal_users after the deployment with securityadmin.sh. However, this shouldn’t happen.
Please create a bug report in the OpenSearch security GitHub and share the GitHub issue link here.
@idemery If this is a fresh installation, could you try removing the PVCs related to OpenSearch pods and try deployment with extraEnvs and without internal_users?
@pablo I got it working.
1- As you mentioned I set persistence.enabled to false and deleted all PVCs.
2- I configured all certificates as per documentation for all nodes, admin, and clients.
3- Uncommented the internal users back.
4- Tested everything with opensearch dashboards and get it working
5- Uninstalled everything, enabled persistence back, installed both charts, everything works.
When i applied the extraEnvs with name and value in helm values.yaml and installed, did not take effect. I will have to troubleshoot this one
As a work around, installed Opensearch and modified opensearch statefulset and added directly. Post update of statefulset, all pods started with running state
Thanks.
Adding the var’s in to statefulset.yaml helped and container did not shout for the OPENSEARCH_INITIAL_ADMIN_PASSWORD but then I ran in to apache error =
amey@in-lp-22:~/git/cloud-argo-deployment/gcs-dev/logging$ k logs gmsp-es-logging-0
Defaulted container "elasticsearch" out of: elasticsearch, configure-sysctl (init)
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.15.0-102-generic amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.13.0
Detected OpenSearch Security Version: 2.13.0.0
Admin password set successfully.
### Success
### Execute this script now on all your nodes and then start all nodes
### OpenSearch Security will be automatically initialized.
### If you like to change the runtime configuration
### change the files in ../../../config/opensearch-security and execute:
sudo "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/config/opensearch-security" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv
### or run ./securityadmin_demo.sh
### To use the Security Plugin ConfigurationGUI
### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/<your-custom-admin-password>.
### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
Exception in thread "main" org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual header=1633764165 vs expected header=1071082519 (resource=Endianness reverser Checksum Index Input wrapper)
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:187)
at org.opensearch.common.settings.KeyStoreWrapper.load(KeyStoreWrapper.java:251)
at org.opensearch.common.settings.KeyStoreWrapper.load(KeyStoreWrapper.java:232)
at org.opensearch.common.settings.HasPasswordKeyStoreCommand.execute(HasPasswordKeyStoreCommand.java:61)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:56)
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.13.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
Exception in thread "main" org.opensearch.bootstrap.BootstrapException: org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual header=1633764165 vs expected header=1071082519 (resource=Endianness reverser Checksum Index Input wrapper)
Likely root cause: org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual header=1633764165 vs expected header=1071082519 (resource=Endianness reverser Checksum Index Input wrapper)
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:187)
at org.opensearch.common.settings.KeyStoreWrapper.load(KeyStoreWrapper.java:251)
at org.opensearch.common.settings.KeyStoreWrapper.load(KeyStoreWrapper.java:232)
at org.opensearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:257)
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:368)
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)
Based on the errror, it appears to be the initial admin password is not configured properly. Share your statefulset, configmap yaml files, will try to find if any indent/format issue (mask the password value)
Quick note seems pretty picky on the password, escaped chars didn’t work, had to set in /etc/environment. export wasn’t working even though printing out all env clearly showed it existed. Hope that saves some trouble…