Opensearch cluster keep restart after setting securityconfig

OpenSearch
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
(helm-charts | ☸ A community repository for Helm Charts of OpenSearch Project.)
I’m using official helm chart to install both the cluster and dashboard.
opensearch cluster ver: 2.16
dashboard ver: 2.16

Describe the issue:
After I have configured the securityconfig, it did not take effect. The cluster is still using the demo config, and the cluster pod keeps restarting infinitely during the init phase (single-node). The log is as follows and keep repeat it:

Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.10.220-209.869.amzn2.x86_64 amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.16.0
Detected OpenSearch Security Version: 2.16.0.0

Configuration:
config.yaml:

securityConfig:
  enabled: true
  path: "/usr/share/opensearch/config/opensearch-security"
  actionGroupsSecret:
  configSecret:
  internalUsersSecret:
  rolesSecret:
  rolesMappingSecret:
  tenantsSecret:
  config:
    securityConfigSecret: ""
    dataComplete: true
    data: 
      roles_mapping.yml: |-
        _meta:
          type: "rolesmapping"
          config_version: 2
        all_access:
          reserved: true
          hidden: false
          backend_roles:
          - "admin"
          - "all_access"
          users:
          - "email@xxx.com"
          description: "Maps an openid user to all_access"
      config.yml: |-
        _meta:
          type: "config"
          config_version: 2
        config:
          dynamic:
            http:
              anonymous_auth_enabled: false
            authc:
              basic_internal_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: 0
                http_authenticator:
                  type: basic
                  challenge: true
                authentication_backend:
                  type: intern
              openid_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: 1
                http_authenticator:
                  type: openid
                  challenge: false
                  config:
                    subject_key: email
                    roles_key: roles
                    openid_connect_url: https://accounts.google.com/.well-known/openid-configuration
                    jwt_header: Authorization
                    openid_connect_idp:
                        enable_ssl: false
                  authentication_backend:
                    type: noop

dashboard.yml:

opensearch_dashboards.yml: |
    opensearch.hosts: ["https://my-opensearch-cluster-domain:9200"]
    opensearch.ssl.verificationMode: none
    opensearch.username: admin
    opensearch.password: xxxxx
    opensearch.requestHeadersAllowlist: ["securitytenant","Authorization"]
    opensearch_security.multitenancy.enabled: false
    opensearch_security.readonly_mode.roles: ["readall"]
    opensearch_security.cookie.secure: false
    server.host: "0.0.0.0"
    opensearch_security.auth.multiple_auth_enabled: true 
    opensearch_security.auth.type: ["openid", "basicauth"]
    opensearch_security.openid.connect_url: "https://accounts.google.com/.well-known/openid-configuration" 
    opensearch_security.openid.client_id: "xxxxx
    opensearch_security.openid.client_secret: "xxxxx
    opensearch_security.openid.verify_hostnames: false
    opensearch_security.openid.refresh_tokens: false
    opensearch_security.openid.base_redirect_url: "https://my-opensearch-dashboard-domain"

**Relevant Logs or Screenshots**:
2

Hi @harlan262612 ,

Have you added OPENSEARCH_INITIAL_ADMIN_PASSWORD to your values.yaml ?

3

If not, please have a have a look at the following example:

4

I’m not familiar with Kubernetes installation process, but is it running plugins/opensearch-security/tools/securityadmin.sh after updating config.yml?

5

hi there, recently i setup a new opensearch cluster with newer version and i get the same issue u had. not sure if my solution works for u.

the installation stuck right after Detected OpenSearch Security Version: xxx and the pods keep restarting. no further exception is being thrown.

i managed to figure out the root cause of this.
it’s due to missing securityConfig.config.data.
u may refer to Modifying the YAML files - OpenSearch Documentation

for me, i have to specifically configure all the *.yml files in securityConfig.config.data.
once the *.yml files are provided, u will notice the installation no longer stuck on the logs Detected OpenSearch Security Version: xxx and it continues to setup.

btw, if OPENSEARCH_INITIAL_ADMIN_PASSWORD is not provided, the logs will explicitly mention it and the installation will quit.
at the beginning i was confused with the potential issue everyone said that OPENSEARCH_INITIAL_ADMIN_PASSWORD is not provided. but end up i found it’s different error.