When setting config.yml (through helm values) in securityContext the opensearch pod won't start

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch helm chart: 2.31.0
opensearch: 2.19.0

Describe the issue:
If I don’t touch the security config the pod starts just fine and I can connect to opensearch with the admin user.
But as soon as i change the security config the pods just restarts after a few seconds. This is the output from the log:

Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.15.0-130-generic amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.19.0
Detected OpenSearch Security Version: 2.19.0.0

And it just keeps looping that text every 30-60sec.
I can see that the opensearch-cluster-master-securityconfig secret is populated with the stuff I put inside the securityContext.config.data.

Why does my pod not start?
I have tried to also add this under extraEnvs:

- name: "JAVA_OPTS"
  value: "-Dlog4j2.debug=true"

But it doesn’t show me any more logs.

Configuration:
My goal is to get ldap working but even if I take the default content of the security files config.yml, roles.yml and rolesmapping.yml it still does not work.

The default configs are quite long but here they are:

      helm:
        values: |
          singleNode: true
          protocol: https
          extraEnvs:
            - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
              value: Jagheterviktor123!
          opensearch.yml: |
            cluster.name: opensearch-cluster

            # Bind to all interfaces because we don't know what IP address Docker will assign to us.
            network.host: 0.0.0.0

            # Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
            # Implicitly done if ".singleNode" is set to "true".
            discovery.type: single-node
          securityConfig:
            enabled: true
            config:
              data: 
                config.yml: |-
                  _meta:
                    type: "config"
                    config_version: 2

                  config:
                    dynamic:
                      # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
                      # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
                      # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
                      #filtered_alias_mode: warn
                      #do_not_fail_on_forbidden: false
                      #kibana:
                      # Kibana multitenancy
                      #multitenancy_enabled: true
                      #private_tenant_enabled: true
                      #default_tenant: ""
                      #server_username: kibanaserver
                      #index: '.kibana'
                      http:
                        anonymous_auth_enabled: false
                        xff:
                          enabled: false
                          internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
                          #internalProxies: '.*' # trust all internal proxies, regex pattern
                          #remoteIpHeader:  'x-forwarded-for'
                          ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
                          ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
                          ###### and here https://tools.ietf.org/html/rfc7239
                          ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
                      authc:
                        kerberos_auth_domain:
                          http_enabled: false
                          transport_enabled: false
                          order: 6
                          http_authenticator:
                            type: kerberos
                            challenge: true
                            config:
                              # If true a lot of kerberos/security related debugging output will be logged to standard out
                              krb_debug: false
                              # If true then the realm will be stripped from the user name
                              strip_realm_from_principal: true
                          authentication_backend:
                            type: noop
                        basic_internal_auth_domain:
                          description: "Authenticate via HTTP Basic against internal users database"
                          http_enabled: true
                          transport_enabled: true
                          order: 4
                          http_authenticator:
                            type: basic
                            challenge: true
                          authentication_backend:
                            type: intern
                        proxy_auth_domain:
                          description: "Authenticate via proxy"
                          http_enabled: false
                          transport_enabled: false
                          order: 3
                          http_authenticator:
                            type: proxy
                            challenge: false
                            config:
                              user_header: "x-proxy-user"
                              roles_header: "x-proxy-roles"
                          authentication_backend:
                            type: noop
                        jwt_auth_domain:
                          description: "Authenticate via Json Web Token"
                          http_enabled: false
                          transport_enabled: false
                          order: 0
                          http_authenticator:
                            type: jwt
                            challenge: false
                            config:
                              signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
                              jwt_header: "Authorization"
                              jwt_url_parameter: null
                              jwt_clock_skew_tolerance_seconds: 30
                              roles_key: null
                              subject_key: null
                          authentication_backend:
                            type: noop
                        clientcert_auth_domain:
                          description: "Authenticate via SSL client certificates"
                          http_enabled: false
                          transport_enabled: false
                          order: 2
                          http_authenticator:
                            type: clientcert
                            config:
                              username_attribute: cn #optional, if omitted DN becomes username
                            challenge: false
                          authentication_backend:
                            type: noop
                        ldap:
                          description: "Authenticate via LDAP or Active Directory"
                          http_enabled: false
                          transport_enabled: false
                          order: 5
                          http_authenticator:
                            type: basic
                            challenge: false
                          authentication_backend:
                            # LDAP authentication backend (authenticate users against a LDAP or Active Directory)
                            type: ldap
                            config:
                              # enable ldaps
                              enable_ssl: false
                              # enable start tls, enable_ssl should be false
                              enable_start_tls: false
                              # send client certificate
                              enable_ssl_client_auth: false
                              # verify ldap hostname
                              verify_hostnames: true
                              hosts:
                              - localhost:8389
                              bind_dn: null
                              password: null
                              userbase: 'ou=people,dc=example,dc=com'
                              # Filter to search for users (currently in the whole subtree beneath userbase)
                              # {0} is substituted with the username
                              usersearch: '(sAMAccountName={0})'
                              # Use this attribute from the user as username (if not set then DN is used)
                              username_attribute: null
                      authz:
                        roles_from_myldap:
                          description: "Authorize via LDAP or Active Directory"
                          http_enabled: false
                          transport_enabled: false
                          authorization_backend:
                            # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
                            type: ldap
                            config:
                              # enable ldaps
                              enable_ssl: false
                              # enable start tls, enable_ssl should be false
                              enable_start_tls: false
                              # send client certificate
                              enable_ssl_client_auth: false
                              # verify ldap hostname
                              verify_hostnames: true
                              hosts:
                              - localhost:8389
                              bind_dn: null
                              password: null
                              rolebase: 'ou=groups,dc=example,dc=com'
                              # Filter to search for roles (currently in the whole subtree beneath rolebase)
                              # {0} is substituted with the DN of the user
                              # {1} is substituted with the username
                              # {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
                              rolesearch: '(member={0})'
                              # Specify the name of the attribute which value should be substituted with {2} above
                              userroleattribute: null
                              # Roles as an attribute of the user entry
                              userrolename: disabled
                              #userrolename: memberOf
                              # The attribute in a role entry containing the name of that role, Default is "name".
                              # Can also be "dn" to use the full DN as rolename.
                              rolename: cn
                              # Resolve nested roles transitive (roles which are members of other roles and so on ...)
                              resolve_nested_roles: true
                              userbase: 'ou=people,dc=example,dc=com'
                              # Filter to search for users (currently in the whole subtree beneath userbase)
                              # {0} is substituted with the username
                              usersearch: '(uid={0})'
                              # Skip users matching a user name, a wildcard or a regex pattern
                              #skip_users:
                              #  - 'cn=Michael Jackson,ou*people,o=TEST'
                              #  - '/\S*/'
                        roles_from_another_ldap:
                          description: "Authorize via another Active Directory"
                          http_enabled: false
                          transport_enabled: false
                          authorization_backend:
                            type: ldap
                            #config goes here ...
                    #    auth_failure_listeners:
                    #      ip_rate_limiting:
                    #        type: ip
                    #        allowed_tries: 10
                    #        time_window_seconds: 3600
                    #        block_expiry_seconds: 600
                    #        max_blocked_clients: 100000
                    #        max_tracked_clients: 100000
                    #      internal_authentication_backend_limiting:
                    #        type: username
                    #        authentication_backend: intern
                    #        allowed_tries: 10
                    #        time_window_seconds: 3600
                    #        block_expiry_seconds: 600
                    #        max_blocked_clients: 100000
                    #        max_tracked_clients: 100000
                roles.yml: |-
                  _meta:
                    type: "roles"
                    config_version: 2

                  # Restrict users so they can only view visualization and dashboard on OpenSearchDashboards
                  kibana_read_only:
                    reserved: true

                  # The security REST API access role is used to assign specific users access to change the security settings through the REST API.
                  security_rest_api_access:
                    reserved: true

                  security_rest_api_full_access:
                    reserved: true
                    cluster_permissions:
                      - 'restapi:admin/actiongroups'
                      - 'restapi:admin/allowlist'
                      - 'restapi:admin/config/update'
                      - 'restapi:admin/internalusers'
                      - 'restapi:admin/nodesdn'
                      - 'restapi:admin/roles'
                      - 'restapi:admin/rolesmapping'
                      - 'restapi:admin/ssl/certs/info'
                      - 'restapi:admin/ssl/certs/reload'
                      - 'restapi:admin/tenants'

                  # Allows users to view monitors, destinations and alerts
                  alerting_read_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/opendistro/alerting/alerts/get'
                      - 'cluster:admin/opendistro/alerting/destination/get'
                      - 'cluster:admin/opendistro/alerting/monitor/get'
                      - 'cluster:admin/opendistro/alerting/monitor/search'
                      - 'cluster:admin/opensearch/alerting/comments/search'
                      - 'cluster:admin/opensearch/alerting/findings/get'
                      - 'cluster:admin/opensearch/alerting/remote/indexes/get'
                      - 'cluster:admin/opensearch/alerting/workflow/get'
                      - 'cluster:admin/opensearch/alerting/workflow_alerts/get'

                  # Allows users to view and acknowledge alerts
                  alerting_ack_alerts:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/opendistro/alerting/alerts/*'
                      - 'cluster:admin/opendistro/alerting/chained_alerts/*'
                      - 'cluster:admin/opendistro/alerting/workflow_alerts/*'
                      - 'cluster:admin/opensearch/alerting/comments/*'

                  # Allows users to use all alerting functionality
                  alerting_full_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/opendistro/alerting/*'
                      - 'cluster:admin/opensearch/alerting/*'
                      - 'cluster:admin/opensearch/notifications/feature/publish'
                      - 'cluster_monitor'
                    index_permissions:
                      - index_patterns:
                          - '*'
                        allowed_actions:
                          - 'indices:admin/aliases/get'
                          - 'indices:admin/mappings/get'
                          - 'indices_monitor'

                  # Allow users to read Anomaly Detection detectors and results
                  anomaly_read_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/opendistro/ad/detector/info'
                      - 'cluster:admin/opendistro/ad/detector/search'
                      - 'cluster:admin/opendistro/ad/detector/validate'
                      - 'cluster:admin/opendistro/ad/detectors/get'
                      - 'cluster:admin/opendistro/ad/result/search'
                      - 'cluster:admin/opendistro/ad/result/topAnomalies'
                      - 'cluster:admin/opendistro/ad/tasks/search'

                  # Allows users to use all Anomaly Detection functionality
                  anomaly_full_access:
                    reserved: true
                    cluster_permissions:
                      - "cluster:admin/ingest/pipeline/delete"
                      - "cluster:admin/ingest/pipeline/put"
                      - 'cluster:admin/opendistro/ad/*'
                      - 'cluster_monitor'
                    index_permissions:
                      - index_patterns:
                          - '*'
                        allowed_actions:
                          - 'indices:admin/aliases/get'
                          - 'indices:admin/mappings/fields/get'
                          - 'indices:admin/mappings/fields/get*'
                          - 'indices:admin/mappings/get'
                          - 'indices:admin/resolve/index'
                          - 'indices:admin/setting/put'
                          - 'indices:data/read/field_caps*'
                          - 'indices:data/read/search'
                          - 'indices_monitor'

                  # Allow users to execute read only k-NN actions
                  knn_read_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/knn_get_model_action'
                      - 'cluster:admin/knn_search_model_action'
                      - 'cluster:admin/knn_stats_action'

                  # Allow users to use all k-NN functionality
                  knn_full_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/knn_delete_model_action'
                      - 'cluster:admin/knn_get_model_action'
                      - 'cluster:admin/knn_remove_model_from_cache_action'
                      - 'cluster:admin/knn_search_model_action'
                      - 'cluster:admin/knn_stats_action'
                      - 'cluster:admin/knn_training_job_route_decision_info_action'
                      - 'cluster:admin/knn_training_job_router_action'
                      - 'cluster:admin/knn_training_model_action'
                      - 'cluster:admin/knn_update_model_graveyard_action'
                      - 'cluster:admin/knn_warmup_action'

                  # Allow users to execute read only ip2geo datasource action
                  ip2geo_datasource_read_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/geospatial/datasource/get'

                  # Allow users to use all ip2geo datasource action
                  ip2geo_datasource_full_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/geospatial/datasource/*'

                  # Allows users to read Notebooks
                  notebooks_read_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/opendistro/notebooks/get'
                      - 'cluster:admin/opendistro/notebooks/list'

                  # Allows users to all Notebooks functionality
                  notebooks_full_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/opendistro/notebooks/create'
                      - 'cluster:admin/opendistro/notebooks/delete'
                      - 'cluster:admin/opendistro/notebooks/get'
                      - 'cluster:admin/opendistro/notebooks/list'
                      - 'cluster:admin/opendistro/notebooks/update'

                  # Allows users to read observability objects
                  observability_read_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/opensearch/observability/get'

                  # Allows users to all Observability functionality
                  observability_full_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/opensearch/observability/create'
                      - 'cluster:admin/opensearch/observability/delete'
                      - 'cluster:admin/opensearch/observability/get'
                      - 'cluster:admin/opensearch/observability/update'

                  # Allows users to all PPL functionality
                  ppl_full_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/opensearch/ppl'
                    index_permissions:
                      - index_patterns:
                          - '*'
                        allowed_actions:
                          - 'indices:admin/mappings/get'
                          - 'indices:data/read/search*'
                          - 'indices:monitor/settings/get'

                  # Allows users to read and download Reports
                  reports_instances_read_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/opendistro/reports/instance/get'
                      - 'cluster:admin/opendistro/reports/instance/list'
                      - 'cluster:admin/opendistro/reports/menu/download'

                  # Allows users to read and download Reports and Report-definitions
                  reports_read_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/opendistro/reports/definition/get'
                      - 'cluster:admin/opendistro/reports/definition/list'
                      - 'cluster:admin/opendistro/reports/instance/get'
                      - 'cluster:admin/opendistro/reports/instance/list'
                      - 'cluster:admin/opendistro/reports/menu/download'
                      <I HAD TO REMOVE STUFF HERE OTHERWISE THE TOPIC WAS TOO LARGE>
                      - 'cluster:admin/opensearch/securityanalytics/threatintel/alerts/get'
                      - 'cluster:admin/opensearch/securityanalytics/threatintel/iocs/findings/get'
                      - 'cluster:admin/opensearch/securityanalytics/threatintel/iocs/list'
                      - 'cluster:admin/opensearch/securityanalytics/threatintel/monitors/search'
                      - 'cluster:admin/opensearch/securityanalytics/threatintel/sources/get'
                      - 'cluster:admin/opensearch/securityanalytics/threatintel/sources/search'

                  # Allows users to use all security analytics functionality
                  security_analytics_full_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/opensearch/securityanalytics/alerts/*'
                      - 'cluster:admin/opensearch/securityanalytics/connections/*'
                      - 'cluster:admin/opensearch/securityanalytics/correlationAlerts/*'
                      - 'cluster:admin/opensearch/securityanalytics/correlations/*'
                      - 'cluster:admin/opensearch/securityanalytics/detector/*'
                      - 'cluster:admin/opensearch/securityanalytics/findings/*'
                      - 'cluster:admin/opensearch/securityanalytics/logtype/*'
                      - 'cluster:admin/opensearch/securityanalytics/mapping/*'
                      - 'cluster:admin/opensearch/securityanalytics/rule/*'
                      - 'cluster:admin/opensearch/securityanalytics/threatintel/*'
                    index_permissions:
                      - index_patterns:
                          - '*'
                        allowed_actions:
                          - 'indices:admin/mapping/put'
                          - 'indices:admin/mappings/get'

                  # Allows users to view and acknowledge alerts
                  security_analytics_ack_alerts:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/opensearch/securityanalytics/alerts/*'
                      - 'cluster:admin/opensearch/securityanalytics/correlationAlerts/*'
                      - 'cluster:admin/opensearch/securityanalytics/threatintel/alerts/*'

                  # Allows users to use all Flow Framework functionality
                  flow_framework_full_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/opensearch/flow_framework/*'
                      - 'cluster_monitor'
                    index_permissions:
                      - index_patterns:
                          - '*'
                        allowed_actions:
                          - 'indices:admin/aliases/get'
                          - 'indices:admin/mappings/get'
                          - 'indices_monitor'

                  # Allow users to read flow framework's workflows and their state
                  flow_framework_read_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/opensearch/flow_framework/workflow/get'
                      - 'cluster:admin/opensearch/flow_framework/workflow/search'
                      - 'cluster:admin/opensearch/flow_framework/workflow_state/get'
                      - 'cluster:admin/opensearch/flow_framework/workflow_state/search'
                      - 'cluster:admin/opensearch/flow_framework/workflow_step/get'

                  # Allows users to use all query insights APIs
                  query_insights_full_access:
                    reserved: true
                    cluster_permissions:
                      - 'cluster:admin/opensearch/insights/top_queries/*'
                    index_permissions:
                      - index_patterns:
                          - 'top_queries_by_*'
                        allowed_actions:
                          - "indices_all"

                  # Allow users to execute read only LTR actions
                  ltr_read_access:
                      reserved: true
                      cluster_permissions:
                        - cluster:admin/ltr/caches/stats
                        - cluster:admin/ltr/featurestore/list
                        - cluster:admin/ltr/stats

                  # Allow users to execute all LTR actions
                  ltr_full_access:
                      reserved: true
                      cluster_permissions:
                        - cluster:admin/ltr/*
                roles_mapping.yml: |-
                  _meta:
                    type: "rolesmapping"
                    config_version: 2

                  # Define your roles mapping here

                  ## Demo roles mapping

                  all_access:
                    reserved: false
                    backend_roles:
                    - "admin"
                    description: "Maps admin to all_access"

                  own_index:
                    reserved: false
                    users:
                    - "*"
                    description: "Allow full access to an index named like the username"

                  logstash:
                    reserved: false
                    backend_roles:
                    - "logstash"

                  kibana_user:
                    reserved: false
                    backend_roles:
                    - "kibanauser"
                    description: "Maps kibanauser to kibana_user"

                  readall:
                    reserved: false
                    backend_roles:
                    - "readall"

                  manage_snapshots:
                    reserved: false
                    backend_roles:
                    - "snapshotrestore"

                  kibana_server:
                    reserved: true
                    users:
                    - "kibanaserver"
          volumeClaimTemplate:
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 10Gi

Relevant Logs or Screenshots:

Hi @ViktorEriksson,

I can’t see your internal_users.yml portion of the config.

Try something like:

                internla_users.yml: |-
                 _meta:
                   type: "internalusers"
                   config_version: 2
                 admin:
                   hash: "<hashed_password>"
                   reserved: true
                   backend_roles:
                   - "admin"
                   description: "Demo admin user"
                 anomalyadmin:
                   hash: "<hashed_password>"
                   reserved: false
                   opendistro_security_roles:
                   - "anomaly_full_access"
                   description: "Demo anomaly admin user, using internal role"
                 kibanaserver:
                   hash: "<hashed_password>"
                   reserved: true
                   description: "Demo OpenSearch Dashboards user"

You can find how to hash a password here:

best,
mj

I thought that if I don’t define it, it will use the default one?
Is that not correct assumption?

And today I have this:

          extraEnvs:
            - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
              value: Jagheterviktor123!

Will this override the hashed password in internal_users?

Ok, I finally got into the pod and looked around and you are correct. The default files are gone, so there are no initial_users.yml file. So I added it to the config. But now I get this error:

Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin 
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user. 
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string. 
If a password is not provided, the setup will quit. 
 For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.15.0-130-generic amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.19.1
Detected OpenSearch Security Version: 2.19.1.0
Exception updating the admin password : Unable to update the internal users file with the hashed password.

I guess it is trying to update the admin password with the one I have here:

helm:
        values: |
          singleNode: true
          protocol: https
          extraEnvs:
            - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
              value: Jagheterviktor123!

But why can’t it?

Another question. You say that I should hash my password. But since I use a helm chart, I don’t have opensearch running before I apply my helm chart. So I assume that for helm charts i should just set a random hashed password in initial_users.yml and then rely on some script to update the hashed password with a hash of OPENSEARCH_INITIAL_ADMIN_PASSWORD?