HELM securityConfigSecret not taking effect at bootstrap time

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch 2.11.0
helm chart 2.16.1

Describe the issue:
I was under the impression that the chart’s securityConfig-section is there to bootstrap the security of the Opensearch. So, no need to manually run securityadmin.sh, right ?

However, when including “securityConfigSecret” in the values.yaml of the helm chart, the secret is mounted correctly in the pod (although some warnings about wrong file/directory permissions.)
But it seems to not initialize with these settings and declares that “[2023-10-20T14:24:05,349][ERROR][o.o.s.a.BackendRegistry ] [opensearch-test-master-0] Not yet initialized (you may need to run securityadmin)”

Here the log shows it correctly finds the mounted secrets’ content:

[2023-10-20T14:03:51,366][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] Directory /usr/share/opensearch/config/opensearch-security has insecure file permissions (should be 0700)
[2023-10-20T14:03:51,366][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/audit.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,366][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/internal_users.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,366][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/action_groups.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,366][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/roles.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,366][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/config.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,367][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/nodes_dn.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,367][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/whitelist.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,367][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/tenants.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,367][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/roles_mapping.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,367][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/..data has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,367][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] Directory /usr/share/opensearch/config/opensearch-security/..2023_10_20_14_03_44.3698745204 has insecure file permissions (should be 0700)
[2023-10-20T14:03:51,367][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/..2023_10_20_14_03_44.3698745204/internal_users.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,367][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/..2023_10_20_14_03_44.3698745204/audit.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,368][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/..2023_10_20_14_03_44.3698745204/config.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,368][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/..2023_10_20_14_03_44.3698745204/nodes_dn.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,368][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/..2023_10_20_14_03_44.3698745204/whitelist.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,368][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/..2023_10_20_14_03_44.3698745204/tenants.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,368][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/..2023_10_20_14_03_44.3698745204/roles_mapping.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,368][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/..2023_10_20_14_03_44.3698745204/action_groups.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,368][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch-security/..2023_10_20_14_03_44.3698745204/roles.yml has insecure file permissions (should be 0600)
[2023-10-20T14:03:51,369][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-test-master-0] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)

Configuration:
here’s the securityConfig of the values file:

  securityConfig:
    enabled: true
    path: "/usr/share/opensearch/config/opensearch-security"
    actionGroupsSecret:
    configSecret:
    internalUsersSecret:
    rolesSecret:
    rolesMappingSecret:
    tenantsSecret:
    # The following option simplifies securityConfig by using a single secret and
    # specifying the config files as keys in the secret instead of creating
    # different secrets for for each config file.
    # Note that this is an alternative to the individual secret configuration
    # above and shouldn't be used if the above secrets are used.
    config:
      # There are multiple ways to define the configuration here:
      # * If you define anything under data, the chart will automatically create
      #   a secret and mount it. This is best option to choose if you want to override all the
      #   existing yml files at once.
      # * If you define securityConfigSecret, the chart will assume this secret is
      #   created externally and mount it. This is best option to choose if your intention is to
      #   only update a single yml file.
      # * It is an error to define both data and securityConfigSecret.
      securityConfigSecret: "opensearch-test-config"
      dataComplete: true
      data: {}
        # config.yml: |-
        # internal_users.yml: |-
        # roles.yml: |-
        # roles_mapping.yml: |-
        # action_groups.yml: |-
        # tenants.yml: |-
2

I wanted to delete the post, since I’ve figured out what the problem was/is
There is a “chown -R 1000:1000” in the statefulsets when persistent volumes are being used.
And since I’m deploying on Openshift with SCC in place, there is no USER 1000 obviously.

Once, the InitChown was removed from the values files, it all worked as expected