Custom config files cannot take effect

[alendill]

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch: 2.5.0
Openserach_Helm-chart: 2.10.0
Opensearch-dashboard: 2.5.0
Opensearch-dashboard_Helm-chart: 2.8.0

Describe the issue:
We are trying to manage authentication and authorization via config.yml as Document says.
But we found it never take effect. This caused us to be unable to pass OIDC authentication and login(Always receive {“statusCode”:401,“error”:“Unauthorized”,“message”:“Unauthorized”})
Our opensearch is deployed via Helm-chart in GKE environment, and manage by ArgoCD.
And we also checked the configuration in the values.
And the “config.yml” under the default config path(“/usr/share/opensearch/config/opnsearch-security”) in each pod can also see our own configuration content.

Configuration:

securityConfig:
  enabled: true
  path: "/usr/share/opensearch/config/opnsearch-security"
  config:
    securityConfigSecret: ""
    dataComplete: true
    data:
      config.yml: |
      ... some our own config(e.g. There is no "openid_auth_domain" in default value, we added it. And "jwt_auth_domain.http_enabled" default is false, we change false to true)

To Reproduce

1. Deploy opensearch via helm-chart.
2. Use “values.yml” as a template to add custom content.
3. Sync application in ArgoCD.
4. When each pod online, logon opensearch-dashboard page via admin, and into “Security” → “Authentication”
5. There is no our own config effect, no “openid_auth_domain” displayed on “Authentication sequences” and “jwt_auth_domain.http_enabled” still is false.
6. We tried comment all of auth domain except “basic_internal_auth_domain”

Relevant Logs or Screenshots:
Sorry for these camera photos, because of company policy, me cannot post screenshots directly here
Logs of Opensearch backend:

[pablo]

@alendill I understand you’ve deployed the cluster using securityconfig.config.data: in values.yaml

Did you define internal_users.yml, roles.yml, roles_mapping.yml, action_groups.yml and tenants.yml?

What is your desired type of authentication?
If OIDC, are you getting redirected to the IDP for authentication or you’re getting the 401 error straight away?

[alendill]

Hi pablo,
Thank you for your help.
First, these config files are currently defined as well, but are just copy-pasted defaults.
And then here are some things to add: At present, we want to change the auth_domain list so that only the auth_domain we configured will be displayed on the opensearch-dashboard “Security” → “Authentication” page.
We add “openid_auth_domain” in config.yml, but it did not shown on that page.
At this point we don’t know if the two issues are related.

securityConfig:
  enabled: true
  path: "/usr/share/opensearch/config/opnsearch-security"
  config:
    securityConfigSecret: ""
    dataComplete: true
    data:
      config.yml: |
      ...
        basic_internal_auth_domain:
          description: "Authenticate via HTTP Basic against internal users database"
          http_enabled: true
            transport_enabled: true
            order: 0
            http_authenticator:
              type: basic
              challenge: false
            authentication_backend:
              type: intern
        openid_auth_domain:
          http_enabled: true
          transport_enabled: true
          order: 1
          http_authenticator:
            type: openid
            challenge: false
            config:
              subject_key: preferred_username
              roles_key: roles
              openid_connect_url: https://domain/auth/realms/myrealm/.well-known/openid-configuration
              enable_ssl: true
              verify_hostnames: false
          authentication_backend:
            type: noop
      ...

As you can see, there is no “openid_auth_domain”.

[pablo]

@alendill If this is an exact copy of your config, then your indent is incorrect.

basic_internal_auth_domain: and openid_auth_domain: must be in the same column.

When you deploy the OpenSearch cluster with Helm charts, you will see a secret called opensearch-cluster-master-securityconfig. It will contain all the security plugin configs encoded with base64.

Try to decode the config.yml one.

kubectl get secret opensearch-cluster-master-securityconfig -o jsonpath={'.data.config\.yml'} | base64 -d

It will show you what was uploaded to the secret.

[alendill]

Hi pablo,
I’m sorry posted a wrong format sample config file here, it is not our real config file. And I reformatted it to match ours. Please check.
I input this command line and the config of output as same as we configured.
But the point is, we added the “openid_auth_domain” to config, but we can not see it in “Security” → “Authentication” page. Even we added all of “internal_users.yml”, “roles.yml”, “roles_mapping.yml”, “action_groups.yml”, “tenants.yml”.

[pablo]

@alendill A quick question about the helm chart deployment. When you make the change in the config.yml file do you run helm upgrade or reinstall helm charts?

If you reinstall, do you remove the PVC and clean PV before installation?

[pablo]

@alendill I reviewed the charts and I can’t find anything related to securityadmin.sh. This means that config is uploaded only once during the cluster creation.
If you run helm upgrade the config will get updated in the secret and that will update the config file inside the pod (/usr/share/opensearch/config/opensearch-security). However, it won’t be uploaded to the OpenSearch security plugin.

You need to run securityadmin.sh to upload the configuration.

[alendill]

Hi pablo,
Thank you for your help and suggest.
We tried to reinstall opensearch backend today, then config file worked.
And next we tried to run securityadmin.sh to upload the configuration, it worked as well.
Me posted command line here, hope it helps others.

./securityadmin.sh -cd ../securityconfig/ -icl -nhnv \
  -cacert ../../../config/root-ca.pem \
  -cert ../../../config/kirk.pem \
  -key ../../../config/kirk-key.pem

Anyway, thank you again.