Header securitytenant doesn't work in opensearch-dashboard

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.18.0

Describe the issue:

my default tenant is set to apploggingtenant.
I tested to use curl to change a setting in opensearch dashboard as below:

curl --silent -X POST ${OPENSEARCH_DASHBOARD_URL}/api/opensearch-dashboards/settings/theme:darkMode
-w “\n” -k -H “Content-Type: application/json;” -H “osd-xsrf: true”
-H “securitytenant: apploggingtenant”
-u admin:${AdminRuntimePassword}
-d ‘{“value”:true}’

no error, but the setting always goes to the Global tenant.

I checked, the securitytenant is already added into opensearch.requestHeadersAllowlist

also, I tried to config admin (all_access) user in opensearch dashboard, it doesn’t work.
Configuration:

Relevant Logs or Screenshots:

I made 2 changes, and it seems the issue is resolved:

  1. adding annotation to nginx ingress:
    nginx.ingress.kubernetes.io/configuration-snippet: |
    proxy_set_header securitytenant $http_securitytenant;

  2. change back kibanaserver user. (it seems the admin user doesn’t cover all priv of kibanaserver user)

FYI What it special about kibanaserver is that its the default “internaluser”. i.e. the user defined here in the dynamic security config: security/config/config.yml at main · opensearch-project/security · GitHub

The creds for this user would also need to be set in opensearch_dashboards.yml in the opensearch.username and opensearch.password settings. This user can access dashboards endpoints.