Hello,
I’m trying to setup OpenSearch and OpenSearch Dashboards using Helm Charts (and GitOps/ArgoCD, but I think that’s not too relevant). Its a bit complicated to describe the entire process (libsonnet parsing / ArgoCD applying the result), so I’m going to describe the resulting behaviour and configs on the Pods instead, I hope that’s fine.
OpenSearch itself is running in a three-node environment and reachable at https://opensearch.my-domain.com. I can login using both the admin and kibanaserver user from the internal_users.yml file, so this part worked. I did run the “securityadmin.sh” script to get these user accounts updated.
The config.yml file that is placed in /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml looks as follows:
"_meta":
"config_version": 2
"type": "config"
"config":
"dynamic":
"authc":
"basic_internal_auth_domain":
"authentication_backend":
"type": "internal"
"description": "Authenticate via HTTP Basic against internal users database"
"http_authenticator":
"challenge": false
"type": "basic"
"http_enabled": true
"order": 1
"transport_enabled": true
"openid_auth_domain":
"authentication_backend":
"type": "noop"
"http_authenticator":
"challenge": false
"config":
"enable_ssl": true
"openid_connect_url": "https://login.microsoftonline.com/common/v2.0/.well-known/openid-configuration"
"roles_key": "roles"
"subject_key": "preferred_username"
"verify_hostnames": false
"type": "openid"
"http_enabled": true
"order": 0
"transport_enabled": true
"authz": {}
"http":
"anonymous_auth_enabled": true
"xff":
"enabled": false
"internalProxies": ".*"
"remoteIpHeader": "x-forwarded-for"
This is applied using securityadmin.sh:
$ plugins/opensearch-security/tools/securityadmin.sh -f plugins/opensearch-security/securityconfig/config.yml -icl -nhnv -cacert config/root-ca.pem -cert config/kirk.pem -key config/kirk-key.pemrt config/kirk.pem -key config/kirk
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755 **
**************************************************************************
Security Admin v7
Will connect to localhost:9200 ... done
Connected as "CN=kirk,OU=client,O=client,L=test,C=de"
OpenSearch Version: 2.1.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: opensearch-cluster
Clusterstate: GREEN
Number of nodes: 3
Number of data nodes: 3
.opendistro_security index already exists, so we do not need to create one.
Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!
Populate config from /usr/share/opensearch
Will update '/config' with plugins/opensearch-security/securityconfig/config.yml (legacy mode)
SUCC: Configuration for 'config' created or updated
SUCC: Expected 1 config types for node {"updated_config_types":["config"],"updated_config_size":1,"message":null} is 1 (["config"]) due to: null
SUCC: Expected 1 config types for node {"updated_config_types":["config"],"updated_config_size":1,"message":null} is 1 (["config"]) due to: null
SUCC: Expected 1 config types for node {"updated_config_types":["config"],"updated_config_size":1,"message":null} is 1 (["config"]) due to: null
Done with success
The dashboards config looks as follows (also taken from the pod directly, located in /usr/share/opensearch-dashboards/config/opensearch_dashboards.ym:
opensearch.requestHeadersWhitelist:
- authorization
- securitytenant
opensearch.ssl.verificationMode: none
opensearch_security.auth.type: openid
opensearch_security.cookie.secure: false
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred:
- Global
- Private
opensearch_security.openid.base_redirect_url: https://logs.my-domain.com
opensearch_security.openid.client_id: ${OIDC_CLIENT_ID}
opensearch_security.openid.client_secret: ${OIDC_CLIENT_SECRET}
opensearch_security.openid.connect_url: https://login.microsoftonline.com/${OIDC_TENANT}/v2.0/.well-known/openid-configuration
opensearch_security.openid.scope: openid profile email
opensearch_security.readonly_mode.roles:
- kibana_read_only
server.host: "0"
The environment variables are correctly set (sanitised for obvious reasons):
$ echo $OIDC_CLIENT_ID
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
$ echo $OIDC_CLIENT_SECRET
xxxxxxxxxxxxxxxxxxxxxxxx
$ echo $OIDC_TENANT
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
Still, when accessing https://logs.my-domain.com (the Dashboards URL), I’m getting the following messages:
error
root_cause
0
type "security_exception"
reason "no permissions for [cluster:monitor/main] and User [name=opendistro_security_anonymous, backend_roles=[opendistro_security_anonymous_backendrole], requestedTenant=null]"
type "security_exception"
reason "no permissions for [cluster:monitor/main] and User [name=opendistro_security_anonymous, backend_roles=[opendistro_security_anonymous_backendrole], requestedTenant=null]"
status 403
It seems like its not even trying to authenticate (and I’m not even redirected to AzureAD for authentication). Also I don’t find any trace in the opensearch logs (on none of the three nodes) that I tried to login somehow. Logs of opensearch-dashboards show nothing either, no hint of my login attempt.
Any ideas where I could start looking?