Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch and OpenSearch Dashboards version 2.12 (I’ve subsequently upgraded OpenSearch to 2.13, but no change in issue)
RHEL 8.9 servers
This is my dev environment. I have 2 hot, 2 warm, 2 cold storage nodes. 3 coordinating and 3 master nodes. Single OpenSearch Dashboards server
Client is Windows 10, and I have tried with Firefox 124.0.2 (64-bit), Edge 123.0.2420.81 (Official build) (64-bit), and Chrome 123.0.6312.106 (Official Build) (64-bit)
Describe the issue:
We have been successfully authenticating users to ElasticSearch with OpenDistro for several years, and I piloted OpenSearch 2.2.1 last year and was using external authentication with the same configuration. We are in the process of moving to OpenSearch. OpenSearch / OpenDashboards was set up and working with local user accounts.
I added the configuration to use PingID as the OAUTH IdP, and all logon attempts yield 401 errors. I have tried to follow the OpenID troubleshooting instructions and increase logging on every OpenSearch server in the environment:
logger.securityjwt.name = com.amazon.dlic.auth.http.jwt
logger.securityjwt.level = trace
However there are no additional details logged. I’ve tried setting the global logging to trace, but this generates an unusable amount of logging data (and I’m still not finding my logon ID in an attempt to grep across all of the logs). I upgraded the OpenSearch environment from 2.12 to 2.13 to see if that would sort the lack of trace logging, but that didn’t pan out.
Configuration:
OpenSearch Dashboard ./config/opensearch_dashboards.yml
# I normally have an array, but limited the hosts to a single server in an attempt to simply troubleshooting
opensearch.hosts: ["https://SingleOpensearchServer.example.net:9200"]
## PingID config
## Comment out lines below to disable
opensearch_security.auth.type: "openid"
opensearch_security.openid.connect_url: "https://pingid-dev.example.com/.well-known/openid-configuration"
opensearch_security.openid.client_id: "REDACTED-BIG-STRING"
opensearch_security.openid.client_secret: "REDACTED-BIG-STRING"
opensearch_security.openid.scope: "openid"
opensearch_security.openid.header: "Authorization"
opensearch_security.openid.base_redirect_url: "https://opensearchdashboard.example.net:5601/auth/openid/login"
OpenSearch ./config/opensearch-security/config.yml
_meta:
type: "config"
config_version: 2
config:
dynamic:
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
authc:
openid_auth_domain:
http_enabled: true
transport_enabled: true
order: 1
http_authenticator:
type: "openid"
challenge: false
config:
openid_connect_idp:
enable_ssl: true
verify_hostnames: false
openid_connect_url: https://pingid-dev.example.com/.well-known/openid-configuration
authentication_backend:
type: noop
kerberos_auth_domain:
http_enabled: false
transport_enabled: false
order: 6
http_authenticator:
type: kerberos
challenge: true
config:
krb_debug: false
strip_realm_from_principal: true
authentication_backend:
type: noop
basic_internal_auth_domain:
description: "Authenticate via HTTP Basic against internal users database"
http_enabled: true
transport_enabled: true
order: 4
http_authenticator:
type: basic
challenge: true
authentication_backend:
type: intern
proxy_auth_domain:
description: "Authenticate via proxy"
http_enabled: false
transport_enabled: false
order: 3
http_authenticator:
type: proxy
challenge: false
config:
user_header: "x-proxy-user"
roles_header: "x-proxy-roles"
authentication_backend:
type: noop
jwt_auth_domain:
description: "Authenticate via Json Web Token"
http_enabled: false
transport_enabled: false
order: 0
http_authenticator:
type: jwt
challenge: false
config:
signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
jwt_header: "Authorization"
jwt_url_parameter: null
jwt_clock_skew_tolerance_seconds: 30
roles_key: null
subject_key: null
authentication_backend:
type: noop
clientcert_auth_domain:
description: "Authenticate via SSL client certificates"
http_enabled: false
transport_enabled: false
order: 2
http_authenticator:
type: clientcert
config:
username_attribute: cn #optional, if omitted DN becomes username
challenge: false
authentication_backend:
type: noop
ldap:
description: "Authenticate via LDAP or Active Directory"
http_enabled: false
transport_enabled: false
order: 5
http_authenticator:
type: basic
challenge: false
authentication_backend:
# LDAP authentication backend (authenticate users against a LDAP or Active Directory)
type: ldap
config:
enable_ssl: false
enable_start_tls: false
enable_ssl_client_auth: false
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
userbase: 'ou=people,dc=example,dc=com'
usersearch: '(sAMAccountName={0})'
username_attribute: null
authz:
roles_from_myldap:
description: "Authorize via LDAP or Active Directory"
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap
config:
enable_ssl: false
enable_start_tls: false
enable_ssl_client_auth: false
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
rolebase: 'ou=groups,dc=example,dc=com'
rolesearch: '(member={0})'
userroleattribute: null
userrolename: disabled
rolename: cn
resolve_nested_roles: true
userbase: 'ou=people,dc=example,dc=com'
usersearch: '(uid={0})'
roles_from_another_ldap:
description: "Authorize via another Active Directory"
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap
OpenSearch ./config//opensearch.yml
# plugins.security.ssl.transport.truststore_filepath needed to be set for OpenID auth to work
plugins.security.ssl.transport.truststore_filepath: /opt/elk/opensearch_config/certs/cacerts
plugins.security.ssl.transport.truststore_type: JKS
plugins.security.ssl.transport.truststore_password: REDACTED
Relevant Logs or Screenshots:
There are, unfortunately, few relevant logs even with my attempt to enable trace logging on authentication. The only OpenSearch log entries are from o.o.j.s.JobScheduler, o.o.s.a.BackendRegistry, and o.o.i.i.ManagedIndexRunner.
opensearch-dashboards.log
{"type":"response","@timestamp":"2024-04-10T16:20:09Z","tags":[],"pid":2850232,"method":"get","statusCode":401,"req":{"url":"/auth/openid/login/auth/openid/login?code=REDACTED&state=REDACTED","method":"get","headers":{"host":"dashboard.example.net:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://pingid-dev.example.com/","dnt":"1","sec-gpc":"1","connection":"keep-alive","upgrade-insecure-requests":"1","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"cross-site","sec-fetch-user":"?1"},"remoteAddress":"10.108.240.186","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0","referer":"https://pingid-dev.example.com/"},"res":{"statusCode":401,"responseTime":4,"contentLength":9},"message":"GET /auth/openid/login/auth/openid/login?code=REDACTED&state=REDACTED 401 4ms - 9.0B"}
{"type":"response","@timestamp":"2024-04-10T16:20:09Z","tags":[],"pid":2850232,"method":"get","statusCode":401,"req":{"url":"/favicon.ico","method":"get","headers":{"host":"dashboard.example.net:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0","accept":"image/avif,image/webp,*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","dnt":"1","sec-gpc":"1","connection":"keep-alive","referer":"https://dashboard.example.net:5601/auth/openid/login/auth/openid/login?code=REDACTED&state=REDACTED","sec-fetch-dest":"image","sec-fetch-mode":"no-cors","sec-fetch-site":"cross-site"},"remoteAddress":"10.108.240.186","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0","referer":"https://dashboard.example.net:5601/auth/openid/login/auth/openid/login?code=REDACTED&state=REDACTED"},"res":{"statusCode":401,"responseTime":2,"contentLength":9},"message":"GET /favicon.ico 401 2ms - 9.0B"}
All of the role mappings that we use with OpenDistro are in place, and these worked with our earlier OpenSearch pilot. My ID is mapped to all_access among other custom roles. In case the IdP is providing the userPrincipalName style ID (uid@example.com), I have added version of my ID to the role mapping as well.
When we had piloted OpenSearch, increasing the logging level for IdP authentication worked & I had been able to troubleshoot/resolve issues on my own. Without the increased logging, I am at a loss as to how to proceed.
