Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpensSearch 2.19.1, Ubuntu, Chrome/Firefox
Describe the issue:
I successfully configured OpenSearch to use SAML authentication with Azure on a single node system. When I tried to apply that same configuration to a cluster, I get a ‘failed to parse SAML header’ and a 500 internal server error.
This seems to be related to some sort of SSL error since I see “ERR_SSL_SSLV3_ALERT_CERTIFICATE_UNKNOWN” in the cluster dashboard logs, but not in the single node logs.
Configuration:
I have 3 nodes connected, running, and dashboards working. I can log in with basic auth and there are no obvious errors in the OpenSearch logs. I have self-signed certificates for each node and PEM certificates and keys configured.
Relevant Logs or Screenshots:
On the single node system, the dashboard logs report:
{"type":"response","@timestamp":"2026-01-29T16:22:17Z","tags":[],"pid":1529,"method":"get","statusCode":200,"req":{"url":"/auth/saml/captureUrlFragment","method":"get","headers":{"host":"10.101.3.21:5601","connection":"keep-alive","sec-ch-ua":"\"Not(A:Brand\";v=\"8\", \"Chromium\";v=\"144\", \"Google Chrome\";v=\"144\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","sec-fetch-site":"same-origin","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.101.2.182","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36"},"res":{"statusCode":200,"responseTime":4,"contentLength":9},"message":"GET /auth/saml/captureUrlFragment 200 4ms - 9.0B"}
{"type":"response","@timestamp":"2026-01-29T16:22:17Z","tags":[],"pid":1529,"method":"get","statusCode":200,"req":{"url":"/auth/saml/captureUrlFragment.js","method":"get","headers":{"host":"10.101.3.21:5601","connection":"keep-alive","sec-ch-ua-platform":"\"macOS\"","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36","sec-ch-ua":"\"Not(A:Brand\";v=\"8\", \"Chromium\";v=\"144\", \"Google Chrome\";v=\"144\"","sec-ch-ua-mobile":"?0","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"https://10.101.3.21:5601/auth/saml/captureUrlFragment","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.101.2.182","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36","referer":"https://10.101.3.21:5601/auth/saml/captureUrlFragment"},"res":{"statusCode":200,"responseTime":2,"contentLength":9},"message":"GET /auth/saml/captureUrlFragment.js 200 2ms - 9.0B"}
{"type":"response","@timestamp":"2026-01-29T16:22:17Z","tags":[],"pid":1529,"method":"get","statusCode":302,"req":{"url":"/auth/saml/login?redirectHash=false","method":"get","headers":{"host":"10.101.3.21:5601","connection":"keep-alive","sec-ch-ua":"\"Not(A:Brand\";v=\"8\", \"Chromium\";v=\"144\", \"Google Chrome\";v=\"144\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","sec-fetch-site":"same-origin","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","referer":"https://10.101.3.21:5601/auth/saml/captureUrlFragment","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.101.2.182","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36","referer":"https://10.101.3.21:5601/auth/saml/captureUrlFragment"},"res":{"statusCode":302,"responseTime":63,"contentLength":9},"message":"GET /auth/saml/login?redirectHash=false 302 63ms - 9.0B"}
On the cluster node running dashboards I see:
{"type":"response","@timestamp":"2026-01-29T16:25:17Z","tags":[],"pid":126111,"method":"get","statusCode":200,"req":{"url":"/auth/saml/captureUrlFragment","method":"get","headers":{"host":"10.101.3.152:5601","connection":"keep-alive","sec-ch-ua":"\"Not(A:Brand\";v=\"8\", \"Chromium\";v=\"144\", \"Google Chrome\";v=\"144\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","sec-fetch-site":"same-origin","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.101.2.182","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36"},"res":{"statusCode":200,"responseTime":2,"contentLength":9},"message":"GET /auth/saml/captureUrlFragment 200 2ms - 9.0B"}
{"type":"response","@timestamp":"2026-01-29T16:25:17Z","tags":[],"pid":126111,"method":"get","statusCode":200,"req":{"url":"/auth/saml/captureUrlFragment.js","method":"get","headers":{"host":"10.101.3.152:5601","connection":"keep-alive","sec-ch-ua-platform":"\"macOS\"","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36","sec-ch-ua":"\"Not(A:Brand\";v=\"8\", \"Chromium\";v=\"144\", \"Google Chrome\";v=\"144\"","sec-ch-ua-mobile":"?0","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"https://10.101.3.152:5601/auth/saml/captureUrlFragment","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.101.2.182","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36","referer":"https://10.101.3.152:5601/auth/saml/captureUrlFragment"},"res":{"statusCode":200,"responseTime":2,"contentLength":9},"message":"GET /auth/saml/captureUrlFragment.js 200 2ms - 9.0B"}
{"type":"error","@timestamp":"2026-01-29T16:25:17Z","tags":["connection","client","error"],"pid":126111,"level":"error","error":{"message":"0078CF68367F0000:error:0A000416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1586:SSL alert number 46\n","name":"Error","stack":"Error: 0078CF68367F0000:error:0A000416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1586:SSL alert number 46\n","code":"ERR_SSL_SSLV3_ALERT_CERTIFICATE_UNKNOWN"},"message":"0078CF68367F0000:error:0A000416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1586:SSL alert number 46\n"}
{"type":"log","@timestamp":"2026-01-29T16:25:18Z","tags":["error","plugins","securityDashboards"],"pid":126111,"message":"Failed to get saml header: Error: Error: failed parsing SAML config"}
{"type":"error","@timestamp":"2026-01-29T16:25:18Z","tags":[],"pid":126111,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n at HapiResponseAdapter.toError (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:127:19)\n at HapiResponseAdapter.toHapiResponse (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:83:19)\n at HapiResponseAdapter.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:79:17)\n at Router.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:175:34)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at handler (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:140:50)\n at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n at Object.internals.handler (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)\n at exports.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)\n at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)\n at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)"},"url":"https://10.101.3.152:5601/auth/saml/login?redirectHash=false","message":"Internal Server Error"}
{"type":"response","@timestamp":"2026-01-29T16:25:18Z","tags":[],"pid":126111,"method":"get","statusCode":500,"req":{"url":"/auth/saml/login?redirectHash=false","method":"get","headers":{"host":"10.101.3.152:5601","connection":"keep-alive","sec-ch-ua":"\"Not(A:Brand\";v=\"8\", \"Chromium\";v=\"144\", \"Google Chrome\";v=\"144\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","sec-fetch-site":"same-origin","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","referer":"https://10.101.3.152:5601/auth/saml/captureUrlFragment","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.101.2.182","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36","referer":"https://10.101.3.152:5601/auth/saml/captureUrlFragment"},"res":{"statusCode":500,"responseTime":15,"contentLength":9},"message":"GET /auth/saml/login?redirectHash=false 500 15ms - 9.0B"}
{"type":"error","@timestamp":"2026-01-29T16:25:18Z","tags":["connection","client","error"],"pid":126111,"level":"error","error":{"message":"0078CF68367F0000:error:0A000416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1586:SSL alert number 46\n","name":"Error","stack":"Error: 0078CF68367F0000:error:0A000416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1586:SSL alert number 46\n","code":"ERR_SSL_SSLV3_ALERT_CERTIFICATE_UNKNOWN"},"message":"0078CF68367F0000:error:0A000416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1586:SSL alert number 46\n"}
{"type":"response","@timestamp":"2026-01-29T16:25:18Z","tags":[],"pid":126111,"method":"get","statusCode":401,"req":{"url":"/favicon.ico","method":"get","headers":{"host":"10.101.3.152:5601","connection":"keep-alive","sec-ch-ua-platform":"\"macOS\"","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36","sec-ch-ua":"\"Not(A:Brand\";v=\"8\", \"Chromium\";v=\"144\", \"Google Chrome\";v=\"144\"","sec-ch-ua-mobile":"?0","accept":"image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"image","referer":"https://10.101.3.152:5601/auth/saml/login?redirectHash=false","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.101.2.182","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36","referer":"https://10.101.3.152:5601/auth/saml/login?redirectHash=false"},"res":{"statusCode":401,"responseTime":2,"contentLength":9},"message":"GET /favicon.ico 401 2ms - 9.0B"}
Each node in the cluster has a nearly identical opensearch.yml configuration, just the node certificates are different:
cluster.name: MyOrg-flow
node.name: "support-aflow1"
node.roles: [master,data,data_content,data_hot,data_warm,data_cold,data_frozen,ingest,ml,remote_cluster_client,transform]
path.data: /mnt/data1/opensearch
path.logs: /var/log/opensearch
network.host: 0.0.0.0
http.port: 9200
http.compression: true
cluster.initial_cluster_manager_nodes: ["support-aflow1","support-aflow2","support-aflow3"]
discovery.seed_hosts: ["10.101.3.150","10.101.3.151","10.101.3.152"]
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.ssl.transport.pemcert_filepath: support-aflow1.pem
plugins.security.ssl.transport.pemkey_filepath: support-aflow1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: support-aflow1.pem
plugins.security.ssl.http.pemkey_filepath: support-aflow1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.audit.type: internal_opensearch
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-config", ".plugins-ml-connector", ".plugins-ml-model-group", ".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".ql-datasources", ".opendistro-asynchronous-search-response*", ".replication-metadata-store", ".opensearch-knn-models"]
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.password_validation_regex: '(?=.*[A-Z])(?=.*[^a-zA-Z\d])(?=.*[0-9])(?=.*[a-z]).{8,}'
plugins.security.restapi.password_validation_error_message: "Password must be minimum 8 characters long and must contain at least one uppercase letter, one lowercase letter, one digit, and one special character."
plugins.security.authcz.admin_dn:
- "CN=admin,OU=support,O=MyOrgFlow,L=Nocona,ST=Texas,C=US"
# Update nodes_dn to match the same style (No spaces around =)
plugins.security.nodes_dn:
- "CN=support-aflow1.lab.local,OU=support,O=MyOrgFlow,L=Nocona,ST=Texas,C=US"
- "CN=support-aflow2.lab.local,OU=support,O=MyOrgFlow,L=Nocona,ST=Texas,C=US"
- "CN=support-aflow3.lab.local,OU=support,O=MyOrgFlow,L=Nocona,ST=Texas,C=US"
Each node has an identical config.yml and the securityadmin.sh was successfully run to push the changes:
---
_meta:
type: "config"
config_version: 2
config:
dynamic:
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
authc:
basic_internal_auth_domain:
description: "Authenticate via HTTP Basic against internal users database"
http_enabled: true
transport_enabled: true
order: 0
http_authenticator:
type: basic
challenge: true
authentication_backend:
type: intern
saml_auth_domain:
description: "Authenticate via SAML"
http_enabled: true
transport_enabled: false
order: 1
http_authenticator:
type: saml
challenge: true
config:
idp:
entity_id: https://sts.windows.net/xxxxxxxxxxxxxxxxxx/
metadata_file: /etc/opensearch/azure-metadata.xml
sp:
entity_id: apstra-flow-opensearch
kibana_url: https://10.101.3.152:5601
exchange_key: 34ee1dc97007b3e97a313e54bf3827c3100e1abe0732409174122a4fe68c7994
roles_key: http://schemas.microsoft.com/ws/2008/06/identity/claims/groups
authentication_backend:
type: noop
authz:
roles_from_myldap:
description: "Authorize via LDAP or Active Directory"
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap
config:
enable_ssl: false
enable_start_tls: false
enable_ssl_client_auth: false
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
rolebase: 'ou=groups,dc=example,dc=com'
rolesearch: '(member={0})'
userroleattribute: null
userrolename: disabled
rolename: cn
resolve_nested_roles: true
userbase: 'ou=people,dc=example,dc=com'
usersearch: '(uid={0})'
roles_from_another_ldap:
description: "Authorize via another Active Directory"
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap
What other information do you need to help me troubleshoot/test?
This has been an ongoing issue for a while and I really need to get it resolve so any help appreciated.
Thanks!
