All,
I have an issue with OpenSearch -2.12.0 initialize auth domain saml_auth_domain. Brief history, I’ve been upgrading OpenSearch/OpenSearch-dashboard since 1.2.0. My last successful upgrade was Opensearch-2.11.1. This time around It failed to enable my SAML authentication configuration.
Error:
2024-03-12T20:45:12,450][ERROR][o.o.s.s.DynamicConfigModelV7] [opensearch. domain.com] Unable to initialize auth domain saml_auth_domain=AuthcDomain [http_enabled=true, order=1, http_authenticator=HttpAuthenticator [challenge=true, type=saml, config={idp={metadata_url=https://zitadel-build. domain.com/saml/v2/metadata, entity_id=https://zitadel-build. domain.com/saml/v2/metadata}, sp={entity_id=https://opensearch. domain.com:5601}, kibana_url=https://opensearch.domain.com:5601, subject_key=Email, roles_key=Role, exchange_key=G5SjOsBQ3AqPaxWsxCv1q0lOLub3c21a2NN6mn0GhSOl7p2B9w0NlKe7mt3trtG95rc4nZ7I4+0qZXWi4R6yIOQ7R4BYXFD0Y8YTbRmkIwMnnrxmY9aTpuwIDAQABo0gwRjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwIwHwYDVR0jBBgwFoAUQKkdCLAULR6Co9zXpwHbDrmfhMUwDQYJKoZIhvcNAQELBQADggIBAFef1nnNUyn5WqDWQA/Ws4EQQfq6iR43Khdsv0/Bn8dKYJRhtMoJYkvczDwI2WZ0Af0Eq0OjDZv+n8iyu/UXgO8XDAIaXjwv3occV6h6/TXfyFoEyupuC5oo4tlt77anu4Do6faUJNIbE23LRvDplUDqdJcWZ+5WluDddoQhI5M/gaeiYYes4GxDrUQtxpTKLdbtr4f5STfvcr9aNTbPplGRAzCkg7+/gcxBkBgHFr8h0Mf3NbHfasUh3nE9B1iht6pvZ/g/FoCkfZmguRv9tKhCnnI1lddVRa/px46lkfkyFsiUNhFx8fdhPKiwWRgBXmUFTFwDOmYWPsdOreB26xqhXLdNKyc0gvXHwp+WQpfTe45kuLHi4jq2DJ9h0hXe1/XZSv3gWzLumK1ErouAm+dAdmeiUIZEheSY5L2MzEh/l0fUw1uHPT3GnicXbpB7Tjxa9Hwhk0uar6KydLw36WQDeGTJjlca3G/6hJorR6ycJV3m0tSk6UeUrAgdy+rD9KxnENnAH2ISoOQf5YBrjQkudCZsTReHJnkPjw1BiC8ozhjaNBR0TqYP8wOJwuYAukbsIVbZhHDZSBcxjMYSVblwVlEZLMcsrLeeMcpRYRCDyaAABJ3tqawAUUkGAwX7+vuuVoOPBpGYYIzmsA8fuzka0AFmNDKmeOT7La0hfP91}], authentication_backend=AuthcBackend [type=noop, config={}], description=null] due to OpenSearchException[java.lang.reflect.InvocationTargetException]; nested: InvocationTargetException; nested: RuntimeException[java.lang.IllegalArgumentException: Illegal base64 character 2f]; nested: IllegalArgumentException[Illegal base64 character 2f];
org.opensearch.OpenSearchException: java.lang.reflect.InvocationTargetException
at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:73) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.securityconf.DynamicConfigModelV7.lambda$newInstance$1(DynamicConfigModelV7.java:426) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
at org.opensearch.security.securityconf.DynamicConfigModelV7.newInstance(DynamicConfigModelV7.java:424) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.securityconf.DynamicConfigModelV7.buildAAA(DynamicConfigModelV7.java:323) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.securityconf.DynamicConfigModelV7.<init>(DynamicConfigModelV7.java:101) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:285) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:430) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:419) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:402) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:390) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.action.configupdate.TransportConfigUpdateAction.nodeOperation(TransportConfigUpdateAction.java:128) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.action.configupdate.TransportConfigUpdateAction.nodeOperation(TransportConfigUpdateAction.java:52) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:200) [opensearch-2.12.0.jar:2.12.0]
at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:328) [opensearch-2.12.0.jar:2.12.0]
at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:324) [opensearch-2.12.0.jar:2.12.0]
at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:206) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:211) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:105) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:795) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:114) [opensearch-index-management-2.12.0.0.jar:2.12.0.0 `
at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) [opensearch-performance-analyzer-2.12.0.0.jar:2.12.0.0]
at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) [opensearch-2.12.0.jar:2.12.0]
at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1059) [opensearch-2.12.0.jar:2.12.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-2.12.0.jar:2.12.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.12.0.jar:2.12.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
Another error.
[2024-03-12T20:57:00,595][ERROR][c.a.d.a.h.s.HTTPSamlAuthenticator] [opensearch.domain.com] Error creating HTTPSamlAuthenticator. SAML authentication will not work
java.lang.IllegalArgumentException: Illegal base64 character 2f
at java.base/java.util.Base64$Decoder.decode0(Base64.java:852) ~[?:?]
at java.base/java.util.Base64$Decoder.decode(Base64.java:570) ~[?:?]
at java.base/java.util.Base64$Decoder.decode(Base64.java:593) ~[?:?]
at com.amazon.dlic.auth.http.saml.AuthTokenProcessorHandler.createJwkFromSettings(AuthTokenProcessorHandler.java:245) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
at com.amazon.dlic.auth.http.saml.AuthTokenProcessorHandler.<init>(AuthTokenProcessorHandler.java:113) ~[opensearch-security-2.12.0.0.jar:2.12.0.0]
at com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator.<init>(HTTPSamlAuthenticator.java:148) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.securityconf.DynamicConfigModelV7.lambda$newInstance$1(DynamicConfigModelV7.java:426) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) [?:?]
at org.opensearch.security.securityconf.DynamicConfigModelV7.newInstance(DynamicConfigModelV7.java:424) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.securityconf.DynamicConfigModelV7.buildAAA(DynamicConfigModelV7.java:323) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.securityconf.DynamicConfigModelV7.<init>(DynamicConfigModelV7.java:101) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:285) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:430) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:419) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:402) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.configuration.ConfigurationRepository.initalizeClusterConfiguration(ConfigurationRepository.java:227) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at org.opensearch.security.configuration.ConfigurationRepository.lambda$initOnNodeStart$0(ConfigurationRepository.java:318) [opensearch-security-2.12.0.0.jar:2.12.0.0]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
[2024-03-12T20:57:00,601][WARN ][o.o.s.s.ReflectionHelper ] [opensearch. domain.com] Unable to enable 'com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator' due to java.lang.reflect.InvocationTargetException
What I have done so far
Create a new instance for OpenSearch/Opensearch-dashboards-2.12.0.
Followed the exact guidelines in the documentation for enabling Production mode and configuration needed.
The file /etc/environment was edited for new version 2.12.0 Logged out and back in.
root@opensearch:/usr/share/opensearch/plugins/opensearch-security/tools# cat /etc/environment. Logged out and back in.
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
OPENSEARCH_INITIAL_ADMIN_PASSWORD=PasswordHoward123!
Edited my config.yml file.
basic_internal_auth_domain:
description: "Authenticate via HTTP Basic against internal users database"
http_enabled: true
transport_enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
saml_auth_domain:
http_enabled: true
transport_enabled: false
order: 1
http_authenticator:
type: saml
challenge: true
config:
idp:
metadata_url: https://zitadel-build.domain.com/saml/v2/metadata
#metadata_file: /etc/opensearch/zitadel.xml
entity_id: https://zitadel-build. domain.com/saml/v2/metadata
sp:
entity_id: https://opensearch.domain.com:5601
kibana_url: https://opensearch.domain.com:5601
subject_key: Email
roles_key: Role
exchange_key: G5SjOsBQ3AqPaxWsxCv1q0lOLub3c21a2NN6mn0GhSOl7p2B9w0NlKe7mt3trtG95rV9/Vgxr4RU4rwYz43wDMPS0+JbUYkEG5G5SjOsBQG95rc4nZ7I4+ /Ws4EQQfq6iR43Khdsv0/Bn8dKYJRhtMoJYkvczDwI2WZ0Af0Eq0OjDZv+n8iyu/ +5WluDddoQhI5M/gaeiYYes4GxDrUQtxpTKLdbtr4f5STfvcr9aNTbPplGRAzCkg7+/ /px46lkfkyFsiUNhFx8fdhPKiwWRgBXmUFTFwDOmYWPsdOreB26xqhXLdNKyc0gvXHwp+ /6hJorR6ycJV3m0tSk6UeUrAgdy+
authentication_backend:
type: noop
As you can see, I tried both metadata_file: && metadata_url, No joy.
I executed my security script after adjusting my config.yml file.
root@opensearch:/usr/share/opensearch/plugins/opensearch-security/tools# ./securityadmin.sh -h opensearch.domain.com -f /etc/opensearch/opensearch-security/config.yml -cacert /etc/opensearch/root-ca.pem -cert /etc/opensearch/admin.pem -key /etc/opensearch/admin-key.pem -icl -nhnv
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755 **
**************************************************************************
Security Admin v7
Will connect to opensearch.domain.com:9200 ... done
Connected as "CN=ADMIN,OU=ADMIN,O=ZITADEL,L=CEDAR,ST=IOWA,C=US"
OpenSearch Version: 2.12.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: opensearch
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /usr/share/opensearch/plugins/opensearch-security/tools
Will update '/config' with /etc/opensearch/opensearch-security/config.yml
SUCC: Configuration for 'config' created or updated
SUCC: Expected 1 config types for node {"updated_config_types":["config"],"updated_config_size":1,"message":null} is 1 (["config"]) due to: null
Done with success
root@opensearch:/usr/share/opensearch/plugins/opensearch-security/tools#
Not sure what going on or if I missed something from the new release I’m supposed to configure/Adjust. As for now I rolled back out production instance to version-2.11.1.
IDP, we were using Keycloak until last year, so we moved to Zitadel. Still did not have a issue with upgrade or connection to our IDP.
IDP XML file.
<?xml version="1.0"?>
<md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" entityID="https://opensearch.domain.com:5601">
<md:SPSSODescriptor protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol urn:oasis:names:tc:SAML:1.1:protocol">
<md:AssertionConsumerService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://opensearch.domain.com:5601/_opendistro/_security/saml/acs" index="0"/>
</md:SPSSODescriptor>
</md:EntityDescriptor>
Since Opensearch fail to enable SAML my dashboard is scream about the security plugin
{"type":"log","@timestamp":"2024-03-13T02:17:59Z","tags":["error","plugins","securityDashboards"],"pid":5779,"message":"StatusCodeError: Authorization Exception\n at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)\n at IncomingMessage.emit (node:events:529:35)\n at IncomingMessage.emit (node:domain:489:12)\n at endReadableNT (node:internal/streams/readable:1400:12)\n at processTicksAndRejections (node:internal/process/task_queues:82:21) {\n status: 403,\n displayName: 'AuthorizationException',\n path: '/_plugins/_security/tenantinfo',\n query: {},\n body: undefined,\n statusCode: 403,\n response: '',\n toString: [Function (anonymous)],\n toJSON: [Function (anonymous)]\n}"}
{"type":"log","@timestamp":"2024-03-13T02:18:20Z","tags":["error","plugins","securityDashboards"],"pid":5779,"message":"Failed to get saml header: Authentication Exception :: {\"path\":\"/_plugins/_security/authinfo\",\"query\":{},\"statusCode\":401,\"response\":\"Authentication finally failed\"}"}
This was a new Instance, most default settings
- Ubuntu-22.0.4
- Full Updates/Upgrades
- Opensearch-2.12.0
- Opensearch-Dashboards-2.12.0
- JAVA-17 which comes with installation
- JAVA_HOME is set
- ADMIN password is set in environment file.
- Create new certificates from documentation guide.
Opensearch Yaml file
root@opensearch:/usr/share/opensearch/plugins/opensearch-security/tools# cat /etc/opensearch/opensearch.yml
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 10.10.10.10
http.port: 9200
discovery.type: single-node
plugins.security.disabled: false
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-config", ".plugins-ml-connector", ".plugins-ml-model-group", ".plugins-ml-model", ".plugins-ml-task", ".plugins-ml-conversation-meta", ".plugins-ml-conversation-interactions", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".ql-datasources", ".opendistro-asynchronous-search-response*", ".replication-metadata-store", ".opensearch-knn-models", ".geospatial-ip2geo-data*"]
node.max_local_storage_nodes: 3
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/node1.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/node1.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/node1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.authcz.admin_dn:
- 'CN=ADMIN,OU=ADMIN,O=ZITADEL,L=CEDAR,ST=IOWA,C=US'
plugins.security.nodes_dn:
- 'CN=opensearch.domain.com,OU=ADMIN,O=ZITADEL,L=CEDAR,ST=IOWA,C=US'
OpenSearch-Dashboards YAML file
root@opensearch:/usr/share/opensearch/plugins/opensearch-security/tools# cat /etc/opensearch-dashboards/opensearch_dashboards.yml
---
server.port: 5601
server.host: "opensearch.domain.com"
server.name: "opensearch.domain.com"
opensearchDashboards.index: ".opensearch_dashboards"
opensearchDashboards.defaultAppId: "home"
logging.dest: /var/log/opensearch-dashboards/opensearch-dashboards.log
ml_commons_dashboards.enabled: true
opensearch_security.ui.saml.login.buttonname: Zitadel
opensearch_security.auth.type: ["basicauth","saml"]
opensearch_security.auth.multiple_auth_enabled: true
opensearch.hosts: [https://10.10.10.10:9200]
opensearch.ssl.verificationMode: none
opensearch.username: admin
opensearch.password: PasswordHoward123!
opensearch.requestHeadersWhitelist: [authorization, securitytenant]
#opensearch.requestHeadersAllowlist: ["securitytenant","Authorization"]
server.ssl.enabled: true
server.xsrf.allowlist: ["/_opendistro/_security/saml/acs", "/_opendistro/_security/saml/logout"]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
opensearch_security.cookie.secure: true
server.ssl.certificate: /etc/opensearch-dashboards/node1.pem
server.ssl.key: /etc/opensearch-dashboards/node1-key.pem
opensearch.ssl.certificateAuthorities: /etc/opensearch-dashboards/root-ca.pem
My first thougth after upgrading was I missed something or configuration made from years ago may have an effect. so I created a fresh install with default configuration to ensure I either resolved this or confirmed this error stil exsists.
Any Advice or suggestion would be appreciated.
Thank you in advance.
-Gsmitt
