Opensearch with Okta SSO intergration (multitenancy) 500 internal server error

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch 2.7.0 (opensource)
Opensearch Dashboards 2.7.0
OS: Amazon Linux 2023
Browser: Version 114.0.5735.106 (Official Build) (arm64)

Describe the issue:
I have run into a peculiar and hard to explain issue. I have configured SSO with Okta for our first Opensearch cluster (clusterA) running on AWS account on EC2 instances and it works. The setup is as follows:
Opensearch is running on 3 nodes (3 ec2 instances), Opensearch Dashboards is running on a k8s pod. In the same fashion, our cluster B residing in a distinct AWS account (distinct VPC) with same configuration (besides domain) does not work with okta. I have tried creating distinct okta apps for Opensearch (Opensearch_A and Opensearch_B) as well as one okta application (Opensearch) with multitenancy.

When I try SP initiated login for clusterA ( kl. exampleA .io ) i get redirected to OKTA single sign on page, i sign in and I get transferred to opensearch dashboards page offering sign in using user/pass and SSO. I select SSO (also user/pass works) and I can get inside the Opensearch Dashboards with correct user mapping and so on. When I try the same for clusterB ( kl. exampleB. io ) i get redirected to OKTA single sign on page, i sign in and I get transferred to opensearch dashboards page offering sign in using user/pass and SSO. I select SSO and i get 500 internal error. What is peculiar is that in the second case i see the following URL: /auth/saml/login?nextUrl=%2F&redirectHash=false

I’ve tried a lot of things, like using single okta app with multitenancy, or 2 distinct ones for each domain, or just using the okta app for the working cluster and changing its configuration with clusterB. However, nothing works, and this leads me to believe that the problem should lie on the Opensearch side. However, as configuration is almost identical (as seen in the following section), I don’t know what could be the issue.

Configuration:
ClusterA (working):
opensearch.yml

# Security plugin configuration
plugins.security.ssl.transport.pemcert_filepath: tls/node.pem
plugins.security.ssl.transport.pemkey_filepath: tls/node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: tls/root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: tls/node.pem
plugins.security.ssl.http.pemkey_filepath: tls/node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: tls/root-ca.pem
plugins.security.nodes_dn:
  - 'CN=osearch-logs-*.stg.internal,OU=infra,O=Dino,C=ES'
plugins.security.authcz.admin_dn:
  - 'CN=dino.admin.opensearch,OU=infra,O=Dino,C=ES'

Opensearch-security config.yml

    authc:
      saml_auth_domain:
        http_enabled: true
        transport_enabled: false
        order: 1
        http_authenticator:
          type: saml
          challenge: true
          config:
            idp:
              metadata_url: https:// example. okta-emea. com/app/exkahm4k463RUdobH0i7/sso/saml/metadata #SAML's metadata url, provided by your IdP
              entity_id:  http:// www. okta. com/exkahm4k463RUdobH0i7 #SAML's IdP entity ID, provided by your IdP
            sp:
              entity_id: opensearch-dashboards-saml
            kibana_url: https:// kl. exampleA. io/
            roles_key: Role
            exchange_key: "b7d5d3fe26c0d1d8c630bf6401904e3f9c388dde1c4b8479f4a85961fce94f01"
        authentication_backend:
          type: noop
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern

opensearch_dashboards.yml

opensearch.hosts: [https:// localhost: 9200]
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: kibanaserver
opensearch.requestHeadersWhitelist: [authorization, securitytenant]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
# Use this setting if you are running opensearch-dashboards without https
opensearch_security.cookie.secure: false
server.host: '0.0.0.0'
opensearch_security.auth.type: "saml"
opensearch_security.auth.multiple_auth_enabled: false
server.xsrf.allowlist: ["/_plugins/_security/saml/acs/idpinitiated", "/_plugins/_security/saml/acs", "/_plugins/_security/saml/logout", "/_opendistro/_security/saml/acs", "/_opendistro/_security/saml/acs/idpinitiated", "/_opendistro/_security/saml/logout"]

ClusterB (not working):
opensearch.yml

# Security plugin configuration
plugins.security.ssl.transport.pemcert_filepath: tls/node.pem
plugins.security.ssl.transport.pemkey_filepath: tls/node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: tls/root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: tls/node.pem
plugins.security.ssl.http.pemkey_filepath: tls/node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: tls/root-ca.pem
plugins.security.nodes_dn:
  - 'CN=osearch-logs-*.prod.internal,OU=infra,O=Dino,C=ES'
plugins.security.authcz.admin_dn:
  - 'CN=dino.admin.opensearch,OU=infra,O=Dino,C=ES'

Opensearch-security config.yml

    authc:
      saml_auth_domain:
        http_enabled: true
        transport_enabled: false
        order: 1
        http_authenticator:
          type: saml
          challenge: true
          config:
            idp:
              metadata_url: https:// example. okta-emea. com/app/exkahm4k463RUdobH0i7/sso/saml/metadata #SAML's metadata url, provided by your IdP
              entity_id:  http:// www. okta. com/exkahm4k463RUdobH0i7 #SAML's IdP entity ID, provided by your IdP
            sp:
              entity_id: opensearch-dashboards-saml
            kibana_url: https:// kl. exampleB. io/
            roles_key: Role
            exchange_key: "b7d5d3fe26c0d1d8c630bf6401904e3f9c388dde1c4b8479f4a85961fce94f01"
        authentication_backend:
          type: noop
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern

opensearch_dashboards.yml

opensearch.hosts: [https:// localhost: 9200]
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: obscured
opensearch.requestHeadersWhitelist: [authorization, securitytenant]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
# Use this setting if you are running opensearch-dashboards without https
opensearch_security.cookie.secure: false
server.host: '0.0.0.0'
opensearch_security.auth.type: "saml"
opensearch_security.auth.multiple_auth_enabled: false
server.xsrf.allowlist: ["/_plugins/_security/saml/acs/idpinitiated", "/_plugins/_security/saml/acs", "/_plugins/_security/saml/logout", "/_opendistro/_security/saml/acs", "/_opendistro/_security/saml/acs/idpinitiated", "/_opendistro/_security/saml/logout"]

The configuration of the okta app that works is as follows. Note that this works only for the exampleA domain, while exampleB does not redirect:

Single Sign On URL:    https:// kl. exampleA .io/_opendistro/_security/saml/acs
Requestable SSO URLs                                   URLIndex:
https:// kl. exampleA. io/_opendistro/_security/saml/acs   0
https:// kl. exampleB. io/_opendistro/_security/saml/acs     1
Recipient URL:      https:// kl. exampleA. io/_opendistro/_security/saml/acs
Destination URL:    https:// kl. exampleA. io/_opendistro/_security/saml/acs
Audience Restriction:         opensearch-dashboards-saml
Default Relay:
StateName ID Format:   Unspecified
Response:      Signed
Assertion Signature:   Signed
Signature Algorithm:   RSA_SHA256
Digest Algorithm:   SHA256Assertion
Encryption: Unencrypted
SAML Single Logout:   Disabled
SAML Signed Request:  Disabled
authnContextClassRef:  PasswordProtectedTransport
Honor Force Authentication:    Yes
Assertion Inline Hook:    None (disabled)
SAML Issuer ID:  http:// www. okta. com/${org.externalKey}
ATTRIBUTE STATEMENTS
Name        Name Format        Value

GROUP ATTRIBUTE STATEMENTS
Name           Name Format           Filter
Role Unspecified  Starts with: okta_opensearch_

Metadata URL: https:// myorg. okta-emea. com/app/exkahm4k463RUdobH0i7/sso/saml/metadata
Sign on URL: https:// myorg. okta-emea. com/app/myorg_opensearch_4/exkahm4k463RUdobH0i7/sso/saml
Issuer: http:// www. okta. com/exkahm4k463RUdobH0i7

Relevant Logs or Screenshots:
opensearch dashboards logs (clusterB):

{"type":"response","@timestamp":"2023-06-14T09:55:30Z","tags":[],"pid":453,"method":"get","statusCode":200,"req":{"url":"/auth/saml/captureUrlFragment.js","method":"get","headers":{"x-forwarded-for":"185.49.168.193","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kl. exampleB. io","x-amzn-trace-id":"Root=1-64898e92-7de0e472756e2e9868d71500","x-amzn-oidc-data":"eyJ0eXAiOiJKV1QiLCJraWQiOiI3ZGYzZGU4Mi1kMjBkLTQ1Y2EtOGE0NC00YWE1ODAyMzYzYzYiLCJhbGciOiJFUzI1NiIsImlzcyI6Imh0dHBzOi8vZGlub3RlY2gub2t0YS1lbWVhLmNvbSIsImNsaWVudCI6IjBvYTJ4Y2dpYXpUNjFOWGt1MGk3Iiwic2lnbmVyIjoiYXJuOmF3czplbGFzdGljbG9hZGJhbGFuY2luZzpldS13ZXN0LTE6MzE2Mzk5ODM4MzA5OmxvYWRiYWxhbmNlci9hcHAvNDExMGMyYTYtZGVmYXVsdC1vYXV0aGdhdGUtOGUwMS81Y2EyMzlmNTY3NmM0MzNjIiwiZXhwIjoxNjg2NzM2NjUwfQ==.eyJzdWIiOiIwMHU0aGE3dGlzcklHMzBtbjBpNyIsImV4cCI6MTY4NjczNjY1MCwiaXNzIjoiaHR0cHM6Ly9kaW5vdGVjaC5va3RhLWVtZWEuY29tIn0=.VRcJ0uXOlxPwp2q4PfVgjYiiDTwKYPr7WIVfV1yN6saWjrY5kwIBYep8mIbeRy9MjKPeuy_8zThxvv3Ggovezg==","x-amzn-oidc-identity":"00u4ha7tisrIG30mn0i7","x-amzn-oidc-accesstoken":"eyJraWQiOiI2clFNMjVYcGFZV29BUEMtWUNySGJtVWVhNEN2bjNDOUdiZkRiX3d4bHlBIiwiYWxnIjoiUlMyNTYifQ.eyJ2ZXIiOjEsImp0aSI6IkFULjNnVDJIenhQVGVJVEFhUU96WS0wSnRpbHJ0ZUVDTzZoZ1hqSXFBMTc1LVEiLCJpc3MiOiJodHRwczovL2Rpbm90ZWNoLm9rdGEtZW1lYS5jb20iLCJhdWQiOiJodHRwczovL2Rpbm90ZWNoLm9rdGEtZW1lYS5jb20iLCJzdWIiOiJzdGVmYW5vcy5wbGlha29zQGRpbm90ZWNoLmNvbSIsImlhdCI6MTY4NjY2MDg3NCwiZXhwIjoxNjg2NjY0NDc0LCJjaWQiOiIwb2EyeGNnaWF6VDYxTlhrdTBpNyIsInVpZCI6IjAwdTRoYTd0aXNySUczMG1uMGk3Iiwic2NwIjpbIm9wZW5pZCJdLCJhdXRoX3RpbWUiOjE2ODY2NjA0Mzl9.QGNSCNpBJ8y4ncb8c-cAlWwrshL4zKllyRn4PCBddrrJg-XqEqFsjO2gekAVniDwsXplHrrTYmEHhiCvtNwPzUn4OuX1rzFCmJBiFFclLRY7hWZ9TpcqiAUGEv9WK45U4lXI3QdC-WiDl95WeTsiBbfHGZOaF4z35otDzaX77HK6Nm5D0tnJATBGFOQ8aEiscuzBuS-I0vF2FDXhRZ1i85-l3fcccRtQXIWIesoOblPPBPa-7gc3ZdwCBAk_JMkPRA9AsSVds4y_A8ClJUim2QXqsUd37TG5G7EFOCHqn6Br3xmUky2SgR5vIddSxyx2eZ61uaFJR-Ds6TSETuCQvw","sec-ch-ua":"\"Not.A/Brand\";v=\"8\", \"Chromium\";v=\"114\", \"Google Chrome\";v=\"114\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"macOS\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"https:// kl.exampleB.io/auth/saml/captureUrlFragment?nextUrl=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"172.31.47.145","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36","referer":"https:// kl. exampleB .io/auth/saml/captureUrlFragment?nextUrl=%2F"},"res":{"statusCode":200,"responseTime":1,"contentLength":9},"message":"GET /auth/saml/captureUrlFragment.js 200 1ms - 9.0B"}
Error: failed parsing SAML config
    at SecurityClient.getSamlHeader (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:212:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/saml/routes.ts:78:30
    at Router.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:163:44)
    at handler (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:124:50)
    at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
    at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)
{"type":"log","@timestamp":"2023-06-14T09:55:30Z","tags":["error","plugins","securityDashboards"],"pid":453,"message":"Failed to get saml header: Error: Error: failed parsing SAML config"}
{"type":"error","@timestamp":"2023-06-14T09:55:30Z","tags":[],"pid":453,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:143:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:97:19)\n    at HapiResponseAdapter.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:92:17)\n    at Router.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:164:34)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at handler (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:124:50)\n    at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)\n    at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)"},"url":"http:// kl.exampleB.io/auth/saml/login?nextUrl=%2F&redirectHash=false","message":"Internal Server Error"}

Opensearch logs_server.json (clusterB)

{"type": "server", "timestamp": "2023-06-14T10:00:11,234Z", "level": "INFO", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_10: Next refresh cycle for metadata provider 'https:// myorg. okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata' will occur on '2023-06-14T10:01:11.234Z' ('2023-06-14T10:01:11.234Z' local time)", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA"  }
{"type": "server", "timestamp": "2023-06-14T10:00:14,544Z", "level": "ERROR", "component": "o.o.s.m.r.i.HTTPMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_4: Non-ok status code 404 returned from remote metadata source https:// myorg. okta-emea. com/app/exkaiuls0joQbD0YA0i7/sso/saml/metadata", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA"  }
{"type": "server", "timestamp": "2023-06-14T10:00:14,545Z", "level": "ERROR", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_4: Error occurred while attempting to refresh metadata from 'https:// myorg. okta-emea. com/app/exkaiuls0joQbD0YA0i7/sso/saml/metadata'", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA"  }
{"type": "server", "timestamp": "2023-06-14T10:00:14,545Z", "level": "INFO", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_4: Next refresh cycle for metadata provider 'https:// myorg.okta-emea.com/app/exkaiuls0joQbD0YA0i7/sso/saml/metadata' will occur on '2023-06-14T10:01:14.545Z' ('2023-06-14T10:01:14.545Z' local time)", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA"  }
{"type": "server", "timestamp": "2023-06-14T10:00:15,172Z", "level": "ERROR", "component": "o.o.s.m.r.i.HTTPMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_7: Non-ok status code 404 returned from remote metadata source https:// myorg.okta-emea.com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA"  }
{"type": "server", "timestamp": "2023-06-14T10:00:15,172Z", "level": "ERROR", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_7: Error occurred while attempting to refresh metadata from 'https:// myorg. okta-emea .com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata'", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA"  }
{"type": "server", "timestamp": "2023-06-14T10:00:15,172Z", "level": "INFO", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_7: Next refresh cycle for metadata provider 'https:// myorg. okta-emea .com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata' will occur on '2023-06-14T10:01:15.172Z' ('2023-06-14T10:01:15.172Z' local time)", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA"  }
{"type": "server", "timestamp": "2023-06-14T10:00:17,420Z", "level": "ERROR", "component": "o.o.s.m.r.i.HTTPMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_9: Non-ok status code 404 returned from remote metadata source https:// myorg. okta-emea.com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA"  }
{"type": "server", "timestamp": "2023-06-14T10:00:17,420Z", "level": "ERROR", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_9: Error occurred while attempting to refresh metadata from 'https:// myorg. okta-emea.com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata'", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA"  }
{"type": "server", "timestamp": "2023-06-14T10:00:17,420Z", "level": "INFO", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_9: Next refresh cycle for metadata provider 'https:// myorg. okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata' will occur on '2023-06-14T10:01:17.420Z' ('2023-06-14T10:01:17.420Z' local time)", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA"  }

logs.log: (clusterB)

[2023-06-14T10:00:46,559][ERROR][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_2: Error occurred while attempting to refresh metadata from 'https:// myorg. okta-emea.com/app/exkaiu3a25JguKMEj0i7/sso/saml/metadata'
[2023-06-14T10:00:46,559][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_2: Next refresh cycle for metadata provider 'https:// myorg. okta-emea .com/app/exkaiu3a25JguKMEj0i7/sso/saml/metadata' will occur on '2023-06-14T10:01:46.559Z' ('2023-06-14T10:01:46.559Z' local time)
[2023-06-14T10:00:50,161][ERROR][o.o.s.m.r.i.HTTPMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_5: Non-ok status code 404 returned from remote metadata source https:// myorg .okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata
[2023-06-14T10:00:50,161][ERROR][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_5: Error occurred while attempting to refresh metadata from 'https:// myorg. okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata'
[2023-06-14T10:00:50,161][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_5: Next refresh cycle for metadata provider 'https:// myorg. okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata' will occur on '2023-06-14T10:01:50.161Z' ('2023-06-14T10:01:50.161Z' local time)
[2023-06-14T10:01:01,682][ERROR][o.o.s.m.r.i.HTTPMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_3: Non-ok status code 404 returned from remote metadata source https:// myorg. okta-emea 
 .com/app/exkaiuls0joQbD0YA0i7/sso/saml/metadata
[2023-06-14T10:01:01,682][ERROR][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_3: Error occurred while attempting to refresh metadata from 'https:// myorg .okta-emea. com/app/exkaiuls0joQbD0YA0i7/sso/saml/metadata'
[2023-06-14T10:01:01,682][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_3: Next refresh cycle for metadata provider 'https:// myorg. okta-emea. com/app/exkaiuls0joQbD0YA0i7/sso/saml/metadata' will occur on '2023-06-14T10:02:01.682Z' ('2023-06-14T10:02:01.682Z' local time)
[2023-06-14T10:01:03,993][ERROR][o.o.s.m.r.i.HTTPMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_8: Non-ok status code 404 returned from remote metadata source https:// myorg. okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata
[2023-06-14T10:01:03,993][ERROR][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_8: Error occurred while attempting to refresh metadata from 'https:// myorg. okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata'
[2023-06-14T10:01:03,993][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_8: Next refresh cycle for metadata provider 'https:// myorg. okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata' will occur on '2023-06-14T10:02:03.993Z' ('2023-06-14T10:02:03.993Z' local time)

These are logs from one okta application with multitenancy configured

Hi @spliakos

Are you accessing both “Kibana_A” and “Kibana_B” from the same web browser ?

Could you please try either: 1) using a new “incognito” window, 2) clearing the browser cache, 3) using separate browsers / machines ?

Hi @Eugene7 , yes, I have tried using incognito and/or different browser, but every time from same machine. I have also tried clearing browser cache. I get the same 500 error in case of kl.exampleB.io while everything works as intended in case of kl.exampleA.io.
The errors I see in the logs are as above.

Hey @spliakos

Just chimming in, I seen this error.

 "http:// kl.exampleB.io/auth/saml/login?nextUrl=%2F&redirectHash=false","message":"Internal Server Error"}

I assume its from this configuration below.

I also seen this in your config.

transport_enabled: false

Question: Is that your fullpath?

plugins.security.ssl.transport.pemcert_filepath: tls/node.pem

Question2: Does this

plugins.security.nodes_dn:
  - 'CN=osearch-logs-*.stg.internal,OU=infra,O=Dino,C=ES'

Match this?

Hey @Gsmitt, sorry for late response.
To answer your questions now:

  • Question1: No this is not the full path, it’s the relative path from the installation directory.

  • Question2: No, this matches the Opensearch nodes internal domain hostnames.

The kibana_url field is named after the external zone URL, but essentially Opensearch dashboards is configured with

OPENSEARCH_HOSTS:                 https://osearchlogs.stg.internal:9200

which points to a LB which then routes the traffic to the internal nodes.
As per Opensearch documentation:
The security plugin needs to identify inter-cluster requests (i.e. requests between the nodes). so the plugins.security.nodes_dn: in that case covers the internal nodes addresses and yes, this is matched by the nodes_dn

Hi @spliakos

Please check if all OpenSearch nodes in the ClusterB can access this URL:
https:// myorg. okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata

Hello, @Eugene7. The appID is configured as exkahm4k463RUdobH0i7 in both okta and opensearch (clusters A&B) . With this ID from clusterB I get a response from all servers:

curl https://myorg.okta-emea.com/app/exkahm4k463RUdobH0i7/sso/saml/metadata

However, the errors I see in logs_server.json are strange because as you see here:

{"type": "server", "timestamp": "2023-07-05T11:47:30,207Z", "level": "INFO", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-c01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_8: Next refresh cycle for metadata provider 'https://myorg.okta-emea.com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata' will occur on '2023-07-05T11:48:30.207Z' ('2023-07-05T11:48:30.207Z' local time)", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "LQT1wzPfQ9WkAn_LvAFiVQ"  }
{"type": "server", "timestamp": "2023-07-05T11:47:31,935Z", "level": "ERROR", "component": "o.o.s.m.r.i.HTTPMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-c01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_4: Non-ok status code 404 returned from remote metadata source https://myorg.okta-emea.com/app/exkaiuls0joQbD0YA0i7/sso/saml/metadata", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "LQT1wzPfQ9WkAn_LvAFiVQ"  }
{"type": "server", "timestamp": "2023-07-05T11:47:31,936Z", "level": "ERROR", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-c01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_4: Error occurred while attempting to refresh metadata from 'https://myorg.okta-emea.com/app/exkaiuls0joQbD0YA0i7/sso/saml/metadata'", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "LQT1wzPfQ9WkAn_LvAFiVQ"  }
{"type": "server", "timestamp": "2023-07-05T11:47:31,936Z", "level": "INFO", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-c01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_4: Next refresh cycle for metadata provider 'https://myorg.okta-emea.com/app/exkaiuls0joQbD0YA0i7/sso/saml/metadata' will occur on '2023-07-05T11:48:31.936Z' ('2023-07-05T11:48:31.936Z' local time)", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "LQT1wzPfQ9WkAn_LvAFiVQ"  }

The URL you asked me about returns resource not found since there is no such resource. Not sure why this different appID is seen in logs_server.json. To confirm i just did a backup security config using a script i created that uses securityadmin.sh which confirms the configuration i have shared in the description.

This has been resolved finally. Thanks everyone for the suggestions and for reading

1 Like