Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch 2.7.0 (opensource)
Opensearch Dashboards 2.7.0
OS: Amazon Linux 2023
Browser: Version 114.0.5735.106 (Official Build) (arm64)
Describe the issue:
I have run into a peculiar and hard to explain issue. I have configured SSO with Okta for our first Opensearch cluster (clusterA) running on AWS account on EC2 instances and it works. The setup is as follows:
Opensearch is running on 3 nodes (3 ec2 instances), Opensearch Dashboards is running on a k8s pod. In the same fashion, our cluster B residing in a distinct AWS account (distinct VPC) with same configuration (besides domain) does not work with okta. I have tried creating distinct okta apps for Opensearch (Opensearch_A and Opensearch_B) as well as one okta application (Opensearch) with multitenancy.
When I try SP initiated login for clusterA ( kl. exampleA .io ) i get redirected to OKTA single sign on page, i sign in and I get transferred to opensearch dashboards page offering sign in using user/pass and SSO. I select SSO (also user/pass works) and I can get inside the Opensearch Dashboards with correct user mapping and so on. When I try the same for clusterB ( kl. exampleB. io ) i get redirected to OKTA single sign on page, i sign in and I get transferred to opensearch dashboards page offering sign in using user/pass and SSO. I select SSO and i get 500 internal error. What is peculiar is that in the second case i see the following URL: /auth/saml/login?nextUrl=%2F&redirectHash=false
I’ve tried a lot of things, like using single okta app with multitenancy, or 2 distinct ones for each domain, or just using the okta app for the working cluster and changing its configuration with clusterB. However, nothing works, and this leads me to believe that the problem should lie on the Opensearch side. However, as configuration is almost identical (as seen in the following section), I don’t know what could be the issue.
Configuration:
ClusterA (working):
opensearch.yml
# Security plugin configuration
plugins.security.ssl.transport.pemcert_filepath: tls/node.pem
plugins.security.ssl.transport.pemkey_filepath: tls/node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: tls/root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: tls/node.pem
plugins.security.ssl.http.pemkey_filepath: tls/node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: tls/root-ca.pem
plugins.security.nodes_dn:
- 'CN=osearch-logs-*.stg.internal,OU=infra,O=Dino,C=ES'
plugins.security.authcz.admin_dn:
- 'CN=dino.admin.opensearch,OU=infra,O=Dino,C=ES'
Opensearch-security config.yml
authc:
saml_auth_domain:
http_enabled: true
transport_enabled: false
order: 1
http_authenticator:
type: saml
challenge: true
config:
idp:
metadata_url: https:// example. okta-emea. com/app/exkahm4k463RUdobH0i7/sso/saml/metadata #SAML's metadata url, provided by your IdP
entity_id: http:// www. okta. com/exkahm4k463RUdobH0i7 #SAML's IdP entity ID, provided by your IdP
sp:
entity_id: opensearch-dashboards-saml
kibana_url: https:// kl. exampleA. io/
roles_key: Role
exchange_key: "b7d5d3fe26c0d1d8c630bf6401904e3f9c388dde1c4b8479f4a85961fce94f01"
authentication_backend:
type: noop
basic_internal_auth_domain:
description: "Authenticate via HTTP Basic against internal users database"
http_enabled: true
transport_enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
opensearch_dashboards.yml
opensearch.hosts: [https:// localhost: 9200]
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: kibanaserver
opensearch.requestHeadersWhitelist: [authorization, securitytenant]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
# Use this setting if you are running opensearch-dashboards without https
opensearch_security.cookie.secure: false
server.host: '0.0.0.0'
opensearch_security.auth.type: "saml"
opensearch_security.auth.multiple_auth_enabled: false
server.xsrf.allowlist: ["/_plugins/_security/saml/acs/idpinitiated", "/_plugins/_security/saml/acs", "/_plugins/_security/saml/logout", "/_opendistro/_security/saml/acs", "/_opendistro/_security/saml/acs/idpinitiated", "/_opendistro/_security/saml/logout"]
ClusterB (not working):
opensearch.yml
# Security plugin configuration
plugins.security.ssl.transport.pemcert_filepath: tls/node.pem
plugins.security.ssl.transport.pemkey_filepath: tls/node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: tls/root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: tls/node.pem
plugins.security.ssl.http.pemkey_filepath: tls/node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: tls/root-ca.pem
plugins.security.nodes_dn:
- 'CN=osearch-logs-*.prod.internal,OU=infra,O=Dino,C=ES'
plugins.security.authcz.admin_dn:
- 'CN=dino.admin.opensearch,OU=infra,O=Dino,C=ES'
Opensearch-security config.yml
authc:
saml_auth_domain:
http_enabled: true
transport_enabled: false
order: 1
http_authenticator:
type: saml
challenge: true
config:
idp:
metadata_url: https:// example. okta-emea. com/app/exkahm4k463RUdobH0i7/sso/saml/metadata #SAML's metadata url, provided by your IdP
entity_id: http:// www. okta. com/exkahm4k463RUdobH0i7 #SAML's IdP entity ID, provided by your IdP
sp:
entity_id: opensearch-dashboards-saml
kibana_url: https:// kl. exampleB. io/
roles_key: Role
exchange_key: "b7d5d3fe26c0d1d8c630bf6401904e3f9c388dde1c4b8479f4a85961fce94f01"
authentication_backend:
type: noop
basic_internal_auth_domain:
description: "Authenticate via HTTP Basic against internal users database"
http_enabled: true
transport_enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: intern
opensearch_dashboards.yml
opensearch.hosts: [https:// localhost: 9200]
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: obscured
opensearch.requestHeadersWhitelist: [authorization, securitytenant]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
# Use this setting if you are running opensearch-dashboards without https
opensearch_security.cookie.secure: false
server.host: '0.0.0.0'
opensearch_security.auth.type: "saml"
opensearch_security.auth.multiple_auth_enabled: false
server.xsrf.allowlist: ["/_plugins/_security/saml/acs/idpinitiated", "/_plugins/_security/saml/acs", "/_plugins/_security/saml/logout", "/_opendistro/_security/saml/acs", "/_opendistro/_security/saml/acs/idpinitiated", "/_opendistro/_security/saml/logout"]
The configuration of the okta app that works is as follows. Note that this works only for the exampleA domain, while exampleB does not redirect:
Single Sign On URL: https:// kl. exampleA .io/_opendistro/_security/saml/acs
Requestable SSO URLs URLIndex:
https:// kl. exampleA. io/_opendistro/_security/saml/acs 0
https:// kl. exampleB. io/_opendistro/_security/saml/acs 1
Recipient URL: https:// kl. exampleA. io/_opendistro/_security/saml/acs
Destination URL: https:// kl. exampleA. io/_opendistro/_security/saml/acs
Audience Restriction: opensearch-dashboards-saml
Default Relay:
StateName ID Format: Unspecified
Response: Signed
Assertion Signature: Signed
Signature Algorithm: RSA_SHA256
Digest Algorithm: SHA256Assertion
Encryption: Unencrypted
SAML Single Logout: Disabled
SAML Signed Request: Disabled
authnContextClassRef: PasswordProtectedTransport
Honor Force Authentication: Yes
Assertion Inline Hook: None (disabled)
SAML Issuer ID: http:// www. okta. com/${org.externalKey}
ATTRIBUTE STATEMENTS
Name Name Format Value
GROUP ATTRIBUTE STATEMENTS
Name Name Format Filter
Role Unspecified Starts with: okta_opensearch_
Metadata URL: https:// myorg. okta-emea. com/app/exkahm4k463RUdobH0i7/sso/saml/metadata
Sign on URL: https:// myorg. okta-emea. com/app/myorg_opensearch_4/exkahm4k463RUdobH0i7/sso/saml
Issuer: http:// www. okta. com/exkahm4k463RUdobH0i7
Relevant Logs or Screenshots:
opensearch dashboards logs (clusterB):
{"type":"response","@timestamp":"2023-06-14T09:55:30Z","tags":[],"pid":453,"method":"get","statusCode":200,"req":{"url":"/auth/saml/captureUrlFragment.js","method":"get","headers":{"x-forwarded-for":"185.49.168.193","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kl. exampleB. io","x-amzn-trace-id":"Root=1-64898e92-7de0e472756e2e9868d71500","x-amzn-oidc-data":"eyJ0eXAiOiJKV1QiLCJraWQiOiI3ZGYzZGU4Mi1kMjBkLTQ1Y2EtOGE0NC00YWE1ODAyMzYzYzYiLCJhbGciOiJFUzI1NiIsImlzcyI6Imh0dHBzOi8vZGlub3RlY2gub2t0YS1lbWVhLmNvbSIsImNsaWVudCI6IjBvYTJ4Y2dpYXpUNjFOWGt1MGk3Iiwic2lnbmVyIjoiYXJuOmF3czplbGFzdGljbG9hZGJhbGFuY2luZzpldS13ZXN0LTE6MzE2Mzk5ODM4MzA5OmxvYWRiYWxhbmNlci9hcHAvNDExMGMyYTYtZGVmYXVsdC1vYXV0aGdhdGUtOGUwMS81Y2EyMzlmNTY3NmM0MzNjIiwiZXhwIjoxNjg2NzM2NjUwfQ==.eyJzdWIiOiIwMHU0aGE3dGlzcklHMzBtbjBpNyIsImV4cCI6MTY4NjczNjY1MCwiaXNzIjoiaHR0cHM6Ly9kaW5vdGVjaC5va3RhLWVtZWEuY29tIn0=.VRcJ0uXOlxPwp2q4PfVgjYiiDTwKYPr7WIVfV1yN6saWjrY5kwIBYep8mIbeRy9MjKPeuy_8zThxvv3Ggovezg==","x-amzn-oidc-identity":"00u4ha7tisrIG30mn0i7","x-amzn-oidc-accesstoken":"eyJraWQiOiI2clFNMjVYcGFZV29BUEMtWUNySGJtVWVhNEN2bjNDOUdiZkRiX3d4bHlBIiwiYWxnIjoiUlMyNTYifQ.eyJ2ZXIiOjEsImp0aSI6IkFULjNnVDJIenhQVGVJVEFhUU96WS0wSnRpbHJ0ZUVDTzZoZ1hqSXFBMTc1LVEiLCJpc3MiOiJodHRwczovL2Rpbm90ZWNoLm9rdGEtZW1lYS5jb20iLCJhdWQiOiJodHRwczovL2Rpbm90ZWNoLm9rdGEtZW1lYS5jb20iLCJzdWIiOiJzdGVmYW5vcy5wbGlha29zQGRpbm90ZWNoLmNvbSIsImlhdCI6MTY4NjY2MDg3NCwiZXhwIjoxNjg2NjY0NDc0LCJjaWQiOiIwb2EyeGNnaWF6VDYxTlhrdTBpNyIsInVpZCI6IjAwdTRoYTd0aXNySUczMG1uMGk3Iiwic2NwIjpbIm9wZW5pZCJdLCJhdXRoX3RpbWUiOjE2ODY2NjA0Mzl9.QGNSCNpBJ8y4ncb8c-cAlWwrshL4zKllyRn4PCBddrrJg-XqEqFsjO2gekAVniDwsXplHrrTYmEHhiCvtNwPzUn4OuX1rzFCmJBiFFclLRY7hWZ9TpcqiAUGEv9WK45U4lXI3QdC-WiDl95WeTsiBbfHGZOaF4z35otDzaX77HK6Nm5D0tnJATBGFOQ8aEiscuzBuS-I0vF2FDXhRZ1i85-l3fcccRtQXIWIesoOblPPBPa-7gc3ZdwCBAk_JMkPRA9AsSVds4y_A8ClJUim2QXqsUd37TG5G7EFOCHqn6Br3xmUky2SgR5vIddSxyx2eZ61uaFJR-Ds6TSETuCQvw","sec-ch-ua":"\"Not.A/Brand\";v=\"8\", \"Chromium\";v=\"114\", \"Google Chrome\";v=\"114\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"macOS\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"script","referer":"https:// kl.exampleB.io/auth/saml/captureUrlFragment?nextUrl=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"172.31.47.145","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36","referer":"https:// kl. exampleB .io/auth/saml/captureUrlFragment?nextUrl=%2F"},"res":{"statusCode":200,"responseTime":1,"contentLength":9},"message":"GET /auth/saml/captureUrlFragment.js 200 1ms - 9.0B"}
Error: failed parsing SAML config
at SecurityClient.getSamlHeader (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/backend/opensearch_security_client.ts:212:15)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/saml/routes.ts:78:30
at Router.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:163:44)
at handler (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:124:50)
at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
at Object.internals.handler (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)
at exports.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)
at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)
{"type":"log","@timestamp":"2023-06-14T09:55:30Z","tags":["error","plugins","securityDashboards"],"pid":453,"message":"Failed to get saml header: Error: Error: failed parsing SAML config"}
{"type":"error","@timestamp":"2023-06-14T09:55:30Z","tags":[],"pid":453,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n at HapiResponseAdapter.toError (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:143:19)\n at HapiResponseAdapter.toHapiResponse (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:97:19)\n at HapiResponseAdapter.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:92:17)\n at Router.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:164:34)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at handler (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:124:50)\n at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n at Object.internals.handler (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)\n at exports.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)\n at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)\n at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)"},"url":"http:// kl.exampleB.io/auth/saml/login?nextUrl=%2F&redirectHash=false","message":"Internal Server Error"}
Opensearch logs_server.json (clusterB)
{"type": "server", "timestamp": "2023-06-14T10:00:11,234Z", "level": "INFO", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_10: Next refresh cycle for metadata provider 'https:// myorg. okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata' will occur on '2023-06-14T10:01:11.234Z' ('2023-06-14T10:01:11.234Z' local time)", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA" }
{"type": "server", "timestamp": "2023-06-14T10:00:14,544Z", "level": "ERROR", "component": "o.o.s.m.r.i.HTTPMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_4: Non-ok status code 404 returned from remote metadata source https:// myorg. okta-emea. com/app/exkaiuls0joQbD0YA0i7/sso/saml/metadata", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA" }
{"type": "server", "timestamp": "2023-06-14T10:00:14,545Z", "level": "ERROR", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_4: Error occurred while attempting to refresh metadata from 'https:// myorg. okta-emea. com/app/exkaiuls0joQbD0YA0i7/sso/saml/metadata'", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA" }
{"type": "server", "timestamp": "2023-06-14T10:00:14,545Z", "level": "INFO", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_4: Next refresh cycle for metadata provider 'https:// myorg.okta-emea.com/app/exkaiuls0joQbD0YA0i7/sso/saml/metadata' will occur on '2023-06-14T10:01:14.545Z' ('2023-06-14T10:01:14.545Z' local time)", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA" }
{"type": "server", "timestamp": "2023-06-14T10:00:15,172Z", "level": "ERROR", "component": "o.o.s.m.r.i.HTTPMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_7: Non-ok status code 404 returned from remote metadata source https:// myorg.okta-emea.com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA" }
{"type": "server", "timestamp": "2023-06-14T10:00:15,172Z", "level": "ERROR", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_7: Error occurred while attempting to refresh metadata from 'https:// myorg. okta-emea .com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata'", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA" }
{"type": "server", "timestamp": "2023-06-14T10:00:15,172Z", "level": "INFO", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_7: Next refresh cycle for metadata provider 'https:// myorg. okta-emea .com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata' will occur on '2023-06-14T10:01:15.172Z' ('2023-06-14T10:01:15.172Z' local time)", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA" }
{"type": "server", "timestamp": "2023-06-14T10:00:17,420Z", "level": "ERROR", "component": "o.o.s.m.r.i.HTTPMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_9: Non-ok status code 404 returned from remote metadata source https:// myorg. okta-emea.com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA" }
{"type": "server", "timestamp": "2023-06-14T10:00:17,420Z", "level": "ERROR", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_9: Error occurred while attempting to refresh metadata from 'https:// myorg. okta-emea.com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata'", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA" }
{"type": "server", "timestamp": "2023-06-14T10:00:17,420Z", "level": "INFO", "component": "o.o.s.m.r.i.AbstractReloadingMetadataResolver", "cluster.name": "logs", "node.name": "osearch-logs-a01.prod.internal", "message": "Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_9: Next refresh cycle for metadata provider 'https:// myorg. okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata' will occur on '2023-06-14T10:01:17.420Z' ('2023-06-14T10:01:17.420Z' local time)", "cluster.uuid": "PvySMbfaTTaE83GpiGBQVA", "node.id": "nhd0vXmPRDiN3un5NkTOVA" }
logs.log: (clusterB)
[2023-06-14T10:00:46,559][ERROR][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_2: Error occurred while attempting to refresh metadata from 'https:// myorg. okta-emea.com/app/exkaiu3a25JguKMEj0i7/sso/saml/metadata'
[2023-06-14T10:00:46,559][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_2: Next refresh cycle for metadata provider 'https:// myorg. okta-emea .com/app/exkaiu3a25JguKMEj0i7/sso/saml/metadata' will occur on '2023-06-14T10:01:46.559Z' ('2023-06-14T10:01:46.559Z' local time)
[2023-06-14T10:00:50,161][ERROR][o.o.s.m.r.i.HTTPMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_5: Non-ok status code 404 returned from remote metadata source https:// myorg .okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata
[2023-06-14T10:00:50,161][ERROR][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_5: Error occurred while attempting to refresh metadata from 'https:// myorg. okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata'
[2023-06-14T10:00:50,161][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_5: Next refresh cycle for metadata provider 'https:// myorg. okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata' will occur on '2023-06-14T10:01:50.161Z' ('2023-06-14T10:01:50.161Z' local time)
[2023-06-14T10:01:01,682][ERROR][o.o.s.m.r.i.HTTPMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_3: Non-ok status code 404 returned from remote metadata source https:// myorg. okta-emea
.com/app/exkaiuls0joQbD0YA0i7/sso/saml/metadata
[2023-06-14T10:01:01,682][ERROR][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_3: Error occurred while attempting to refresh metadata from 'https:// myorg .okta-emea. com/app/exkaiuls0joQbD0YA0i7/sso/saml/metadata'
[2023-06-14T10:01:01,682][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_3: Next refresh cycle for metadata provider 'https:// myorg. okta-emea. com/app/exkaiuls0joQbD0YA0i7/sso/saml/metadata' will occur on '2023-06-14T10:02:01.682Z' ('2023-06-14T10:02:01.682Z' local time)
[2023-06-14T10:01:03,993][ERROR][o.o.s.m.r.i.HTTPMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_8: Non-ok status code 404 returned from remote metadata source https:// myorg. okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata
[2023-06-14T10:01:03,993][ERROR][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_8: Error occurred while attempting to refresh metadata from 'https:// myorg. okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata'
[2023-06-14T10:01:03,993][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [osearch-logs-a01.prod.internal] Metadata Resolver SamlHTTPMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_8: Next refresh cycle for metadata provider 'https:// myorg. okta-emea. com/app/exkaivlazlZS26XUg0i7/sso/saml/metadata' will occur on '2023-06-14T10:02:03.993Z' ('2023-06-14T10:02:03.993Z' local time)
These are logs from one okta application with multitenancy configured