AD FS auth not working

Hi,

I’m having an on-prem OpenSearch + Dashboards cluster and I’m unable to make it work with our (also on-prem) AD FS.
I have it working on OpenDistro, but for some reason it’s not working with opensearch. Over the Dashboards interface I get

{
statusCode: 500,
error: "Internal Server Error",
message: "Internal Error",
}
_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
    # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
    # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
    #filtered_alias_mode: warn
    #do_not_fail_on_forbidden: false
    #kibana:
    # Kibana multitenancy
    #multitenancy_enabled: true
    #server_username: kibanaserver
    #index: '.kibana'
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
        #internalProxies: '.*' # trust all internal proxies, regex pattern
        #remoteIpHeader:  'x-forwarded-for'
        ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
        ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
        ###### and here https://tools.ietf.org/html/rfc7239
        ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern
      saml_auth:
        http_enabled: true
        transport_enabled: false
        order: 1
        http_authenticator:
          type: saml
          challenge: true
          config:
            idp:
              #enable_ssl: true
              verify_hostnames: false
              metadata_file: FederationMetadata.xml
              entity_id: http://sso.DOMAIN.com/adfs/services/trust
            sp:
              entity_id: https://xxxxxx.DOMAIN.com/
              force_authn: true
            kibana_url: https://xxxxxx.DOMAIN.com
            subject_key: NameID
            roles_key: Roles
            exchange_key: 515346422bdb16e089cf98671ebdb6ee6dcf0a9508a0c1fb31f8992a3c8ab943
        authentication_backend:
          type: noop

This is the only thing that shows in logs:

May  6 11:23:05 demovm opensearch-dashboards[4110246]: { Error: Authentication Exception
May  6 11:23:05 demovm opensearch-dashboards[4110246]:     at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
May  6 11:23:05 demovm opensearch-dashboards[4110246]:     at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
May  6 11:23:05 demovm opensearch-dashboards[4110246]:     at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
May  6 11:23:05 demovm opensearch-dashboards[4110246]:     at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
May  6 11:23:05 demovm opensearch-dashboards[4110246]:     at IncomingMessage.emit (events.js:203:15)
May  6 11:23:05 demovm opensearch-dashboards[4110246]:     at endReadableNT (_stream_readable.js:1145:12)
May  6 11:23:05 demovm opensearch-dashboards[4110246]:     at process._tickCallback (internal/process/next_tick.js:63:19)
May  6 11:23:05 demovm opensearch-dashboards[4110246]:   status: 401,
May  6 11:23:05 demovm opensearch-dashboards[4110246]:   displayName: 'AuthenticationException',
May  6 11:23:05 demovm opensearch-dashboards[4110246]:   message: 'Authentication Exception',
May  6 11:23:05 demovm opensearch-dashboards[4110246]:   path: '/_plugins/_security/api/authtoken',
May  6 11:23:05 demovm opensearch-dashboards[4110246]:   query: {},
May  6 11:23:05 demovm opensearch-dashboards[4110246]:   body: 'Authentication finally failed',
May  6 11:23:05 demovm opensearch-dashboards[4110246]:   statusCode: 401,
May  6 11:23:05 demovm opensearch-dashboards[4110246]:   response: 'Authentication finally failed',
May  6 11:23:05 demovm opensearch-dashboards[4110246]:   toString: [Function],
May  6 11:23:05 demovm opensearch-dashboards[4110246]:   toJSON: [Function],
May  6 11:23:05 demovm opensearch-dashboards[4110246]:   isBoom: true,
May  6 11:23:05 demovm opensearch-dashboards[4110246]:   isServer: false,
May  6 11:23:05 demovm opensearch-dashboards[4110246]:   data: null,
May  6 11:23:05 demovm opensearch-dashboards[4110246]:   output:
May  6 11:23:05 demovm opensearch-dashboards[4110246]:    { statusCode: 401,
May  6 11:23:05 demovm opensearch-dashboards[4110246]:      payload:
May  6 11:23:05 demovm opensearch-dashboards[4110246]:       { statusCode: 401,
May  6 11:23:05 demovm opensearch-dashboards[4110246]:         error: 'Unauthorized',
May  6 11:23:05 demovm opensearch-dashboards[4110246]:         message: 'Authentication Exception' },
May  6 11:23:05 demovm opensearch-dashboards[4110246]:      headers:
May  6 11:23:05 demovm opensearch-dashboards[4110246]:       { 'WWW-Authenticate': 'Basic realm="Authorization Required"' } },
May  6 11:23:05 demovm opensearch-dashboards[4110246]:   reformat: [Function],
May  6 11:23:05 demovm opensearch-dashboards[4110246]:   [Symbol(OpenSearchError)]: 'OpenSearch/notAuthorized' }
May  6 11:23:05 demovm opensearch-dashboards[4110246]: {"type":"log","@timestamp":"2022-05-06T11:23:05Z","tags":["error","plugins","securityDashboards"],"pid":4110246,"message":"SAML IDP initiated authentication workflow failed: Error: failed to get token"}
May  6 11:23:05 demovm opensearch-dashboards[4110246]: {"type":"error","@timestamp":"2022-05-06T11:23:05Z","tags":[],"pid":4110246,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:145:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:99:19)\n    at HapiResponseAdapter.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:94:17)\n    at Router.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:164:34)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/_opendistro/_security/saml/acs/idpinitiated","path":"/_opendistro/_security/saml/acs/idpinitiated","href":"/_opendistro/_security/saml/acs/idpinitiated"},"message":"Internal Server Error"}

AD FS has the following settings:

Endpoit URL: https://xxxxxx.DOMAIN.com/_opendistro/_security/saml/acs/idpinitiated

I’ve also compared the SAML response for the OpenDisto endpoint with the working one of Opensearch and they are the same in terms of elements / structure.

Any ideas what would be the reason?

Best regards,
Vlad

@fuzzy_one Could you try commenting out force_authn:true and subject_key: NameID?

@pablo Thanks for the suggestion. I’ve made the changes, applied them with securityadmin.sh, restarted the cluster, but the result is the same
SAML IDP initiated authentication workflow failed: Error: failed to get token

@fuzzy_one Did you notice any errors in the logs when uploading the configuration?
Is the FederationMetadata.xml up to date? You have to update this file every time you make changes in the SAML configuration in ADFS.

Have you tried to use metadata_url: instead?

@pablo The config push completes with no errors

Security Admin v7
Will connect to 172.17.16.153:9300 ... done
Connected as CN=admin.XXXXXXXXXX.com,OU=Ops,O=XXXXXX.com\, Inc.,DC=XXXXXXX.com
OpenSearch Version: 1.3.1
OpenSearch Security Version: 1.3.1.0
Contacting opensearch cluster 'opensearch' ...
Clustername: wazuh
Clusterstate: GREEN
Number of nodes: 4
Number of data nodes: 2
.opendistro_security index already exists, so we do not need to create one.
Populate config from /usr/share/opensearch/plugins/opensearch-security/securityconfig/
Will update '_doc/config' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml
   SUCC: Configuration for 'config' created or updated
Will update '_doc/roles' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update '_doc/rolesmapping' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '_doc/internalusers' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '_doc/actiongroups' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '_doc/tenants' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Will update '_doc/nodesdn' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml
   SUCC: Configuration for 'nodesdn' created or updated
Will update '_doc/whitelist' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml
   SUCC: Configuration for 'whitelist' created or updated
Will update '_doc/audit' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/audit.yml
   SUCC: Configuration for 'audit' created or updated
Done with success

The FederationMetadata.xml is up to date - I’ve just downloaded the file. I did not try with metadata_url, since I had no issues using the xml file with opendistro.

Thanks,
Vlad

I eventually found out that the FederationMetadata.xml was in the wrong location. I fixed that and now I get the following in the dashboard log

{"type":"log","@timestamp":"2022-05-08T20:00:59Z","tags":["error","plugins","securityDashboards"],"pid":1782270,"message":"SAML IDP initiated authentication workflow failed: SyntaxError: Unexpected token v in JSON at position 60"}

In opensearch I get what looks like the correct SAML response and the following warning:

 No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'

This is even more cryptic :slight_smile:

Issue SOLVED! The last error Unexpected token v in JSON at position 60 was actually a bit hard to investigate, and the reason was a very strange one.
I managed to solve it by changing the claims policy in ADFS from Windows Account Name to UPN. The v token in the error was cause by how the SAML JSON response is decoded. With the original Windows Account Name the sub was DOMAIN\\vpetres (and here is where the v came from). Now with UPN the sub is vpetres@DOMAIN.
Hope this helps others having the same issue.

Thank you,
Vlad

@fuzzy_one The authentication should be still successful with <domain_name>\<username> format and Windows Account Name set in the claim.

I know it should, but for some reason it was returned as <domain>\\<username> and something flipped dashboards into thinking the JSON was not correctly formatted and it could not extract the value. I’ll investigate further in a few days and open up a bug report if needed.

Thanks,
Vlad

1 Like

Hello Guys,
I am struggling with the same with OpenSearch 1.2, having the same issue:

Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: {"type":"error","@timestamp":"2022-12-06T03:03:54Z","tags":["connection","client","error"],"pid":9410,"level":"error","error":{"message":"140539448248192:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46\n","name":"Error","stack":"Error: 140539448248192:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46\n"},"message":"140539448248192:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46\n"}
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: { Error: Authentication Exception
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: at respond (/usr/share/wazuh-dashboard/node_modules/elasticsearch/src/lib/transport.js:349:15)
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: at checkRespForFailure (/usr/share/wazuh-dashboard/node_modules/elasticsearch/src/lib/transport.js:306:7)
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: at HttpConnector.<anonymous> (/usr/share/wazuh-dashboard/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: at IncomingMessage.wrapper (/usr/share/wazuh-dashboard/node_modules/lodash/lodash.js:4991:19)
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: at IncomingMessage.emit (events.js:203:15)
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: at endReadableNT (_stream_readable.js:1145:12)
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: at process._tickCallback (internal/process/next_tick.js:63:19)
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: status: 401,
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: displayName: 'AuthenticationException',
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: message: 'Authentication Exception',
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: path: '/_plugins/_security/api/authtoken',
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: query: {},
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: body: 'Unauthorized',
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: statusCode: 401,
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: response: 'Unauthorized',
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: wwwAuthenticateDirective: 'Basic realm="OpenSearch Security"',
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: toString: [Function],
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: toJSON: [Function],
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: isBoom: true,
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: isServer: false,
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: data: null,
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: output:
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: { statusCode: 401,
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: payload:
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: { statusCode: 401,
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: error: 'Unauthorized',
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: message: 'Authentication Exception' },
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: headers:
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: { 'WWW-Authenticate': 'Basic realm="Authorization Required"' } },
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: reformat: [Function],
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: [Symbol(OpenSearchError)]: 'OpenSearch/notAuthorized' }
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: {"type":"log","@timestamp":"2022-12-06T03:03:54Z","tags":["error","plugins","securityDashboards"],"pid":9410,"message":"SAML IDP initiated authentication workflow failed: Error: failed to get token"}
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: {"type":"error","@timestamp":"2022-12-06T03:03:54Z","tags":[],"pid":9410,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/wazuh-dashboard/src/core/server/http/router/response_adapter.js:145:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/wazuh-dashboard/src/core/server/http/router/response_adapter.js:99:19)\n    at HapiResponseAdapter.handle (/usr/share/wazuh-dashboard/src/core/server/http/router/response_adapter.js:94:17)\n    at Router.handle (/usr/share/wazuh-dashboard/src/core/server/http/router/router.js:164:34)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/_opendistro/_security/saml/acs/idpinitiated","path":"/_opendistro/_security/saml/acs/idpinitiated","href":"/_opendistro/_security/saml/acs/idpinitiated"},"message":"Internal Server Error"}
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: {"type":"response","@timestamp":"2022-12-06T03:03:54Z","tags":[],"pid":9410,"method":"post","statusCode":500,"req":{"url":"/_opendistro/_security/saml/acs/idpinitiated","method":"post","headers":{"host":"10.10.10.115","connection":"keep-alive","content-length":"6575","cache-control":"max-age=0","sec-ch-ua":"\"Google Chrome\";v=\"107\", \"Chromium\";v=\"107\", \"Not=A?Brand\";v=\"24\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","upgrade-insecure-requests":"1","origin":"https://naboo.galaxy.com","content-type":"application/x-www-form-urlencoded","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"cross-site","sec-fetch-mode":"navigate","sec-fetch-dest":"document","referer":"https://naboo.galaxy.com/","accept-encoding":"gzip, deflate, br","accept-language":"en"},"remoteAddress":"192.168.0.171","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36","referer":"https://naboo.galaxy.com/"},"res":{"statusCode":500,"responseTime":40,"contentLength":9},"message":"POST /_opendistro/_security/saml/acs/idpinitiated 500 40ms - 9.0B"}
Dec 06 03:03:54 wazuh-server opensearch-dashboards[9410]: {"type":"response","@timestamp":"2022-12-06T03:03:54Z","tags":[],"pid":9410,"method":"get","statusCode":401,"req":{"url":"/favicon.ico","method":"get","headers":{"host":"10.10.10.115","connection":"keep-alive","sec-ch-ua":"\"Google Chrome\";v=\"107\", \"Chromium\";v=\"107\", \"Not=A?Brand\";v=\"24\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"image","referer":"https://10.10.10.115/_opendistro/_security/saml/acs/idpinitiated","accept-encoding":"gzip, deflate, br","accept-language":"en"},"remoteAddress":"192.168.0.171","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36","referer":"https://10.10.10.115/_opendistro/_security/saml/acs/idpinitiated"},"res":{"statusCode":401,"responseTime":4,"contentLength":9},"message":"GET /favicon.ico 401 4ms - 9.0B"}

It seems it is loading correctly my xml:

[2022-12-06T11:54:46,098][INFO ][o.o.s.m.r.i.AbstractReloadingMetadataResolver] [node-1] Metadata Resolver SamlFilesystemMetadataResolver com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator_2: Metadata from '/usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/FederationMetadata.xml' has not changed since last refresh

I have followed the instructions here: Add Single Sign-On to Open Distro for Elasticsearch Kibana Using SAML and ADFS | AWS Open Source Blog adapting them to OpenSearch

Do you have any suggestion?

@Lord_Zetas Would you mind opening a new case and sharing your config.yml file?

1 Like

Hello @pablo
Here is the new post, thank you: SSO Integration with ADFS