XFF not used from Dashboards for Dev Tools / Console

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 2.5.0 and 2.10.0

Describe the issue:
We are using proxy authentication with opensearch (OS) and opensearch dashboards (OSD) using x-forwarded-for, x-proxy-user and x-proxy-roles. For the most part this works well however, requests originating from the Dev Tools Console.
We have a gateway proxy in front of OSD that is providing the x-forwarded-for, x-proxy-user and x-proxy-roles headers. We have a man-in-the-middle gateway between OSD and OS in which we are able to log the headers being passed between the two systems. We have observed the traffic between the two when in the “Discover” tab (which works) and the “Dev Tools Console” (which does not). In neither case is the x-forwarded-for header sent between the two systems. However, in the case of Discover we see this log message log.trace("xff resolved {} to {}", remoteAddress, isa); whereas in the case of Dev Tools we see this log message log.trace("no xff done for {}", request.getClass());.
Additionally, even though the instructions on this page state to configure the x-forwarded-for header as a requestHeadersAllowlist, if we include the header in the list, proxy authentication does not seem to work at all. If we remove it, we get the behavior listed above.
Finally, if we reconfigure the entire system to use a custom header (ap-forwarded-for) it works as expected for all calls from OSD.

Configuration:
OS 2.5.0 & OSD 2.5.0

Hi @winddancerjohn,

Could you please share your config files:

config.yml
opensearch_dashboards.yml

Could you also clarify the OpenSearch version: are the tests performed on 2.5.0 or 2.10.0 (both?)?

Thanks,
Mantas

Tested with both 2.5.0 and 2.10.0.

config.yml

---

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
    # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
    # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
    #filtered_alias_mode: warn
    do_not_fail_on_forbidden: true
    #kibana:
    # Kibana multitenancy
    #multitenancy_enabled: true
    #server_username: kibanaserver
    #index: '.kibana'
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: true
        # internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
        internalProxies: '.*' # trust all internal proxies, regex pattern
        ### The below update is needed if doing proxy authentication with Dev Tools / Console. The default
        ### `x-forwarded-for` works for the rest of dashboards but not Dev Tools / Console. Make sure to
        ### update Dashboards to forward this header and update the AddProxyHeaders filter to use this header.
        # remoteIpHeader:  'ap-forwarded-for'
        
    authc:
 
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 4
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
      proxy_auth_domain:
        description: "Authenticate via proxy"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
        authentication_backend:
          type: noop

      clientcert_auth_domain:
        description: "Authenticate via SSL client certificates"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: clientcert
          config:
            username_attribute: cn #optional, if omitted DN becomes username
          challenge: false
        authentication_backend:
          type: noop
  

opensearch-dashboards

server:
  name: os_dashboards
  host: '0.0.0.0'
  basePath: "/dashboards"
  rewriteBasePath: true
  ssl:
    enabled: true
    certificate: "/usr/share/opensearch-dashboards/config/identity.pem"
    key: "/usr/share/opensearch-dashboards/config/identity.key.pem"

opensearch:
  logQueries: true
  username: "admin"
  password: "admin"
  ssl:
    verificationMode: full

    certificate: "/usr/share/opensearch-dashboards/config/identity.pem"
    key: "/usr/share/opensearch-dashboards/config/identity.key.pem"
    certificateAuthorities: ["/usr/share/opensearch-dashboards/config/cacerts"]

logging.verbose: false

# Disable Dev Tools access
console.enabled: false

opensearch.hosts: ${AUDIT_GATEWAY_URL}
opensearch.ssl.verificationMode: none

opensearch_security.cookie.secure: true
opensearch_security.session.keepalive: false
opensearch_security.auth.type: proxy
opensearch_security.proxycache.user_header: x-proxy-user
opensearch_security.proxycache.roles_header: x-proxy-roles
opensearch_security.readonly_mode.roles: [kibana_read_only]

#opensearch.requestHeadersWhitelist: [authorization, securitytenant, session-id, user-selected-authorities, classification, justification, x-ap-subject-dn, x-ap-issuer-dn, ssl-client-subject-dn,x-proxy-user,x-proxy-roles]
opensearch.requestHeadersAllowlist: ["securitytenant","Authorization", "ap-forwarded-for","x-proxy-user","x-proxy-roles", "session-id", "ssl-client-subject-dn", "ssl-client-issuer-dn", "X-ProxiedEntitiesChain","X-ProxiedIssuersChain"]

Do you see any error messages when x-forwarded-for is added to requestHeadersAllowlist ??

We see “this” exception being thrown.

We also DO NOT see this logging:

Hi @winddancerjohn,

I have extensively tested proxy authentication in my lab using x-forwarded-for, x-proxy-user and x-proxy-roles in all different scenarios, and I am not able to reproduce the same behaviour (note: proxy only in front of OSD and no proxy between OSD and OS)
If the x-forwarded-for is not in the header or/and is not in opensearch.requestHeadersAllowlist: the authentication fails with 401 as expected. Then the x-forwarded-for is configured and sent properly the UI and Dev Tools/Console behaves as per normal.

Could you test your set-up without the “man-in-the-middle gateway between OSD and OS” (with the x-forwarded-for in opensearch.requestHeadersAllowlist:)?

Best,
mj

Thank you for suggesting that test. You are correct, when I removed the man-in-the-middle, the dashboards config worked as advertised.