Is X-Forwarded-For used when validating proxy traffic?

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch 2.12.0
OpenSearch Dashboards 2.12.0

Describe the issue:

I am confused about how, or if, the X-Forwarded-For header is used when validating that traffic from a proxy is trusted.

The proxy documentation says:

To determine whether a request comes from a trusted internal proxy, the Security plugin compares the remote address of the HTTP request with the list of configured internal proxies. If the remote address is not in the list, the plugin treats the request like a client request.

Later on, when describing the configuration of having a proxy in front of OpenSearch Dashboards, which is how we have things deployed, the documentation says:

In this case, the remote address of the HTTP call is the IP of OpenSearch Dashboards, because it sits directly in front of OpenSearch

So in either case, it seems like the value of X-Forwarded-For is not used to determine whether a request matches the trusted IPs set in internalProxies. Instead, the detected remote IP address from the request is checked for a match with internalProxies to determine if the traffic is allowed.

I have tested using a local Docker setup and found that I can set any value for X-Forwarded-For from my proxy and the traffic will succeed as long as the remote address matches the regex in internalProxies. I also enabled trace logging on my Docker OpenSearch nodes and the logs I’ve pasted below seem to confirm these findings.

In my testing, the only caveat I’ve found is that the X-Forwarded-For header is required for proxy traffic to succeed, otherwise you receive a 401 error.

So my understanding is that X-Forwarded-For is required for proxy authentication, but the actual value of the header does determine whether traffic is allowed. Is my understanding correct?

If my understanding is correct, then I am confused how or if the values in the X-Forwarded-For header are used at all for security? And if the X-Forwarded-For header values aren’t used to validate traffic, then why is it required?

Configuration:

config.yml:

---
_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    http:
      xff:
        enabled: true
        # trust IP addresses from Docker network
        internalProxies: '172\.19\.0.*'
        remoteIpHeader: "x-forwarded-for"
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 4
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
      proxy_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: extended-proxy
          challenge: false
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
            attr_header_prefix: "x-proxy-ext-"
        authentication_backend:
          type: noop

opensearch_dashboards.yml:

server.name: opensearchDashboards
server.host: "0.0.0.0"
opensearch.hosts: [https://localhost:9200]

opensearch.ssl.verificationMode: none
opensearch.username: admin
opensearch.password: admin

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
opensearch_security.readonly_mode.roles: ["opensearch_dashboards_read_only"]

# Use this setting if you are running opensearch dashboards without https
opensearch_security.cookie.secure: false

data.search.usageTelemetry.enabled: false
opensearch.requestHeadersAllowlist: ["securitytenant","Authorization","x-forwarded-for","x-proxy-user","x-proxy-roles","x-proxy-ext-spaceids","x-proxy-ext-orgids"]

opensearch_security.auth.type: "proxy"
opensearch_security.proxycache.user_header: "x-proxy-user"
opensearch_security.proxycache.roles_header: "x-proxy-roles"

# disable the welcome screen to make e2e tests less flaky
home.disableWelcomeScreen: true
# disable the new theme modal to make e2e tests less flaky
home.disableNewThemeModal: true

nginx.conf, configuration for Nginx proxy :

events {
  worker_connections  1024;
}

http {
  server {
    listen       8080;

    location / {
      proxy_pass http://opensearch-dashboards:5601;
      proxy_set_header X-Forwarded-For 5.6.7.8;
      proxy_set_header x-proxy-user test;
      proxy_set_header x-proxy-roles admin;
    }
  }
}

Relevant Logs or Screenshots:

[TRACE] 2024-07-24 19:42:03.697 [opensearch[opensearch-node1][transport_worker][T#7]] RemoteIpDetector - originalRemoteAddr 172.19.0.2
[TRACE] 2024-07-24 19:42:03.697 [opensearch[opensearch-node1][transport_worker][T#7]] RemoteIpDetector - concatRemoteIpHeaderValue 5.6.7.8
[TRACE] 2024-07-24 19:42:03.697 [opensearch[opensearch-node1][transport_worker][T#7]] RemoteIpDetector - Incoming request /_plugins/_security/dashboardsinfo with originalRemoteAddr '172.19.0.2', originalRemoteHost='opensearch-dashboards.docker_opensearch-net', will be seen as newRemoteAddr='5.6.7.8'
[TRACE] 2024-07-24 19:42:03.697 [opensearch[opensearch-node1][transport_worker][T#7]] XFFResolver - xff resolved opensearch-dashboards.docker_opensearch-net/172.19.0.2:40364 to /5.6.7.8:40364

Also, I’ve browsed the code that detects the remote IP, and I can see that does read the X-Forwarded-For header and returns the last value from it as the detected IP address, but that only confuses me more about why it doesn’t seem to matter when validating traffic.

And the only integration test I can find for this behavior proves that if the remote IP address does not match internalProxies, then a 401 response is returned. . But there is no corresponding integration test proving that if the value of X-Forwarded-For does not match internalProxies, then the traffic is rejected.

Hi @mark.boyd,

I do believe your understanding is correct, the X-Forwarded-For, as I understand, is used to identify the source of the original requester, once the proxy allows it through, hence the internalProxies: '172\.19\.0.*'. Moreover, if I recall it correctly the X-Forwarded-Forvalue can be any integer (not sure about string?) not necessary an IP (I will double check in my lab).

Let me know if you have any further questions or comments (or if I missed something).

Best,
mj

@Mantas

Thanks for the reply! Unfortunately, I think my previous comment said the exact opposite of what I meant. What I should have said was:

My understanding is that the X-Forwarded-For header is required for proxy authentication, but the actual value of the header does not determine whether traffic is allowed. Is my understanding correct?

Based on our testing, I have concluded that the actual detected source IP address of the proxy server is what is used for comparison to the internalProxies configuration to determine if the traffic is allowed. This conclusion seems supported by the RemoteIpDetector code in the security plugin and seemingly by this integration test behavior.

Does that sound correct?

Thanks again,
Mark

@mark.boyd That is correct (based on my tests and understanding).

Best,
mj