OpensSearch dashboard proxy authentication

[dstefanox]

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Using latest tag for all images in docker-compose.
OpenSearch/Dasboard 2.11.0
Nginx 1.25.2

Describe the issue:
I am trying to setup basic docker-compose of OpenSearch dashboards, which uses single node and Nginx as proxy,
and to use proxy based authentication.
Setup is based on Proxy-based authentication - OpenSearch documentation

First step that I did was to create docker-compose which works, where I can access OpenSearch dashboards
directly through port 5061, or via nginx proxy on port 8090. I used following docker-compose file, with only difference
that volume sections were commented-out (except for nginx configuration):

version: '3'

services:
  opensearch:
    image: opensearchproject/opensearch:latest
    container_name: opensearch
    environment:
      - discovery.type=single-node
      - OPENSEARCH_SECURITY_DISABLED=false  # Temporarily, we'll disable security for simplicity.
    ports:
      - 9200:9200
      - 9600:9600
    networks:
      - opensearch-net
    volumes:
      - ./config.yml:/usr/share/opensearch/config/opensearch-security/config.yml # Mounts the security configuration file to the container

  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    environment:
      OPENSEARCH_HOSTS: 'https://opensearch:9200'
    networks:
      - opensearch-net
    volumes:
      - ./opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml # Mount the configuration file to the container

  nginx:
    image: nginx:latest
    container_name: nginx-proxy
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
    ports:
      - 80:80
      - 8090:8090
    networks:
      - opensearch-net

networks:
  opensearch-net:

Everything works well, I was able to access dashboards both directly on 5601 or via proxy on 8090, of course with login screen.

Now next step was to add config.yml and opensearch_dashboards.yml, which I did by binding them as shown in docker-compose.yml file.

Content of config files is given in section below.

Configuration:
Content of config files is the following:
config.yml:

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: true
        remoteIpHeader: "x-forwarded-for"
        internalProxies: opensearch-dashboards
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 4
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
      proxy_auth_domain:
        description: "Authenticate via proxy"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
        authentication_backend:
          type: noop

opensearch_dashbards.yml

opensearch.hosts: [https://opensearch:9200]
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: kibanaserver
#opensearch.requestHeadersWhitelist: [authorization, securitytenant]
opensearch.requestHeadersAllowlist: ["securitytenant","Authorization","x-forwarded-for","x-proxy-user","x-proxy-roles"]

opensearch_security.auth.type: "proxy"
opensearch_security.proxycache.user_header: "x-proxy-user"
opensearch_security.proxycache.roles_header: "x-proxy-roles"

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
# Use this setting if you are running opensearch-dashboards without https
opensearch_security.cookie.secure: false
server.host: '0.0.0.0'

Relevant Logs or Screenshots:
When accessing via browser, url:, response is:

{"statusCode":401,"error":"Unauthorized","message":"Authentication Exception"}

In log, following entry appears:

e[36mnginx-proxy              |e[0m 192.168.208.1 - - [20/Oct/2023:07:54:12 +0000] "GET / HTTP/1.1" 401 78 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36" "-"
e[32mopensearch-dashboards    |e[0m {"type":"response","@timestamp":"2023-10-20T07:54:12Z","tags":[],"pid":1,"method":"get","statusCode":401,"req":{"url":"/","method":"get","headers":{"x-forwarded-for":"192.168.208.1","x-proxy-user":"kibanaro","x-proxy-roles":"kibanauser,readall","host":"opensearchdash","connection":"close","cache-control":"max-age=0","sec-ch-ua":"\"Google Chrome\";v=\"117\", \"Not;A=Brand\";v=\"8\", \"Chromium\";v=\"117\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,sr-RS;q=0.8,sr;q=0.7"},"remoteAddress":"192.168.208.4","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"},"res":{"statusCode":401,"responseTime":43,"contentLength":9},"message":"GET / 401 43ms - 9.0B"}
e[36mnginx-proxy              |e[0m 192.168.208.1 - - [20/Oct/2023:07:54:12 +0000] "GET /favicon.ico HTTP/1.1" 401 78 "http://localhost:8090/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36" "-"
e[32mopensearch-dashboards    |e[0m {"type":"response","@timestamp":"2023-10-20T07:54:12Z","tags":[],"pid":1,"method":"get","statusCode":401,"req":{"url":"/favicon.ico","method":"get","headers":{"x-forwarded-for":"192.168.208.1","x-proxy-user":"kibanaro","x-proxy-roles":"kibanauser,readall","host":"opensearchdash","connection":"close","sec-ch-ua":"\"Google Chrome\";v=\"117\", \"Not;A=Brand\";v=\"8\", \"Chromium\";v=\"117\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"image","referer":"http://localhost:8090/","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,sr-RS;q=0.8,sr;q=0.7"},"remoteAddress":"192.168.208.4","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36","referer":"http://localhost:8090/"},"res":{"statusCode":401,"responseTime":5,"contentLength":9},"message":"GET /favicon.ico 401 5ms - 9.0B"}

It seems that all relevant headers are passed by Nignx, but 401 response appears every time.
Any idea why this does not work?

[pablo]

@dstefanox

      xff:
        enabled: true
        remoteIpHeader: "x-forwarded-for"
        internalProxies: opensearch-dashboards

Could you test the below instead?

      xff:
        enabled: true
        remoteIpHeader: "x-forwarded-for"
        internalProxies: '.*'

[dstefanox]

@pablo
Thanks for your reply. I tried changes, and your suggestion works. Now I have new problem:

• Only role that works as expected is admin.
• Any other role that I create in roles.yml and map in roles_mapping.yml or even any of existing roles such as readall does not work.
  Behavior is the following:

I can access opensearch dashboards UI, but some of requests for getting data are getting refused and I get “Searching” spinner working on page. When inspecting network traffic in browser developer tools, I notice some of requests are refused (here is one example):
api/saved_objects/_find?fields=title&per_page=10000&type=index-pattern

{"statusCode":403,"error":"Forbidden","message":"no permissions for [indices:data/read/search] and User [name=Name Surname, backend_roles=[readall], requestedTenant=null]: security_exception: [security_exception] Reason: no permissions for [indices:data/read/search] and User [name=Name Surname, backend_roles=[readall], requestedTenant=null]"}

In UI, I see user “Name Surname” appearing as logged in, “View roles and identities” for that user shows roles: “own_index” and “readall” and backend roles “readall”.
It also shows “Global” below user name, which is I assume tenant, but opensearch log says:

Tenant global_tenant is not allowed for user Name Surname

Any ideas what can cause this?

[pablo]

@dstefanox Could you share the role and role mapping configurations for this test user?

[dstefanox]

@pablo As explained in post above, proxy based authentication is used. With each request, Nginx passes headers:
x-proxy-user: “Name Surname”
x-proxy-roles: “readall”
That is the case when I use existing roles that are predefined in opensearch. With further experiments, I noticed that readall role works as expected only if I add role kibana_user.

Similar thing appears when I try to send logs with Filebeat or Fluentbit. It works well only when admin role is used, any other role on its own does not work. There I noticed that adding role “logstash” helps, in which case role that I add with it behaves well. For example, if I create role “log_shipper”, which limits access to certain index only, give it all permissions as needed etc, beat can’t write anything because it gets rejected. If I also add logstash role, then it works with limitations I created.
For example, this does not work:
x-proxy-roles: “log_shipper”
But, following works:
x-proxy-roles: “log_shipper,logstash”

[pablo]

@dstefanox In your scenario the type of authentication is not a problem. To access OpenSearch Dashboards UI the user must have at least read access to .kibana indices. That’s why it worked with kibana_user role.

Alternatively you can map kibanauser backend role to readall role in roles_mapping.yml and then use only readall in the x-proxy-roles.

  "readall" : {
    "hosts" : [ ],
    "users" : [ ],
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [
      "readall",
      "kibanauser"
    ],
    "and_backend_roles" : [ ]
  },

[dstefanox]

@pablo Errors that were appearing in such situations were quite unclear, so it was really hard to figure-out that access to those indexes was missing. Similar, for filebeat, I would expect this kind of role to work out of box:

server_log_shipper:
  cluster_permissions:
    - "cluster:monitor/main"
    - "cluster:monitor/state"
  index_permissions:
    - index_patterns:
        - "server-log-*"
      allowed_actions:
        - "indices:admin/create"
        - "indices:admin/exists"
        - "indices:data/write/bulk*"
        - "indices:data/write/index"
        - "indices:admin/mapping/put"
        - "indices:admin/template/put"

But it simply didn’t work until I combined it with ‘logstash’ role.
What is most misleading in error messages is that it gives you which action is not permitted (and typically for me it was one of those that I explicitly allowed), but it does not mention in which index problem appears.