Cannot create index pattern in private tenant

Security
1

Hi,

We’re running version 2.6.0 of both OS as OS-dashboards.
We have set-up the proxy authentication and I’ve managed to login as admin user by passing the correct headers.

I can create index patterns in the global tenant, but if I try to create these in the private tenant, I cannot do this. Perhaps I have a missing role or something, but I would assume that “admin” has the correct rights to do this.

This is my role_mapping.yml:

   all_access:
            reserved: true
            hidden: false
            backend_roles:
            - "admin"
            hosts: []
            users:
            - "admin"
            and_backend_roles: []
            description: "Maps admin to all_access"

Anyone having similar issues? I didn’t find a bug report so must be my doing…

Kind regards

2

While someone in this channel may know the answer, there might be a better audience in Security - OpenSearch

1 Like
3

@arend.lapere Could you share your config.yml?

4

This is the config.yml I have. I’ve tried with the proxy and also set-up basic auth to test local user authentication but nothing seems to work in private tenant (global tenant works just fine).

config.yml

_meta:
  type: "config"
  config_version: 2
config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: true
        internalProxies: '.*' # trust all internal proxies, regex pattern
        remoteIpHeader:  'x-forwarded-for'
    authc:
      basic_auth_domain:
        description: "Authenticate via basic"
        http_enabled: true
        transport_enabled: false
        order: 3
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: noop
      proxy_auth_domain:
        description: "Authenticate via proxy"
        http_enabled: true
        transport_enabled: false
        order: 2
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
        authentication_backend:
          type: noop
      clientcert_auth_domain:
        description: "Authenticate via SSL client certificates"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: clientcert
          config:
            username_attribute: cn #optional, if omitted DN becomes username
          challenge: false
        authentication_backend:
          type: noop
5

@arend.lapere I forgot to ask. Could you share the curl command that you’ve used to create an index pattern in the private tenant?

6

Hi, do to xsrf protection I cannot past a working CURL, however, here’s the command anyway.
Small disclaimer, it is the cURL command generated from firefox minus the bloat. I actually create the index pattern from the UI:

curl 'http://[fde0::cd94]:5601/api/saved_objects/index-pattern' -X POST -H 'Content-Type: application/json' -H 'X-PROXY-USER: admin' -H 'X-PROXY-ROLES: admin' -H 'X-FORWARDED-FOR: 1.2.3.4' --data-raw '{"attributes":{"title":"opensearch_dashboards_sample_data_flights","timeFieldName":"timestamp","fields":"[{\"count\":0,\"name\":\"AvgTicketPrice\",\"type\":\"number\",\"esTypes\":[\"float\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"Cancelled\",\"type\":\"boolean\",\"esTypes\":[\"boolean\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"Carrier\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"Dest\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"DestAirportID\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"DestCityName\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"DestCountry\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"DestLocation\",\"type\":\"geo_point\",\"esTypes\":[\"geo_point\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"DestRegion\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"DestWeather\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"DistanceKilometers\",\"type\":\"number\",\"esTypes\":[\"float\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"DistanceMiles\",\"type\":\"number\",\"esTypes\":[\"float\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"FlightDelay\",\"type\":\"boolean\",\"esTypes\":[\"boolean\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"FlightDelayMin\",\"type\":\"number\",\"esTypes\":[\"integer\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"FlightDelayType\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"FlightNum\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"FlightTimeHour\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"FlightTimeMin\",\"type\":\"number\",\"esTypes\":[\"float\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"Origin\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"OriginAirportID\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"OriginCityName\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"OriginCountry\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"OriginLocation\",\"type\":\"geo_point\",\"esTypes\":[\"geo_point\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"OriginRegion\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"OriginWeather\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"_id\",\"type\":\"string\",\"esTypes\":[\"_id\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_index\",\"type\":\"string\",\"esTypes\":[\"_index\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_score\",\"type\":\"number\",\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_source\",\"type\":\"_source\",\"esTypes\":[\"_source\"],\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_type\",\"type\":\"string\",\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"dayOfWeek\",\"type\":\"number\",\"esTypes\":[\"integer\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"timestamp\",\"type\":\"date\",\"esTypes\":[\"date\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true}]"},"references":[]}' --output -

7

Also interesting to note is, after creating, I see the detail page of the index pattern; e.g. http://[fde0::cd94]:5601/app/management/opensearch-dashboards/indexPatterns/patterns/96414910-e9a5-11ed-8695-19bd25eea1a8#/?_a=(tab:indexedFields)

And this loads just fine, I just can’t seem to list it.

Another thing that might be interesting is the fact that I cannot view any saved object at all (so in the left hand menu clicking saved objects), I just get a 400:

Looking in the network inspector I can see:

all shards failed: search_phase_execution_exception: [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [type] in order to load field data by uninverting the inverted index. Note that this can use significant memory.

For the “index patterns” page I get no error and even “worse” I get a 200 with an empty list:

{"page":1,"per_page":10000,"total":0,"saved_objects":[]}

The logs for master node nor dashboard node seem to spit out anything I can work with unfortunately :frowning: For the dashboards I see the HTTP requests and for master node I don’t even see any logs when I request something on the dashboard page

8

@arend.lapere I’ve tried your command with basicauth and I’ve got the following error.

{“statusCode”:400,“error”:“Bad Request”,“message”:“Request must contain a osd-xsrf header.”}

When I added -H 'osd-xsrf:true' to the curl command it completed and the index pattern appeared in the index patterns under the Private tenant.

image
image1303×849 56.3 KB

9

@arend.lapere Do you get these errors in the Global or Private tenant?

10

Aah, Didn’t think about adding -H 'osd-xsrf:true' but indeed, with that addition the cURL works!
However, I still don’t see it in the list of the private tenant.

This only seems to misbehave in Private tenant. In global tenant all seems to be working as expected.

These are the assigned roles I’ve got from OSD
image

I’d assume that all_access is like a superuser? Or is there a different role I’d need?

11

@arend.lapere Did you make any changes to the own_index mapping? Could you share your roles_mapping.yml?

This is what I have in my test cluster.

own_index:
  reserved: false
  users:
  - "*"
  description: "Allow full access to an index named like the username"
12

@arend.lapere Also, please share your opensearch_dashboards.yml

13

Thanks for your help Pablo.

Meanwhile I’ve added the role you mentioned and even recreated the cluster.

image

But… now it seems to be worse… I cannot even create things in global tenant as well? Very strange

This is my roles_mapping.yml:

_meta:
  type: "rolesmapping"
  config_version: 2

dashboard_role:
  reserved: true
  users:
  - "kibanaserver"
  - "dashboard"

all_access:
  reserved: true
  hidden: false
  backend_roles:
  - "admin"
  hosts: []
  users:
  - "admin"
  and_backend_roles: []
  description: "Maps admin to all_access"
  
own_index:
  reserved: false
  users:
  - "*"
  description: "Allow full access to an index named like the username"

This is my opensearch_dashboards.yml file:

server:
        name: opensearch-dashboards
        host: "::"

      opensearch:
        ssl:
          verificationMode: full
          certificate: /usr/share/opensearch/config/pki/dashboard.pem
          key: /usr/share/opensearch/config/pki/dashboard.key
          certificateAuthorities: [/usr/share/opensearch/config/pki/ca.pem]
        requestHeadersWhitelist: 
        - "securitytenant"
        - "Authorization"
        - "x-forwarded-for"
        - "x-proxy-user"
        - "x-proxy-roles"

      opensearch_security:
        multitenancy:
          enabled: true
          tenants:
            preferred:
            - "Private"
            - "Global"
        readonly_mode:
          roles: ["kibana_read_only"]
        cookie:
          secure: false
        auth:
          type: "proxy"
        proxycache:
          user_header: "x-proxy-user"
          roles_header: "x-proxy-roles"
15

@arend.lapere Did you replace kibana_server with dashboard_role? If so, did you change that in the security plugin?

16

I’m unsure what you mean? But if you mean, “did you create the roles in the security plugin”, then I have this in my roles.yml:

dashboard_role:
            reserved: true
            cluster_permissions:
              - 'cluster_monitor'
              - 'cluster:monitor/nodes/info'
              - 'cluster:monitor/state'
            tenant_permissions:
              - tenant_patterns:
                - "*"
                allowed_actions:
                - "kibana_all_read"
            index_permissions:
              - index_patterns:
                  - '*'
                allowed_actions:
                  - 'indices:admin/get'
                  - 'indices:admin/create'
                  - 'indices:admin/refresh'
                  - 'indices:admin/refresh[s]'
                  - 'indices:admin/aliases'
                  - 'indices:admin/aliases/get'
                  - 'indices:data/read/search'
                  - 'indices:data/write/bulk'
                  - 'indices:admin/template/put'

To be honest, when setting it up, I didn’t have a lot of luck getting information on what to set as default roles. I’m using the helm chart and I think, at least at the time I started using it, that these files were mandatory to include myself… but perhaps therein lies my biggest mistake.

I’ll see to remove my “custom files” and see if it works.

17

@arend.lapere Sorry I meant the kibanaserver in terms of config.yml. You can use custom user name instead of kibanaserver user but it must be assigned to the kibana_server role or a copy of that role.

Your dashboard_role is more restrictive and is missing many privileges compared to the kibana_server role. This will affect the functionality of the OpenSearch Dashboards.

image
image1924×514 45.8 KB

18

OK, I finally managed to get this to work.
The “dashboard” user which you called was actually being passed from my mTLS set-up between dashboard server and opensearch.
I’ve then went ahead and followed the default roles (the github link: security/static_roles.yml at ae6ac7268d2733e8e38626b55f10447a8c1c6876 · opensearch-project/security · GitHub showed me the roles that pre-existed)

I only, for some reason, needed to add the ability to load the global_tenant via this dashboard user and map the kibana_server to it and Bob’s your uncle.

role_mappings.yml

_meta:
            type: "rolesmapping"
            config_version: 2

          all_access:
            reserved: false
            backend_roles:
            - "admin"
            description: "Maps admin to all_access"

          own_index:
            reserved: false
            users:
            - "*"
            description: "Allow full access to an index named like the username"

          logstash:
            reserved: false
            backend_roles:
            - "logstash"

          kibana_user:
            reserved: false
            backend_roles:
            - "kibanauser"
            description: "Maps kibanauser to kibana_user"

          readall:
            reserved: false
            backend_roles:
            - "readall"

          manage_snapshots:
            reserved: false
            backend_roles:
            - "snapshotrestore"

          kibana_server:
            reserved: true
            users:
            - "dashboard"
          
          dashboard_role:
            reserved: true
            users:
            - "dashboard"

roles.yaml (excerpt but the static roles with this one are really just it)

_meta:
            type: "roles"
            config_version: 2

          dashboard_role:
            reserved: true
            tenant_permissions:
              - tenant_patterns:
                - "*"
                allowed_actions:
                - "kibana_all_read"

Again, thank you so much for your help, couldn’t for the life of me have figured this one out on my own.

I hope this might come in handy for anyone else working with mTLS + dashboards.

2 Likes