Unexpected Access to All Indices

[hm21]

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

• OS: Ubuntu 22.04 LTS
• Version: Docker images 2.11.1
• Firefox

Describe the issue:
When the .kibana index permission is removed from an OpenSearch role configuration, the user unexpectedly gains access to all indices rather than being restricted to specific index patterns defined in other permissions. Specifically, when the permission setting for .kibana is present, access is correctly limited to the specified “logstash-*” index patterns and the .kibana index. However, removing this permission unexpectedly broadens the user’s access to all indices.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Define an OpenSearch role that includes index permissions for “logstash-*” and “.kibana”.
2. Assign this role to a user, verifying that the user’s access is correctly limited to “logstash-*” indices and the .kibana index.
3. Remove the index permission for .kibana from the role.
4. Observe that the user now has access to all indices, contrary to the intended restriction to “logstash-*” indices.

What is the expected behavior?
The expected behavior is for the user’s access to be limited to the “logstash-*” index patterns as defined in the role’s index permissions, regardless of whether the .kibana index permission is present or not. Removing the .kibana index permission should not affect access to other indices outside of those explicitly defined in the role.

Configuration:
Here’s the role configuration that leads to the described behavior:

resource "opensearch_role" "writer" {
  role_name   = "logs_writer"
  description = "Logs writer role"

  cluster_permissions = ["cluster:monitor/health", "cluster:monitor/state"]

  index_permissions {
    index_patterns  = ["logstash-*"]
    allowed_actions = ["write"]
  }

  // When this permission block is removed, the issue occurs.
  index_permissions {
    index_patterns  = [".kibana"]
    allowed_actions = ["indices:admin/", "indices:monitor/", "indices:data/*"]
  }

  tenant_permissions {
    tenant_patterns = ["global_tenant"]
    allowed_actions = ["kibana_all_write"]
  }
}

[Mantas]

Hi @hm21,

Could you share the output of GET _plugins/_security/api/roles before “How can one reproduce the bug?” step 3 and after step 3 ?

Thanks,
mj

[hm21]

before “How can one reproduce the bug?” step 3:

"logs_writer": {
    "reserved": false,
    "hidden": false,
    "description": "Logs writer role",
    "cluster_permissions": [
      "cluster:monitor/health",
      "cluster:monitor/state"
    ],
    "index_permissions": [
      {
        "index_patterns": [
          ".kibana"
        ],
        "dls": "",
        "fls": [],
        "masked_fields": [],
        "allowed_actions": [
          "indices:data/*",
          "indices:monitor/*",
          "indices:admin/*"
        ]
      },
      {
        "index_patterns": [
          "opensearch_dashboards_sample_data_*"
        ],
        "dls": "",
        "fls": [],
        "masked_fields": [],
        "allowed_actions": [
          "indices:data/*",
          "indices:monitor/*",
          "indices:admin/*"
        ]
      }
    ],
    "tenant_permissions": [
      {
        "tenant_patterns": [
          "global_tenant"
        ],
        "allowed_actions": [
          "kibana_all_read",
          "kibana_all_write"
        ]
      }
    ],
    "static": false
  },

after step 3:

"logs_writer": {
    "reserved": false,
    "hidden": false,
    "description": "Logs writer role",
    "cluster_permissions": [
      "cluster:monitor/health",
      "cluster:monitor/state"
    ],
    "index_permissions": [
      {
        "index_patterns": [
          "opensearch_dashboards_sample_data_*"
        ],
        "dls": "",
        "fls": [],
        "masked_fields": [],
        "allowed_actions": [
          "indices:data/*",
          "indices:monitor/*",
          "indices:admin/*"
        ]
      }
    ],
    "tenant_permissions": [
      {
        "tenant_patterns": [
          "global_tenant"
        ],
        "allowed_actions": [
          "kibana_all_read",
          "kibana_all_write"
        ]
      }
    ],
    "static": false
  },

[hm21]

Maybe, the config.yml file is also interesting to troubleshoot.

"dynamic": {
    "http": {
      "anonymous_auth_enabled": false
    },
    "authc": {
      "internal_auth": {
        "order": 0,
        "description": "HTTP basic authentication using the internal user database",
        "http_enabled": true,
        "transport_enabled": true,
        "http_authenticator": {
          "type": "basic",
          "challenge": true
        },
        "authentication_backend": {
          "type": "internal"
        }
      }
    },
    "do_not_fail_on_forbidden": true,
    "do_not_fail_on_forbidden_empty": true,
    "multi_rolespan_enabled": true
  }

[hm21]

I think I found a pattern. The thing is if an index or an index-pattern is passed as “index_patterns” for which there are no indices in the cluster, then and only then you can view all indices.

resource "opensearch_role" "writer" {
  role_name   = "logs_writer"
  description = "Logs writer role"

  cluster_permissions = ["cluster:monitor/health", "cluster:monitor/state"]

  index_permissions {
    index_patterns  = ["opensearch_dashboards_sample_data"]
    allowed_actions = ["indices:admin/*", "indices:monitor/*", "indices:data/*"]
  }

  tenant_permissions {
    tenant_patterns = ["global_tenant"]
    allowed_actions = ["kibana_all_write", "kibana_all_read"]
  }
}

In my cluster there is a index named

opensearch_dashboards_sample_data_logs

Now, let’s say I pass a index name or pattern which is not nonexistens then all indices are existent indices are retrieved. For example:

index_permissions {
    index_patterns  = ["asdf"]
    allowed_actions = ["indices:admin/*", "indices:monitor/*", "indices:data/*"]
  }

I don’t have any index with the name of “asdf”, but now I get all of the indices that are currently available in the cluster.

Another thing is, that I can see all of the indices in the list, but when I try to access them I get an error:

Nonetheless, the plugin shouldn’t even show me the indices, even if the user cannot access them…

Also cat request:

IMPORTANT: Now, when adding the following, then everything works fine.

index_permissions {
    index_patterns  = [".kibana"]
    allowed_actions = ["indices:admin/*", "indices:monitor/*", "indices:data/*"]
  }

And this confuses me. I thought “.kibana” is for dashboard objects and does not have to do anything with security and permissions.

[hm21]

Here is also somebody else who ran into the same thing…

How to get a list of indices for which i have been granted permission?

Interestingly i started by testing the _stats API… and even though i have only granted READ permission on 2 indexes, _stats is returning the information for all indexes, is that expected with those new options set in config.yml? shouldnt i only see the 2 indexes? …same for _cat/indices

[Mantas]

@hm21

    "do_not_fail_on_forbidden": true,
    "do_not_fail_on_forbidden_empty": true,

Hi @hm21,

Is there a reason you have do_not_fail_on_.. fields enabled?

Doing some digging, I’ve found out that it’s being worked on here: [RFC] Refined index authorization (replacement for do_not_fail_on_forbidden) · Issue #3905 · opensearch-project/security · GitHub

Best,
mj

[hm21]

After setting do_not_fail_on_forbidden and do_not_fail_on_forbidden_empty to false, I can’t access and view anything.

[hm21]

@Mantas

Doing some digging, I’ve found out that it’s being worked on here: [RFC] Refined index authorization (replacement for do_not_fail_on_forbidden) · Issue #3905 · opensearch-project/security · GitHub

Yes, this explains it. I knew, it wasn’t a misconfiguration on my side, but a bug.

[Mantas]

@hm21

After setting do_not_fail_on_forbidden and do_not_fail_on_forbidden_empty to false, I can’t access and view anything.

This is because of insufficient permissions to “READ” indices. You can adjust that in your roles.

Best,
mj

[hm21]

@Mantas

This is because of insufficient permissions to “READ” indices. You can adjust that in your roles.

But then users could access all indices. I don’t want that. I want specific users to have read access to specific indices and not all indices.

Or am I wrong and I just misunderstood your statement?

[Mantas]

Hi @hm21,

Yes, you can achieve that using index permissions see here: Permissions - OpenSearch Documentation

The do_not_fail_on_forbidden is used in specific cases i.e: you want to let a user to GET _cat/indices while users have limited access to indices (included but not limited to this sample, to illustrate the do_not_fail_on_forbidden use).

For more granular permission control you can check the following:

Document-level security

Field-level security

I hope that helps, let me know if you have further questions or want to deep-dive into some examples.

Best,
mj

[hm21]

Let say, I set do_not_fail_on_forbidden and do_not_fail_on_forbidden_empty to false.

And now, let’s say, I have 5 different index-patterns in my cluster with the following names e.g.:

1. logs-*
2. metrics-machine-*
3. python-node-*
4. robotic-machine-*
5. fruitcollection-*

One thing here is important. I ship the logs to OS using Fluentd and there I added the logstash format parameter. That means for each new day a new index is created. For example:

logs-2024-02-16
logs-2024-02-15
logs-2024-02-14

and that happens also for the 4 other index patterns I mentioned.

Now, I wanna give the user Alexander access ONLY to logs-* index pattern and NO access to the other 4 index patterns.

How would I create a ROLE as API Request or as Terraform code? Just like the following

@hm21

resource "opensearch_role" "writer" {
  role_name   = "logs_writer"
  description = "Logs writer role"

  cluster_permissions = ["cluster:monitor/health", "cluster:monitor/state"]

  index_permissions {
    index_patterns  = ["opensearch_dashboards_sample_data"]
    allowed_actions = ["indices:admin/*", "indices:monitor/*", "indices:data/*"]
  }

  tenant_permissions {
    tenant_patterns = ["global_tenant"]
    allowed_actions = ["kibana_all_write", "kibana_all_read"]
  }
}

Could you help me here?

[hm21]

this issue explains why im using do_not_fail_on_forbidden setting.

Improve the security permission check in cat indices (_cat/indices) API

opensearch-dashboards  |   displayName: 'AuthorizationException',
opensearch-dashboards  |   path: '/_cat/indices/*',
opensearch-dashboards  |   query: { format: 'json', s: 'index:desc' },
opensearch-dashboards  |   body: {
opensearch-dashboards  |     error: {
opensearch-dashboards  |       root_cause: [Array],
opensearch-dashboards  |       type: 'security_exception',
opensearch-dashboards  |       reason: 'no permissions for [indices:monitor/settings/get] and User [name=test_user, backend_roles=[], requestedTenant=]'
opensearch-dashboards  |     },
opensearch-dashboards  |     status: 403
opensearch-dashboards  |   },
opensearch-dashboards  |   statusCode: 403,
opensearch-dashboards  |   response: '{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [indices:monitor/settings/get] and User [name=test_user, backend_roles=[], requestedTenant=]"}],"type":"security_exception","reason":"no permissions for [indices:monitor/settings/get] and User [name=test_user, backend_roles=[], requestedTenant=]"},"status":403}',
opensearch-dashboards  |   toString: [Function (anonymous)],
opensearch-dashboards  |   toJSON: [Function (anonymous)]

This is the error I get when I remove the do_not_fail_on_forbidden and do_not_fail_on_forbidden_empty

path: ‘/_cat/indices/*’,

This explains it well. The dashboard queries that path. Since I’m granting only access to a single index it will return an error, because the “_cat/indices” request requests all indices “*”.

[hm21]

Strange things happen:

index_permissions {
    index_patterns  = ["opensearch_dashboards_sample_data_logs"]
    allowed_actions = ["indices:admin/*", "indices:monitor/*", "indices:data/*"]
  }

Works just fine, right?

index_permissions {
    index_patterns  = ["opensearch_dashboards_sample_data"]
    allowed_actions = ["indices:admin/*", "indices:monitor/*", "indices:data/*"]
  }

Just removed “logs” from “opensearch_dashboards_sample_data_logs” index patterns field in the index permission and all of a sudden, I can view all indices.

[hm21]

I think, I understand it:
Specified an index pattern for non-existent indices. That showed me all indices then. Still a bug but…

[Mantas]

@hm21

Alexander access ONLY to logs-*

it would look something like:

role-for-alexander:
  cluster_permissions:
    - < choose from: https://opensearch.org/docs/latest/security/access-control/permissions/#cluster-permissions >
  index_permissions:
    - index_patterns:
        - "logs-20*"
    - allowed_actions:
        - < choose from: https://opensearch.org/docs/latest/security/access-control/permissions/#index-permissions >

If a user with an assigned role " role-for-alexander" needs to use _cat/indices, it must be _cat/indices/logs-20* or it will display a not enough permissions error. As a workaround, you can use do_not_fail_on_forbidden, however, you need to be aware of: [RFC] Refined index authorization (replacement for do_not_fail_on_forbidden) · Issue #3905 · opensearch-project/security · GitHub

best,
mj

[Mantas]

@hm21,

Thanks for documenting your findings, this will benefit the community.

Best,
mj

[hm21]

@Mantas

If a user with an assigned role " role-for-alexander" needs to use _cat/indices, it must be _cat/indices/logs-20*

yep, right and when you use _cat/indices/asdf*and there is no index asdf wether you have permissions for that index or not, then you get all indices from the cluster.