No results when using API keys

Versions OpenSearch 3.7

Describe the issue: I am having issues when trying to use API keys. The cluster is a small QA environment. Human users authenticate via SAML to Dashboards and get permissions from LDAP, tools users authenticate via internal credentials and basic auth.

When trying to use API keys, results do not contain any documents, but no errors or other events are logged.

Configuration:

API key is issued via Dashboards, the issuing user is authenticated via SAML and has the roles readall_and_monitor, kibana_user and all_access.

The key looks like:

...
"cluster_permissions": [],
"index_permissions": [
  {
    "index_pattern": [
      "nginx-access-stream"
    ],
    "allowed_actions": [
      "read"
    ]
  }
],
...

Document level security is not in use.

Relevant Logs or Screenshots:

Query: GET nginx-access-stream/_count

Via Basic auth or via Dev Tools as the user issuing the key: {"count":20788106,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}

Via API key: {"count":0,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}

Trying another index, not covered by the key, gives the expected {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [indices:data/read/search] and User [name=token:nginx_reader, backend_roles=[], requestedTenant=null]"}],"type":"security_except
ion","reason":"no permissions for [indices:data/read/search] and User [name=token:nginx_reader, backend_roles=[], requestedTenant=null]"},"status":403}

/_plugins/_security/authinfo returns:

with basic auth:
{
"user": "User [name=cli, backend_roles=[], requestedTenant=null]",
"user_name": "cli",
"user_requested_tenant": null,
"remote_address": "10.84.2.18:34106",
"backend_roles": [],
"custom_attribute_names": [],
"roles": [
"structured_log_read_role",
"mail_log_read_role"
],
"tenants": {
"cli": true
},
"principal": null,
"peer_certificates": "0",
"sso_logout_url": null
}

with api key:

{
"user": "User [name=token:nginx_reader, backend_roles=[], requestedTenant=null]",
"user_name": "token:nginx_reader",
"user_requested_tenant": null,
"remote_address": "10.84.2.18:44214",
"backend_roles": [
""
],
"custom_attribute_names": [],
"roles": [],
"tenants": {
"token:nginx_reader": true
},
"principal": null,
"peer_certificates": "0",
"sso_logout_url": null
}

as issuing user:

{
"user": "User [name=..., backend_roles=[default, admin], requestedTenant=null]",
"user_name": "...",
"user_requested_tenant": null,
"remote_address": "127.0.0.1:34638",
"backend_roles": [
"default",
"admin"
],
"custom_attribute_names": [
"attr.jwt.saml_ni",
"attr.jwt.sub",
"attr.jwt.saml_nif",
"attr.jwt.nbf",
"attr.jwt.exp",
"attr.jwt.saml_si"
],
"roles": [
"readall_and_monitor",
"kibana_user",
"all_access"
],
"tenants": {
"global_tenant": true,
"admin_tenant": true,
"...": true
},
"principal": null,
"peer_certificates": "0",
"sso_logout_url": "https://..."
}

Hi @marcb I suspect that the token has insufficient permissions to perform count, but don’t know what the minimal set would be just yet and how to make sure it doesn’t silently fail. I will take a closer look at this. Thanks!

Raised an issue on the security repo: API key with read permission returns zero hits for _count on allowed index · Issue #6217 · opensearch-project/security · GitHub

Hi, thanks for looking tin to this. But note that count was just a simplification. search and get have the same result, i.e. 0 hits and 404.

This is unexpected, because no finer-grained security is in place.

I will attach some logs to the ticket.