Problem with index permissions

wleight · July 17, 2024, 7:01pm

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch version 2.13.0

Describe the issue:
First, I’m new to OpenSearch, so apologies in advance if I missed something obvious. Also, I should probably say that I’m using OpenSearch because it comes as the backend for the perfSonar network monitoring software. At any rate, no data is being displayed on Grafana, which instead says that there’s an unexpected Opensearch error. The error I see in the log file is in the logs section below: if I’m understanding correctly, it’s saying that the pscheduler_logstash user doesn’t have permission to create an index called prometheus_node. But as far as I can tell, the pscheduler_logstash role, to which the pscheduler_logstash user is mapped – the error explicitly says that it checks this role – should be configured to have index creation permissions for anything starting with prometheus_ (the relevant parts of roles.yml and roles_mapping.yml are below). In particular, according to the documentation, create_index contains the indices:admin/create action that it says it can’t do. Additionally, everything was working fine until I had to restart the server where this was running a few days ago. Any suggestion on how to deal with this problem would be much appreciated.

Configuration:
In roles_mapping.yml, the pscheduler_logstash user is mapped to the pscheduler_logstash role:

pscheduler_logstash:
  reserved: true
  users:
  - "pscheduler_logstash"

And the role is defined as follows in roles.yml:

pscheduler_logstash:
  cluster_permissions:
    - 'cluster_monitor'
    - 'cluster_manage_index_templates'
  index_permissions:
    - index_patterns:
      - 'pscheduler_*'
      - 'prometheus_*'
      allowed_actions:
      - 'write'
      - 'read'
      - 'delete'
      - 'create_index'
      - 'manage'
      - 'indices:admin/template/delete'
      - 'indices:admin/template/get'
      - 'indices:admin/template/put'

Relevant Logs or Screenshots:
[2024-07-17T18:47:22,709][INFO ][o.o.s.p.PrivilegesEvaluator] [net2ps2] No index-level perm match for User [name=pscheduler_logstash, backend_roles=[], requestedTenant=null] Resolved [aliases=[], allIndices=[prometheus_node], types=[*], originalRequested=[prometheus_node], remoteIndices=[]] [Action [indices:admin/create]] [RolesChecked [own_index, pscheduler_logstash]]

Mantas · July 22, 2024, 8:27am

Hi @wleight,

first of all welcome,

Would you mind sharing the outputs of:

GET _plugins/_security/api/roles

GET _plugins/_security/api/rolesmapping

GET _plugins/_security/api/securityconfig

thanks,
mj

wleight · July 22, 2024, 2:03pm

Hi MJ, thanks for the welcome! The outputs are below. I noticed, looking at the output of the roles call, that for the pscheduler_logstash user, only the pscheduler_* index pattern is listed under its index permissions, which is presumably why I see this error. But the prometheus_* index pattern is listed in roles.yml, so I’m still confused. Is there some other place where the roles are defined where this could be overwritten?

best,

will

_plugins/_security/api/roles:

_plugins/_security/api/rolesmapping:

{"manage_snapshots":{"hosts":[],"users":[],"reserved":false,"hidden":false,"backend_roles":["snapshotrestore"],"and_backend_roles":[]},"logstash":{"hosts":[],"users":[],"reserved":false,"hidden":false,"backend_roles":["logstash"],"and_backend_roles":[]},"own_index":{"hosts":[],"users":["*"],"reserved":false,"hidden":false,"backend_roles":[],"and_backend_roles":[],"description":"Allow full access to an index named like the username"},"kibana_user":{"hosts":[],"users":[],"reserved":false,"hidden":false,"backend_roles":["kibanauser"],"and_backend_roles":[],"description":"Maps kibanauser to kibana_user"},"pscheduler_writer":{"hosts":[],"users":["pscheduler_writer"],"reserved":true,"hidden":false,"backend_roles":[],"and_backend_roles":[]},"opendistro_security_anonymous":{"hosts":[],"users":[],"reserved":true,"hidden":false,"backend_roles":["opendistro_security_anonymous_backendrole"],"and_backend_roles":[]},"pscheduler_logstash":{"hosts":[],"users":["pscheduler_logstash"],"reserved":true,"hidden":false,"backend_roles":[],"and_backend_roles":[]},"pscheduler_reader":{"hosts":[],"users":["pscheduler_reader"],"reserved":true,"hidden":false,"backend_roles":["opendistro_security_anonymous_backendrole"],"and_backend_roles":[]},"all_access":{"hosts":[],"users":[],"reserved":false,"hidden":false,"backend_roles":["admin"],"and_backend_roles":[],"description":"Maps admin to all_access"},"readall":{"hosts":[],"users":[],"reserved":false,"hidden":false,"backend_roles":["readall"],"and_backend_roles":[]},"kibana_server":{"hosts":[],"users":["kibanaserver"],"reserved":true,"hidden":false,"backend_roles":[],"and_backend_roles":[]}}

_plugins/_security/api/securityconfig:

{"config":{"dynamic":{"filtered_alias_mode":"warn","disable_rest_auth":false,"disable_intertransport_auth":false,"respect_request_indices_options":false,"kibana":{"multitenancy_enabled":true,"private_tenant_enabled":true,"default_tenant":"","server_username":"kibanaserver","index":".kibana"},"http":{"anonymous_auth_enabled":true,"xff":{"enabled":false,"internalProxies":"192\\.168\\.0\\.10|192\\.168\\.0\\.11","remoteIpHeader":"X-Forwarded-For"}},"authc":{"jwt_auth_domain":{"http_enabled":false,"order":0,"http_authenticator":{"challenge":false,"type":"jwt","config":{"signing_key":"base64 encoded HMAC key or public RSA/ECDSA pem key","jwt_header":"Authorization"}},"authentication_backend":{"type":"noop","config":{}},"description":"Authenticate via Json Web Token"},"ldap":{"http_enabled":false,"order":5,"http_authenticator":{"challenge":false,"type":"basic","config":{}},"authentication_backend":{"type":"ldap","config":{"enable_ssl":false,"enable_start_tls":false,"enable_ssl_client_auth":false,"verify_hostnames":true,"hosts":["localhost:8389"],"userbase":"ou=people,dc=example,dc=com","usersearch":"(sAMAccountName={0})"}},"description":"Authenticate via LDAP or Active Directory"},"basic_internal_auth_domain":{"http_enabled":true,"order":4,"http_authenticator":{"challenge":true,"type":"basic","config":{}},"authentication_backend":{"type":"intern","config":{}},"description":"Authenticate via HTTP Basic against internal users database"},"proxy_auth_domain":{"http_enabled":false,"order":3,"http_authenticator":{"challenge":false,"type":"proxy","config":{"user_header":"x-proxy-user","roles_header":"x-proxy-roles"}},"authentication_backend":{"type":"noop","config":{}},"description":"Authenticate via proxy"},"clientcert_auth_domain":{"http_enabled":false,"order":2,"http_authenticator":{"challenge":false,"type":"clientcert","config":{"username_attribute":"cn"}},"authentication_backend":{"type":"noop","config":{}},"description":"Authenticate via SSL client certificates"},"kerberos_auth_domain":{"http_enabled":false,"order":6,"http_authenticator":{"challenge":true,"type":"kerberos","config":{"krb_debug":false,"strip_realm_from_principal":true}},"authentication_backend":{"type":"noop","config":{}}}},"authz":{"roles_from_another_ldap":{"http_enabled":false,"authorization_backend":{"type":"ldap","config":{}},"description":"Authorize via another Active Directory"},"roles_from_myldap":{"http_enabled":false,"authorization_backend":{"type":"ldap","config":{"enable_ssl":false,"enable_start_tls":false,"enable_ssl_client_auth":false,"verify_hostnames":true,"hosts":["localhost:8389"],"rolebase":"ou=groups,dc=example,dc=com","rolesearch":"(member={0})","userrolename":"disabled","rolename":"cn","resolve_nested_roles":true,"userbase":"ou=people,dc=example,dc=com","usersearch":"(uid={0})"}},"description":"Authorize via LDAP or Active Directory"}},"auth_failure_listeners":{},"do_not_fail_on_forbidden":false,"multi_rolespan_enabled":true,"hosts_resolver_mode":"ip-only","do_not_fail_on_forbidden_empty":false,"on_behalf_of":{"enabled":false}}}}

Mantas · July 23, 2024, 3:35pm

Hi @wleight,

I can see you have pscheduler_logstash and the user pscheduler_logstash mapped to that role (the role should show up as the backend_roles=):

	"pscheduler_logstash": {
		"reserved": false,
		"hidden": false,
		"cluster_permissions": [
			"cluster_monitor",
			"cluster_manage_index_templates"
		],
		"index_permissions": [
			{
				"index_patterns": [
					"pscheduler_*"
				],
				"fls": [],
				"masked_fields": [],
				"allowed_actions": [
					"write",
					"read",
					"delete",
					"create_index",
					"manage",
					"indices:admin/template/delete",
					"indices:admin/template/get",
					"indices:admin/template/put"
				]
			}
		],
		"tenant_permissions": [],
		"static": false
	}

role mapping:

	"pscheduler_logstash": {
		"hosts": [],
		"users": [
			"pscheduler_logstash"
		],
		"reserved": true,
		"hidden": false,
		"backend_roles": [],
		"and_backend_roles": []
	}

Could you please share the output of:

GET _plugins/_security/api/internalusers/pscheduler_logstash?pretty

thanks,
mj

wleight · July 23, 2024, 9:07pm

Hi MJ,
Sure, see below.

thanks,

will


{
  "pscheduler_logstash" : {
    "hash" : "",
    "reserved" : true,
    "hidden" : false,
    "backend_roles" : [ ],
    "attributes" : { },
    "description" : "pscheduler logstash user",
    "opendistro_security_roles" : [ ],
    "static" : false
  }
}

Mantas · July 24, 2024, 12:13pm

@wleight,

All looks in order, could you check with the below if the backend_roles are mapped as expected (please share the output):

curl --insecure -u pscheduler_logstash:<password> -XGET https://<OS_node>:9200/_plugins/_security/authinfo?pretty

best,
mj

wleight · July 24, 2024, 2:26pm

Hi MJ,

Sure, here is the output:

{
  "user" : "User [name=pscheduler_logstash, backend_roles=[], requestedTenant=null]",
  "user_name" : "pscheduler_logstash",
  "user_requested_tenant" : null,
  "remote_address" : "127.0.0.1:53464",
  "backend_roles" : [ ],
  "custom_attribute_names" : [ ],
  "roles" : [
    "own_index",
    "pscheduler_logstash"
  ],
  "tenants" : {
    "pscheduler_logstash" : true
  },
  "principal" : null,
  "peer_certificates" : "0",
  "sso_logout_url" : null
}

thanks,

will

Mantas · July 24, 2024, 2:32pm

@wleight , Ok, so the role is mapped, can you add indices:admin/create to your role as per:

pscheduler_logstash:
  cluster_permissions:
    - 'cluster_monitor'
    - 'cluster_manage_index_templates'
  index_permissions:
    - index_patterns:
      - 'pscheduler_*'
      - 'prometheus_*'
      allowed_actions:
      - 'indices:admin/create'
      - 'write'
      - 'read'
      - 'delete'
      - 'create_index'
      - 'manage'
      - 'indices:admin/template/delete'
      - 'indices:admin/template/get'
      - 'indices:admin/template/put'

best,
mj

wleight · July 24, 2024, 6:43pm

Hi MJ, I did this and then restarted Opensearch, but the error remains the same. My understanding, based on the documentation, was that create_index included the indices:admin/create permissions already.

Also, I noticed in the output of the API roles call that the pscheduler_logstash role doesn’t have permissions for all the indices I expected:

"pscheduler_logstash":{"reserved":false,"hidden":false,"cluster_permissions":["cluster_monitor","cluster_manage_index_templates"],"index_permissions":[{"index_patterns":["pscheduler_*"],"fls":[],"masked_fields":[],"allowed_actions":["write","read","delete","create_index","manage","indices:admin/template/delete","indices:admin/template/get","indices:admin/template/put"]}],"tenant_permissions":[],"static":false}

For some reason, only pscheduler_* is listed here, prometheus_* is missing. And sure enough, the error complains about a lack of permissions for the prometheus_node index. But I don’t understand why prometheus_* doesn’t appear here.

thanks,

will

Mantas · July 25, 2024, 9:36am

@wleight, good call - could you add it and run securityadmin.sh to apply the changes see more here: Applying changes to configuration files - OpenSearch Documentation

best,
mj

wleight · July 26, 2024, 12:57pm

Hi @Mantas ,
Yes, that seems to have done it, thanks a lot!

will

Topic		Replies	Views
Internet user, permissions, roles and role mappings Security configure	3	560	March 3, 2022
Can't give access to specific indices to a user Security	1	970	December 10, 2021
Permissions weird Security	10	769	October 9, 2023
Getting error when giving user to specific index permission Security index-management	5	20	September 19, 2024
Read Only Accounts Security configure	5	354	February 14, 2024

Problem with index permissions

Related topics