Describe the issue:
When activating security the cpu usage on our coordinating nodes which are the first contact of the incoming bulk requests spike up to 100%+ . Without enabled security they are running with ~5-10% cpu usage.
Configuration:
User which performs the bulk requests has all_access role.
Bulk Size 5000
Indices ~9000
User is internal user and basic_internal_auth_domain has prio 0, ldap is configured with prio 5
Security was activated on the cpu spikes and got deactivated arrount 0:00
Output of hot_threads of one of the coordinating nodes:
47.3% (236.6ms out of 500ms) cpu usage by thread 'opensearch[hostname][transport_worker][T#12]'
6/10 snapshots sharing following 110 elements
app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:233)
app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:160)
org.opensearch.security.securityconf.ConfigModelV7$IndexPattern.getResolvedIndexPattern(ConfigModelV7.java:827)
org.opensearch.security.securityconf.ConfigModelV7$IndexPattern.attemptResolveIndexNames(ConfigModelV7.java:791)
org.opensearch.security.securityconf.ConfigModelV7.lambda$impliesTypePerm$4(ConfigModelV7.java:1062)
org.opensearch.security.securityconf.ConfigModelV7$$Lambda$5257/0x00000008013f3040.apply(Unknown Source)
java.base@11.0.16.1/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
java.base@11.0.16.1/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1621)
java.base@11.0.16.1/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
java.base@11.0.16.1/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
java.base@11.0.16.1/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:550)
java.base@11.0.16.1/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
java.base@11.0.16.1/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:517)
org.opensearch.security.securityconf.ConfigModelV7.impliesTypePerm(ConfigModelV7.java:1063)
org.opensearch.security.securityconf.ConfigModelV7$SecurityRoles.impliesTypePermGlobal(ConfigModelV7.java:516)
org.opensearch.security.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:524)
org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:377)
org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165)
Are we configuring something wrong? As the user has all_access the permission check should be quite fast?
its an adapted default config which is running in other cluster quite fine - it seams that the amout of log ingest is just to much for the coordination nodes with enabled security:
Config:
---
# This is the main Open Distro Security configuration file where authentication
# and authorization is defined.
#
# You need to configure at least one authentication domain in the authc of this file.
# An authentication domain is responsible for extracting the user credentials from
# the request and for validating them against an authentication backend like Active Directory for example.
#
# If more than one authentication domain is configured the first one which succeeds wins.
# If all authentication domains fail then the request is unauthenticated.
# In this case an exception is thrown and/or the HTTP status is set to 401.
#
# After authentication authorization (authz) will be applied. There can be zero or more authorizers which collect
# the roles from a given backend for the authenticated user.
#
# Both, authc and auth can be enabled/disabled separately for REST and TRANSPORT layer. Default is true for both.
# http_enabled: true
# transport_enabled: true
#
# For HTTP it is possible to allow anonymous authentication. If that is the case then the HTTP authenticators try to
# find user credentials in the HTTP request. If credentials are found then the user gets regularly authenticated.
# If none can be found the user will be authenticated as an "anonymous" user. This user has always the username "anonymous"
# and one role named "anonymous_backendrole".
# If you enable anonymous authentication all HTTP authenticators will not challenge.
#
#
# Note: If you define more than one HTTP authenticators make sure to put non-challenging authenticators like "proxy" or "clientcert"
# first and the challenging one last.
# Because it's not possible to challenge a client with two different authentication methods (for example
# Kerberos and Basic) only one can have the challenge flag set to true. You can cope with this situation
# by using pre-authentication, e.g. sending a HTTP Basic authentication header in the request.
#
# Default value of the challenge flag is true.
#
#
# HTTP
# basic (challenging)
# proxy (not challenging, needs xff)
# kerberos (challenging)
# clientcert (not challenging, needs https)
# jwt (not challenging)
# host (not challenging) #DEPRECATED, will be removed in a future version.
# host based authentication is configurable in roles_mapping
# Authc
# internal
# noop
# ldap
# Authz
# ldap
# noop
_meta:
type: "config"
config_version: 2
config:
dynamic:
# Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
# Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
# Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
#filtered_alias_mode: warn
#do_not_fail_on_forbidden: false
#kibana:
# Kibana multitenancy
#multitenancy_enabled: true
#server_username: kibanaserver
#index: '.kibana'
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
#internalProxies: '.*' # trust all internal proxies, regex pattern
#remoteIpHeader: 'x-forwarded-for'
###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
###### and here https://tools.ietf.org/html/rfc7239
###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
authc:
kerberos_auth_domain:
http_enabled: false
transport_enabled: false
order: 6
http_authenticator:
type: kerberos
challenge: true
config:
# If true a lot of kerberos/security related debugging output will be logged to standard out
krb_debug: false
# If true then the realm will be stripped from the user name
strip_realm_from_principal: true
authentication_backend:
type: noop
basic_internal_auth_domain:
description: "Authenticate via HTTP Basic against internal users database"
http_enabled: true
transport_enabled: true
order: 4
http_authenticator:
type: basic
challenge: true
authentication_backend:
type: intern
proxy_auth_domain:
description: "Authenticate via proxy"
http_enabled: false
transport_enabled: false
order: 3
http_authenticator:
type: proxy
challenge: false
config:
user_header: "x-proxy-user"
roles_header: "x-proxy-roles"
authentication_backend:
type: noop
jwt_auth_domain:
description: "Authenticate via Json Web Token"
http_enabled: false
transport_enabled: false
order: 0
http_authenticator:
type: jwt
challenge: false
config:
signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
jwt_header: "Authorization"
jwt_url_parameter: null
roles_key: null
subject_key: null
authentication_backend:
type: noop
clientcert_auth_domain:
description: "Authenticate via SSL client certificates"
http_enabled: false
transport_enabled: false
order: 2
http_authenticator:
type: clientcert
config:
username_attribute: cn #optional, if omitted DN becomes username
challenge: false
authentication_backend:
type: noop
ldap:
description: "Authenticate via LDAP or Active Directory"
http_enabled: true
transport_enabled: true
order: 1
http_authenticator:
type: basic
challenge: false
authentication_backend:
# LDAP authentication backend (authenticate users against a LDAP or Active Directory)
type: ldap
config:
# enable ldaps
enable_ssl: true
# enable start tls, enable_ssl should be false
enable_start_tls: false
# send client certificate
enable_ssl_client_auth: false
# verify ldap hostname
verify_hostnames: true
hosts:
- $LDAPSERVER
bind_dn: $LDAPUSER
password: $LDAPPASS
userbase: 'cn=enterprise,o=something,c=de'
# Filter to search for users (currently in the whole subtree beneath userbase)
# {0} is substituted with the username
usersearch: '(uid={0})'
# Use this attribute from the user as username (if not set then DN is used)
username_attribute: cn
authz:
roles_from_myldap:
description: "Authorize via LDAP or Active Directory"
http_enabled: true
transport_enabled: true
authorization_backend:
# LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
type: ldap
config:
# enable ldaps
enable_ssl: true
# enable start tls, enable_ssl should be false
enable_start_tls: false
# send client certificate
enable_ssl_client_auth: false
# verify ldap hostname
verify_hostnames: true
hosts:
- $LDAPSERVER
bind_dn: $LDAPUSER
password: $LDAPPASS
rolebase: 'ou=corporation,cn=enterprise,o=something,c=de'
# Filter to search for roles (currently in the whole subtree beneath rolebase)
# {0} is substituted with the DN of the user
# {1} is substituted with the username
# {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
rolesearch: '(member={0})'
# Specify the name of the attribute which value should be substituted with {2} above
userroleattribute: null
# Roles as an attribute of the user entry
userrolename: disabled
#userrolename: memberOf
# The attribute in a role entry containing the name of that role, Default is "name".
# Can also be "dn" to use the full DN as rolename.
rolename: cn
# Resolve nested roles transitive (roles which are members of other roles and so on ...)
resolve_nested_roles: false
userbase: 'cn=something,o=something,c=de'
# Filter to search for users (currently in the whole subtree beneath userbase)
# {0} is substituted with the username
usersearch: '(uid={0})'
# Skip users matching a user name, a wildcard or a regex pattern
skip_users:
- kibanaserver
# - 'cn=Michael Jackson,ou*people,o=TEST'
# - '/\S*/'
roles_from_another_ldap:
description: "Authorize via another Active Directory"
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap
#config goes here ...
# auth_failure_listeners:
# ip_rate_limiting:
# type: ip
# allowed_tries: 10
# time_window_seconds: 3600
# block_expiry_seconds: 600
# max_blocked_clients: 100000
# max_tracked_clients: 100000
# internal_authentication_backend_limiting:
# type: username
# authentication_backend: intern
# allowed_tries: 10
# time_window_seconds: 3600
# block_expiry_seconds: 600
# max_blocked_clients: 100000
# max_tracked_clients: 100000
