I am using opendistro 1.13.2 version and facing around 50% performance dip in search query when enabling the opendistro security setting compared to security disabled

While creating the Index performance number looks good but when searching the index I see around 50% dip in performance
when security is enabled compared to disabled security setting.

I am using java API client for search query.(elasticsearch-rest-client).

Am I missing somthing is configuration which causing this performance dip ?

below is my security setting in yml file


opendistro_security.disabled: false

opendistro_security.ssl.transport.enabled: true
opendistro_security.ssl.transport.keystore_type: pkcs12
opendistro_security.ssl.transport.keystore_filepath: …/config/certs/abc-keystore.p12
opendistro_security.ssl.transport.keystore_password: ******
opendistro_security.ssl.transport.truststore_type: pkcs12
opendistro_security.ssl.transport.truststore_filepath: …/config/certs/abc-truststore.p12
opendistro_security.ssl.transport.truststore_password: *****
opendistro_security.nodes_dn:

  • CN=client.abc.example.com,OU=ABC Technology Platform,O=ABC,DC=client,DC=abc,DC=example,DC=com’
  • ‘CN=*.example.com,OU=ABC Technology Platform,O=ABC,DC=client,DC=abc,DC=example,DC=com’
  • ‘/CN=.*regex/’

opendistro_security.authcz.admin_dn:

opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.transport.resolve_hostname: false
opendistro_security.allow_default_init_securityindex: true

opendistro_security.ssl.http.enabled: false
opendistro_security.ssl.http.keystore_type: pkcs12
opendistro_security.ssl.http.keystore_filepath: …/config/certs/abc-keystore.p12
opendistro_security.ssl.http.keystore_password: *******
opendistro_security.ssl.http.truststore_type: pkcs12
opendistro_security.ssl.http.truststore_filepath: …/config/certs/abc-truststore.p12
opendistro_security.ssl.http.truststore_password: ******

@szwlhd
Could you share your config.yml, role configuration of logged in user and roles mappings?
Does it happen for any search query?
Did you always have that performance issue or did it appear recently?

I did not find otion to attach the file so pasting my config.yml details below.
Yes it happens for any search query.
Always I had this performance issue its not new.

my config.yml file is as below and I am using “admin” user for all the query


_meta:
type: “config”
config_version: 2

config:
dynamic:
# Set filtered_alias_mode to ‘disallow’ to forbid more than 2 filtered aliases per index
# Set filtered_alias_mode to ‘warn’ to allow more than 2 filtered aliases per index but warns about it (default)
# Set filtered_alias_mode to ‘nowarn’ to allow more than 2 filtered aliases per index silently
#filtered_alias_mode: warn
#do_not_fail_on_forbidden: false
#kibana:
# Kibana multitenancy
#multitenancy_enabled: true
#server_username: kibanaserver
#index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: ‘192.168.0.10|192.168.0.11’ # regex pattern
#internalProxies: ‘.’ # trust all internal proxies, regex pattern
#remoteIpHeader: ‘x-forwarded-for’
###### see Pattern (Java Platform SE 7 ) for regex help
###### more information about XFF X-Forwarded-For - Wikipedia
###### and here RFC 7239: Forwarded HTTP Extension
###### and Apache Tomcat 8 Configuration Reference (8.0.53) - The Valve Component
authc:
kerberos_auth_domain:
http_enabled: false
transport_enabled: false
order: 6
http_authenticator:
type: kerberos
challenge: true
config:
# If true a lot of kerberos/security related debugging output will be logged to standard out
krb_debug: false
# If true then the realm will be stripped from the user name
strip_realm_from_principal: true
authentication_backend:
type: noop
basic_internal_auth_domain:
description: “Authenticate via HTTP Basic against internal users database”
http_enabled: true
transport_enabled: true
order: 4
http_authenticator:
type: basic
challenge: true
authentication_backend:
type: intern
proxy_auth_domain:
description: “Authenticate via proxy”
http_enabled: false
transport_enabled: false
order: 3
http_authenticator:
type: proxy
challenge: false
config:
user_header: “x-proxy-user”
roles_header: “x-proxy-roles”
authentication_backend:
type: noop
jwt_auth_domain:
description: “Authenticate via Json Web Token”
http_enabled: false
transport_enabled: false
order: 0
http_authenticator:
type: jwt
challenge: false
config:
signing_key: “base64 encoded HMAC key or public RSA/ECDSA pem key”
jwt_header: “Authorization”
jwt_url_parameter: null
roles_key: null
subject_key: null
authentication_backend:
type: noop
clientcert_auth_domain:
description: “Authenticate via SSL client certificates”
http_enabled: true
transport_enabled: true
order: 2
http_authenticator:
type: clientcert
config:
username_attribute: cn #optional, if omitted DN becomes username
challenge: false
authentication_backend:
type: noop
ldap:
description: “Authenticate via LDAP or Active Directory”
http_enabled: false
transport_enabled: false
order: 5
http_authenticator:
type: basic
challenge: false
authentication_backend:
# LDAP authentication backend (authenticate users against a LDAP or Active Directory)
type: ldap
config:
# enable ldaps
enable_ssl: false
# enable start tls, enable_ssl should be false
enable_start_tls: false
# send client certificate
enable_ssl_client_auth: false
# verify ldap hostname
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
userbase: ‘ou=people,dc=example,dc=com’
# Filter to search for users (currently in the whole subtree beneath userbase)
# {0} is substituted with the username
usersearch: ‘(sAMAccountName={0})’
# Use this attribute from the user as username (if not set then DN is used)
username_attribute: null
authz:
roles_from_myldap:
description: “Authorize via LDAP or Active Directory”
http_enabled: false
transport_enabled: false
authorization_backend:
# LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
type: ldap
config:
# enable ldaps
enable_ssl: false
# enable start tls, enable_ssl should be false
enable_start_tls: false
# send client certificate
enable_ssl_client_auth: false
# verify ldap hostname
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
rolebase: ‘ou=groups,dc=example,dc=com’
# Filter to search for roles (currently in the whole subtree beneath rolebase)
# {0} is substituted with the DN of the user
# {1} is substituted with the username
# {2} is substituted with an attribute value from user’s directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
rolesearch: ‘(member={0})’
# Specify the name of the attribute which value should be substituted with {2} above
userroleattribute: null
# Roles as an attribute of the user entry
userrolename: disabled
#userrolename: memberOf
# The attribute in a role entry containing the name of that role, Default is “name”.
# Can also be “dn” to use the full DN as rolename.
rolename: cn
# Resolve nested roles transitive (roles which are members of other roles and so on …)
resolve_nested_roles: true
userbase: ‘ou=people,dc=example,dc=com’
# Filter to search for users (currently in the whole subtree beneath userbase)
# {0} is substituted with the username
usersearch: ‘(uid={0})’
# Skip users matching a user name, a wildcard or a regex pattern
#skip_users:
# - 'cn=Michael Jackson,ou
people,o=TEST’
# - ‘/\S*/’
roles_from_another_ldap:
description: “Authorize via another Active Directory”
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap
#config goes here …


Adding role configuration as well .

_meta:
type: “roles”
config_version: 2

Restrict users so they can only view visualization and dashboard on kibana

kibana_read_only:
reserved: true

The security REST API access role is used to assign specific users access to change the security settings through the REST API.

security_rest_api_access:
reserved: true

Allows users to view monitors, destinations and alerts

alerting_read_access:
reserved: true
cluster_permissions:
- ‘cluster:admin/opendistro/alerting/alerts/get’
- ‘cluster:admin/opendistro/alerting/destination/get’
- ‘cluster:admin/opendistro/alerting/monitor/get’
- ‘cluster:admin/opendistro/alerting/monitor/search’

Allows users to view and acknowledge alerts

alerting_ack_alerts:
reserved: true
cluster_permissions:
- ‘cluster:admin/opendistro/alerting/alerts/*’

Allows users to use all alerting functionality

alerting_full_access:
reserved: true
cluster_permissions:
- ‘cluster_monitor’
- ‘cluster:admin/opendistro/alerting/
index_permissions:
- index_patterns:
- '

allowed_actions:
- ‘indices_monitor’
- ‘indices:admin/aliases/get’
- ‘indices:admin/mappings/get’

Allow users to read Anomaly Detection detectors and results

anomaly_read_access:
reserved: true
cluster_permissions:
- ‘cluster:admin/opendistro/ad/detector/info’
- ‘cluster:admin/opendistro/ad/detector/search’
- ‘cluster:admin/opendistro/ad/detectors/get’
- ‘cluster:admin/opendistro/ad/result/search’
- ‘cluster:admin/opendistro/ad/tasks/search’

Allows users to use all Anomaly Detection functionality

anomaly_full_access:
reserved: true
cluster_permissions:
- ‘cluster_monitor’
- ‘cluster:admin/opendistro/ad/
index_permissions:
- index_patterns:
- '

allowed_actions:
- ‘indices_monitor’
- ‘indices:admin/aliases/get’
- ‘indices:admin/mappings/get’

Allows users to read Notebooks

notebooks_read_access:
reserved: true
cluster_permissions:
- ‘cluster:admin/opendistro/notebooks/list’
- ‘cluster:admin/opendistro/notebooks/get’

Allows users to all Notebooks functionality

notebooks_full_access:
reserved: true
cluster_permissions:
- ‘cluster:admin/opendistro/notebooks/create’
- ‘cluster:admin/opendistro/notebooks/update’
- ‘cluster:admin/opendistro/notebooks/delete’
- ‘cluster:admin/opendistro/notebooks/get’
- ‘cluster:admin/opendistro/notebooks/list’

Allows users to read and download Reports

reports_instances_read_access:
reserved: true
cluster_permissions:
- ‘cluster:admin/opendistro/reports/instance/list’
- ‘cluster:admin/opendistro/reports/instance/get’
- ‘cluster:admin/opendistro/reports/menu/download’

Allows users to read and download Reports and Report-definitions

reports_read_access:
reserved: true
cluster_permissions:
- ‘cluster:admin/opendistro/reports/definition/get’
- ‘cluster:admin/opendistro/reports/definition/list’
- ‘cluster:admin/opendistro/reports/instance/list’
- ‘cluster:admin/opendistro/reports/instance/get’
- ‘cluster:admin/opendistro/reports/menu/download’

Allows users to all Reports functionality

reports_full_access:
reserved: true
cluster_permissions:
- ‘cluster:admin/opendistro/reports/definition/create’
- ‘cluster:admin/opendistro/reports/definition/update’
- ‘cluster:admin/opendistro/reports/definition/on_demand’
- ‘cluster:admin/opendistro/reports/definition/delete’
- ‘cluster:admin/opendistro/reports/definition/get’
- ‘cluster:admin/opendistro/reports/definition/list’
- ‘cluster:admin/opendistro/reports/instance/list’
- ‘cluster:admin/opendistro/reports/instance/get’
- ‘cluster:admin/opendistro/reports/menu/download’

Allows users to use all asynchronous-search functionality

asynchronous_search_full_access:
reserved: true
cluster_permissions:
- ‘cluster:admin/opendistro/asynchronous_search/
index_permissions:
- index_patterns:
- '

allowed_actions:
- ‘indices:data/read/search*’

Allows users to read stored asynchronous-search results

asynchronous_search_read_access:
reserved: true
cluster_permissions:
- ‘cluster:admin/opendistro/asynchronous_search/get’

@szwlhd The config looks fine. Have you tried to run queries with basic authentication instead of certificate one?

Could you tell me more about your infrastructure?

  • type of deployment? (physical, docker, Kubernetes)
  • number of nodes?
  • JAVA Xms and Xmx values?
  • number of CPUs and RAM size assigned to a single node or virtual machine?

Just to give more info. I tried my query with Elasticsearch version 7.10.2 with opendistro security enabled. then again I tried the same query with same infrastructure after disabling the opendistro security layer by parameter opendistro_security.disabled: true. And noticed dip in performance number.

Type of deployment physical
I have tried on single node and 3 node as well results are similar
JAVA Xms and Xmx values 16GB

CPU info

-bash-4.2$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
Stepping: 1
CPU MHz: 1200.527
BogoMIPS: 4199.91
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 20480K
NUMA node0 CPU(s): 0-15


Ram Info

-bash-4.2$ free -m
total used free shared buff/cache available
Mem: 31898 12282 720 25 18895 19183
Swap: 7999 0 7999

@szwlhd Could you share examples of the query ran in ODFE with enabled and disabled security plugin and the stats?

Index creation dataset info.

Performance is fine with index creation. I am putting these details FYI it may help you to understand the scenario.

Text file with 25 column and 10 million records

Creating index on 4 column out of 25 column

Creating index in batchmode with batch side 30000 and where primary key is column GNAF_PID.

Below is query for creating the index.

String endpoint = “/” + indexName + “?include_type_name=false”;

HttpEntity entity = new StringEntity( body , ContentType.APPLICATION_JSON);

Request request = new Request(RestCallConstants.PUT, endpoint);

request.setEntity(entity);

Response response = restClientFactory.getClient().performRequest(request);

where body value is

{“settings”:{“index”:{“number_of_shards”:1,“number_of_replicas”:0}},“mappings”:{“properties”:{“lat1”:{“type”:“text”,“index”:false},“lat2”:{“type”:“text”,“index”:false},

“lat3”:{“type”:“text”,“index”:false},

“lat4”:{“type”:“text”,“index”:false},

“Latitude”:{“type”:“text”, “index”:true," analyzer":“standard”},

“NewField”:{“type”:“text”,“index”:false},

“AD10”:{“type”:“text”, “index”:true," analyzer":“standard”},

“Building_Name”:{“type”:“text”,“index”:false},

“PC1”:{“type”:“text”,“index”:false},

“BN1”:{“type”:“text “,“index”:true,” analyzer”:“standard”},

“BN3”:{“type”:“text”,“index”:false},“BN2”:{“type”:“text”,“index”:false},

“BN4”:{“type”:“text”,“index”:false},“Longitude”:{“type”:“text”,“index”:false},

“Postcode”:{“type”:“text”,“index”:false},“AD2”:{“type”:“text”,“index”:false},

“ADC1”:{“type”:“text”,“index”:false},“LLC6”:{“type”:“text”,“index”:false},

“AddressLine1”:{“type”:“text”,“index”:false},

“GNAF_PID”:{“type”:"keyword ",“index”:true },

“AD9”:{“type”:“text”,“index”:false},

“PC11”:{“type”:“text”,“index”:false},

“PC10”:{“type”:“text”,“index”:false},

“NF1”:{“type”:“text”,“index”:false},

“NF4”:{“type”:“text”,“index”:false}}}}

Now search query details which has performance issue.

I am doing search on above created index. Search file has 1 million records and doing search on 3 column.in batchmode with batch size 10000 and max result count 50.

Query is as below

………………………………

httpEntity = new StringEntity( requestBody , ContentType.APPLICATION_JSON);

Request request = new Request(RestCallConstants.GET, endPoint);

request.setEntity(httpEntity);

response = restClientFactory.getClient().performRequest(request);

requestBody

{“query”:{“bool”:{“must”:[{“match_all”:{}}],

“filter”:{“bool”:{“must”:[{“match_phrase”:{“AD10”:{“query”:“14 CHERMSIDE STREET1770”,“boost”:1.0}}},

{“term”:{“BN1”:{“boost”:1.0,“value”:“null1770”}}},

{“term”:{“GNAF_PID”:{“boost”:1.0,“value”:“GAACT7148916451770”}}}]}}}},

“from”:0,“size”:50}

…………………………….

Performance number for search query

Process 1 million records

Max count 50

Batch size 10000

Other infra structure and memory is same for both.

With opendistro security enabled With opendistro security disabled
54.2 min 38.5 min

@szwlhd Thanks for sharing. Are these both environments are production?
Is sharding the same on both indexes?

Not production but similar to production. Yes sharding are same

@szwlhd Have you tried the same scenario with the latest OpenSearch?

What authentication type do you use when querying ES? (baseauth or cert)
If cert could you try baseauth with the internal user instead?

Can you guide me where I can check which authentication type I am using while Querying ES? I have shared elasticsearch.yml config.yml and roles.yml files in above thread if that will help in finding which type of authentication I am using

@szwlhd Based on your config.yml you have clientcert_auth_domain and basic_internal_auth_domain enabled. I can’t see your authentication process in the query so can’t say which one do you use to query ES.

You could try either disabling cliencert_auth_domain or reversing the order and making the basic_internal_auth_domain first.

I disabled cliencert_auth_domain and ran the query took 52.39 Min almost same result.
You guys dont have performance testcase where you can quickly check these scenario ?

@szwlhd Have you tried Performance Analyzer? API - OpenSearch documentation

No I have not used performance analyzer.


I configured performance analyzer but no statistic is coming Am I doing anything wrong?

@szwlhd perfTop needs access to port 9600 of the ES node.
Also, you must enabled performance analyzer in the cluster.

POST _plugins/_performanceanalyzer/config 
{
  "enabled" : true
}

I’m not an expert in this tool however, there is a separate forum group that might answer more detailed questions.

Would you mind sharing roles_mapping.yml file?
Is there any reason why you’re still on ODFE 1.13.2? Did you consider testing OpenSearch instead?

Please find the roles_mapping.yml details

_meta:
type: “rolesmapping”
config_version: 2

.# Define your roles mapping here

.## Demo roles mapping

all_access:
reserved: false
backend_roles:

own_index:
reserved: false
users:

  • “*”
    description: “Allow full access to an index named like the username”

logstash:
reserved: false
backend_roles:

  • “logstash”

kibana_user:
reserved: false
backend_roles:

  • “kibanauser”
    description: “Maps kibanauser to kibana_user”

readall:
reserved: false
backend_roles:

  • “readall”

manage_snapshots:
reserved: false
backend_roles:

  • “snapshotrestore”

kibana_server:
reserved: true
users:

  • “kibanaserver”

OpenSearch not having window artifact and our application support window as well. Also I dont think opensearch will not have this issue as per my understanding ODFE and OpenSearch both have same architecture.

@szwlhd OpenSearch is a successor of ODFE and is constantly developed since 2021.

The latest version 1.3.0 has been released on the 15th of March.

1 Like