Hi,
I have a 8 node opensearch cluster (3 master+5 data nodes). The cluster has internode TLS communication enabled on port 50141 and it accepts external requests on port 50140 also on TLS (https).
For the same query on the same cluster, I consistently see about 50x-100x higher latency with TLS (external+internode) than without it (no TLS external, no TLS internode).
For example, this index search query below takes about 10-35 secs with TLS enabled vs 150-300ms without TLS enabled :
‘curl -vv -u admin:xxxx -X GET “https://xxx.xxx.com:50140/pm_*/_search?size=0&pretty” -k’
without tls:
curl -vv -u admin:xxxx -X GET “http://xxx.xxx.com:50140/pm_*/_search?size=0&pretty”
All kinds of requests have such extremely high latencies, so this does not appear to be related to query of data type.
The cluster has about 680 indexes and 7 primary shards per index on average. Replica set to 1.
(I understand the shard density per node is quite high however the latency comparison here is for the same cluster).
Is such an extreme degradation expected?
security plugin config snippet:
`## Security Plugin Configurations ##
enable advanced features
plugins.security.advanced_modules_enabled: true
inter-node TLS configs
plugins.security.ssl.transport.keystore_filepath: opensearch-keystore.jks
plugins.security.ssl.transport.truststore_filepath: opensearch-truststore.jks
plugins.security.ssl.transport.enforce_hostname_verification: true
plugins.security.ssl.transport.resolve_hostname: true
plugins.security.ssl.transport.enabled_protocols:
- “TLSv1.3”
- “TLSv1.2”
plugins.security.nodes_dn: [“CN=dataxyz*”, “CN=masterxyz*”]
support dynamic management of the whitelisted nodes_dn
plugins.security.nodes_dn_dynamic_config_enabled: true
plugins.security.config_index_name: .security
plugins.security.allow_default_init_securityindex: true
secure client communication configs
plugins.security.ssl.http.enabled: True
plugins.security.ssl.http.keystore_filepath: opensearch-keystore.jks
plugins.security.ssl.http.truststore_filepath: opensearch-truststore.jks
plugins.security.ssl.http.clientauth_mode: OPTIONAL
plugins.security.ssl.http.enabled_protocols:
- “TLSv1.3”
- “TLSv1.2”
Enable role based access to the REST management API
plugins.security.restapi.roles_enabled: [“all_access”]
password policy
plugins.security.restapi.password_validation_regex: ‘(?=.[A-Z])(?=.[^a-zA-Z\d])(?=.[0-9])(?=.[a-z]).{8,}’
plugins.security.restapi.password_validation_error_message: “Password must be minimum 8 characters long and must contain at least one uppercase letter, one lowercase letter, one digit, and one special character.”
enable audit logs
plugins.security.audit.type: log4j
plugins.security.audit.config.log4j.logger_name: audit_log
plugins.security.audit.config.log4j.level: INFO`