./securityadmin.sh ERR: An unexpected IOException occured: Unrecognized SSL message, plaintext connection?

Simon · August 31, 2023, 9:26am

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch 2.9

Describe the issue:
[opensearch@e245aacc23f5 tools]$ ./securityadmin.sh -icl-nhnv -cacert …/…/…/config/root-ca.pem -cert …/…/…/config/admin.pem -key …/…/…/config/admin-key.pem

** This tool will be deprecated in the next major release of OpenSearch **
** [DEPRECATION] Security Plugin Tools will be replaced · Issue #1755 · opensearch-project/security · GitHub **

Security Admin v7
Will connect to localhost:9200 … done
ERR: An unexpected IOException occured: Unrecognized SSL message, plaintext connection?
Trace:
java.io.IOException: Unrecognized SSL message, plaintext connection?
at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:958)
at org.opensearch.client.RestClient.performRequest(RestClient.java:332)
at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:572)
at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:162)
Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at java.base/sun.security.ssl.SSLEngineInputRecord.bytesInCompletePacket(SSLEngineInputRecord.java:145)
at java.base/sun.security.ssl.SSLEngineInputRecord.bytesInCompletePacket(SSLEngineInputRecord.java:64)
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:612)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)
at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:279)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:333)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:545)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:833)

Configuration:
opensearch.yml -------------------------------------------------------------------------------------------------------
plugins.security.disabled: false
plugins.security.ssl.transport.pemcert_filepath: node1.pem
plugins.security.ssl.transport.pemkey_filepath: node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
- ‘CN=A,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA’
plugins.security.nodes_dn:
- ‘CN=node1.dns.a-record,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA’
- ‘CN=node2.dns.a-record,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA’
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
cluster.routing.allocation.disk.threshold_enabled: false
opendistro_security.audit.config.disabled_rest_categories: NONE
opendistro_security.audit.config.disabled_transport_categories: NONE
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [“.plugins-ml-model”, “.plugins-ml-task”, “.opendistro-alerting-config”, “.opendistro-alerting-alert”, “.opendistro-anomaly-results”, “.opendistro-anomaly-detector”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opensearch-notifications-”, “.opensearch-notebooks”, “.opensearch-observability”, “.opendistro-asynchronous-search-response”, “.replication-metadata-store”]

Relevant Logs or Screenshots:

pablo · August 31, 2023, 5:40pm

@Simon Try your command with the FQDN or IP address of your OpenSearch node.

./securityadmin.sh -icl -nhnv -h <OpenSearch_FQDN_or_IP> -cacert …/…/…/config/root-ca.pem -cert …/…/…/config/admin.pem -key …/…/…/config/admin-key.pem

How did you deploy your cluster?

Simon · September 4, 2023, 5:39am

i run with docker，this is my docker-compose.yml file

docker-compose.yml -----------------------------------------------------------------------------
version: ‘3’
services:
opensearch-node1:
image: opensearchproject/opensearch:2
container_name: opensearch-node1
environment:
- network.bind_host=0.0.0.0
- http.publish_host=0.0.0.0
- transport.publish_host=0.0.0.0
- cluster.name=opensearch-cluster
- node.name=opensearch-node1
- discovery.type=single-node
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- “OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m” # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
hard: 65536
volumes:
- opensearch-data1:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
- ./logs:/usr/share/opensearch/logs
- ./root-ca.pem:/usr/share/opensearch/config/root-ca.pem
- ./admin.pem:/usr/share/opensearch/config/admin.pem
- ./admin-key.pem:/usr/share/opensearch/config/admin-key.pem
- ./node1.pem:/usr/share/opensearch/config/node1.pem
- ./node1-key.pem:/usr/share/opensearch/config/node1-key.pem
- ./opensearch.yml:/usr/share/opensearch/config/opensearch.yml
- ./opensearch_dashboards.yml:/usr/share/opensearch/config/opensearch_dashboards.yml
- ./internal_users.yml:/usr/share/opensearch/config/opensearch-security/internal_users.yml
ports:
- 9200:9200
- 9300:9300
- 9600:9600 # required for Performance Analyzer
networks:
- opensearch-net

opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:2
container_name: opensearch-dashboards
ports:
- 5601:5601
expose:
- “5601”
environment:
- ‘OPENSEARCH_HOSTS=[“http://opensearch-node1:9200”]’
networks:
- opensearch-net
volumes:
- ./opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml
- ./logs:/usr/share/opensearch/logs
- ./root-ca.pem:/usr/share/opensearch-dashboards/config/root-ca.pem
- ./admin.pem:/usr/share/opensearch-dashboards/config/admin.pem
- ./admin-key.pem:/usr/share/opensearch-dashboards/config/admin-key.pem
- ./node1.pem:/usr/share/opensearch-dashboards/config/node1.pem
- ./node1-key.pem:/usr/share/opensearch-dashboards/config/node1-key.pem
- ./node2.pem:/usr/share/opensearch/config/node2.pem
- ./node2-key.pem:/usr/share/opensearch/config/node2-key.pem

volumes:
opensearch-data1:

networks:
opensearch-net:

Simon · September 4, 2023, 7:51am

@pablo thx for your reply

pablo · September 4, 2023, 9:44am

@Simon Just rechecked your opensearch.yml. Your HTTP/API endpoint (port 9200) is unsecured.

That’s why securityadmin.sh producing reported errors.

You’re missing these lines in your opensearch.yml file.

plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: node1.pem
plugins.security.ssl.http.pemkey_filepath: node1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem

Please be aware that port 9200 must be secured if you’d like to use securityadmin.sh to backup and upload the configuration. The securitydamin.sh is using certificate authentication to communicate with the OpenSearch cluster.

Simon · September 5, 2023, 3:10am

@pablo when i add these lines to mine opensearch.yml file

plugins.security.disabled: false
plugins.security.ssl.transport.pemcert_filepath: node1.pem
plugins.security.ssl.transport.pemkey_filepath: node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: true
## here
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: admin.pem
plugins.security.ssl.http.pemkey_filepath: admin-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem

plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - 'CN=A,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
plugins.security.nodes_dn:
  - 'CN=node1.dns.a-record,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
  - 'CN=node2.dns.a-record,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
opendistro_security.audit.config.disabled_rest_categories: NONE
opendistro_security.audit.config.disabled_transport_categories: NONE

plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

plugins.security.ssl_cert_reload_enabled: true

[opensearch-node1] show error like this:

pablo · September 5, 2023, 2:03pm

These options require node certificate and key.

Try these instead.

Simon · September 8, 2023, 7:24am

After modified
this mine new opensearch.yml file

plugins.security.disabled: false
plugins.security.ssl.transport.pemcert_filepath: node1.pem
plugins.security.ssl.transport.pemkey_filepath: node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: true
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: node1.pem
plugins.security.ssl.http.pemkey_filepath: node1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - 'CN=A,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
plugins.security.nodes_dn:
  - 'CN=node1.dns.a-record,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
  - 'CN=node2.dns.a-record,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
opendistro_security.audit.config.disabled_rest_categories: NONE
opendistro_security.audit.config.disabled_transport_categories: NONE

plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

plugins.security.ssl_cert_reload_enabled: true

show this:

Gsmitt · September 22, 2023, 4:56am

Hey @Simon

Have you seen this? If not this is what need to happen towards the bottom of the page.

Not sure if you have a NameServer but it should match

plugins.security.authcz.admin_dn:
  - 'CN=A,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
plugins.security.nodes_dn:
  - 'CN=node1.dns.a-record,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
  - 'CN=node2.dns.a-record,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'

Note sure if you creating your own certificates but you should and remove the default ones. A good example would be from Creating Certificates for Opensearch

plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/node1.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/node1.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/node1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - 'CN=opensearch.domain.com,OU=admin,O=enseva,L=cedar,ST=iowa,C=us'
plugins.security.nodes_dn:
  - 'CN=opensearch.domain.com,OU=admin,O=enseva,L=ceda,ST=iowa,C=us'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]

pablo · September 22, 2023, 9:11am

@Simon I think your issue is related to your docker-compose.yaml file.
You have the following configuration in the opensearch-dashboards section.

It should be https://opensearch-node1:9200 as plugins.security.ssl.http.enabled is set to true in opensearch.yml.

Topic		Replies	Views
Securityadmin.sh errors when connecting opensearch through http Security troubleshoot	4	2824	June 2, 2022
Query: To run securityadmin.sh when TLS is disabled on Rest layer Security	1	471	July 18, 2022
Opensearch secuirty admin process is failing , when opensearch is deployed by configuring available ciphers OpenSearch install	3	65	April 23, 2024
The security admin process in OpenSearch fails when deployed with configured ciphers OpenSearch install , security-issue	1	49	April 23, 2024
Securityadmin.sh unable to find valid certification path to requested target error Security configure , security-issue	2	397	November 1, 2023

./securityadmin.sh ERR: An unexpected IOException occured: Unrecognized SSL message, plaintext connection?

Related Topics