Docker OpenSearch : Wildcard SSL TLS handshake issue

Security
1

Hello everyone,

I’m trying to configure WILDCARD certificate on a new cluster OpenSearch

There is the error i’m blocked in

Can someone help me found why ? :confused:

[2022-04-10T07:24:56,363][WARN ][o.o.d.HandshakingTransportAddressConnector] [os01.orgx.com] handshake failed for [connectToRemoteMasterNode[x.x.x.x:9300]]
org.opensearch.transport.RemoteTransportException: [os03.orgx.com][x.x.x.x:9300][internal:transport/handshake]
Caused by: org.opensearch.OpenSearchException: Illegal parameter in http or transport request found.
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/

There is my actual config

OpenSearch-Dashboard.yml

server.name: os_dashboards.orgx.com
server.host: "0.0.0.0"

opensearch.username: "admin"
opensearch.password: "admin"

# Encrypt traffic between the browser and OpenSearch-Dashboards
server.ssl.enabled: true
server.ssl.certificate: "/usr/share/opensearch-dashboards/config/node.pem"
server.ssl.key: "/usr/share/opensearch-dashboards/config/node-key.pem"

# Encrypt traffic between OpenSearch-Dashboards and Opensearch
opensearch.ssl.certificateAuthorities: ["/usr/share/opensearch-dashboards/config/root-ca.pem"]
opensearch.ssl.verificationMode: full

OpenSearch.yml

cluster.name: os-cluster.orgx.com
network.host: 0.0.0.0

bootstrap.memory_lock: "true" # along with the memlock settings below, disables swapping

cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 93%
cluster.routing.allocation.disk.watermark.high: 95%

plugins.security.allow_unsafe_democertificates: true
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemtrustedcas_filepath: node.pem
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false

plugins.security.authcz.admin_dn:
  - "CN=*.orgx.com,OU=X,O=XZ,ST=XT,C=XY"
plugins.security.nodes_dn:
  - "CN=os01.orgx.com,OU=X,O=XZ,ST=XT,C=XY"
  - "CN=os02.orgx.com,OU=X,O=XZ,ST=XT,C=XY"
  - "CN=os03.orgx.com,OU=X,O=XZ,ST=XT,C=XY"

The docker file :

version: '3.1'

services:

  os01.orgx.com:
    restart: always
    image: opensearchproject/opensearch:1.3.1
    environment:
      OPENSEARCH_JAVA_OPTS: "-Xms16g -Xmx16g" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      node.name: os01.orgx.com
      discovery.seed_hosts: os01.orgx.com,os02.orgx.com,os03.orgx.com
      cluster.initial_master_nodes: os01.orgx.com,os02.orgx.com,os03.orgx.com
      plugins.security.ssl.transport.pemkey_filepath: node-key.pem # relative path
      plugins.security.ssl.transport.pemcert_filepath: node.pem
      plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
      plugins.security.ssl.http.pemkey_filepath: node-key.pem
      plugins.security.ssl.http.pemcert_filepath: node.pem
      plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
      DISABLE_INSTALL_DEMO_CONFIG: "true"
      JAVA_HOME: /usr/share/opensearch/jdk
      bootstrap.memory_lock: "true" # along with the memlock settings below, disables swapping
      network.host: "0.0.0.0"
    ulimits: 
      memlock:
        soft: -1
        hard: -1
    volumes:
      - "./OpenSearch/opensearch.yml:/usr/share/opensearch/config/opensearch.yml"
      - "./OpenSearch/internal_users.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml"
      - "./OpenSearch/roles_mapping.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml"
      - "./OpenSearch/tenants.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml"
      - "./OpenSearch/roles.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml"
      - "./OpenSearch/action_groups.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml"
      - "./OpenSearch/DATA1:/usr/share/opensearch/data"
      - "./certs/root-ca.pem:/usr/share/opensearch/config/root-ca.pem"
      - "./certs/node.pem:/usr/share/opensearch/config/node.pem"
      - "./certs/node-key.pem:/usr/share/opensearch/config/node-key.pem"
      - "./certs/admin.pem:/usr/share/opensearch/config/admin.pem"
      - "./certs/admin-key.pem:/usr/share/opensearch/config/admin-key.pem"
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
  
  os02.orgx.com:
    restart: always
    image: opensearchproject/opensearch:1.3.1
    environment:
      OPENSEARCH_JAVA_OPTS: "-Xms16g -Xmx16g" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      node.name: os02.orgx.com
      discovery.seed_hosts: os01.orgx.com,os02.orgx.com,os03.orgx.com
      cluster.initial_master_nodes: os01.orgx.com,os02.orgx.com,os03.orgx.com
      plugins.security.ssl.transport.pemkey_filepath: node-key.pem # relative path
      plugins.security.ssl.transport.pemcert_filepath: node.pem
      plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
      plugins.security.ssl.http.pemkey_filepath: node-key.pem
      plugins.security.ssl.http.pemcert_filepath: node.pem
      plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
      DISABLE_INSTALL_DEMO_CONFIG: "true"
      JAVA_HOME: /usr/share/opensearch/jdk
      bootstrap.memory_lock: "true" # along with the memlock settings below, disables swapping
      network.host: "0.0.0.0"
    ulimits: 
      memlock:
        soft: -1
        hard: -1
    volumes:
      - "./OpenSearch/opensearch.yml:/usr/share/opensearch/config/opensearch.yml"
      - "./OpenSearch/internal_users.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml"
      - "./OpenSearch/roles_mapping.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml"
      - "./OpenSearch/tenants.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml"
      - "./OpenSearch/roles.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml"
      - "./OpenSearch/action_groups.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml"
      - "./OpenSearch/DATA2:/usr/share/opensearch/data"
      - "./certs/root-ca.pem:/usr/share/opensearch/config/root-ca.pem"
      - "./certs/node.pem:/usr/share/opensearch/config/node.pem"
      - "./certs/node-key.pem:/usr/share/opensearch/config/node-key.pem"
      - "./certs/admin.pem:/usr/share/opensearch/config/admin.pem"
      - "./certs/admin-key.pem:/usr/share/opensearch/config/admin-key.pem"

  os03.orgx.com:
    restart: always
    image: opensearchproject/opensearch:1.3.1
    environment:
      OPENSEARCH_JAVA_OPTS: "-Xms16g -Xmx16g" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      node.name: os03.orgx.com
      discovery.seed_hosts: os01.orgx.com,os02.orgx.com,os03.orgx.com
      cluster.initial_master_nodes: os01.orgx.com,os02.orgx.com,os03.orgx.com
      plugins.security.ssl.transport.pemkey_filepath: node-key.pem # relative path
      plugins.security.ssl.transport.pemcert_filepath: node.pem
      plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
      plugins.security.ssl.http.pemkey_filepath: node-key.pem
      plugins.security.ssl.http.pemcert_filepath: node.pem
      plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
      DISABLE_INSTALL_DEMO_CONFIG: "true"
      JAVA_HOME: /usr/share/opensearch/jdk
      bootstrap.memory_lock: "true" # along with the memlock settings below, disables swapping
      network.host: "0.0.0.0"
    ulimits: 
      memlock:
        soft: -1
        hard: -1
    volumes:
      - "./OpenSearch/opensearch.yml:/usr/share/opensearch/config/opensearch.yml"
      - "./OpenSearch/internal_users.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml"
      - "./OpenSearch/roles_mapping.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml"
      - "./OpenSearch/tenants.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml"
      - "./OpenSearch/roles.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml"
      - "./OpenSearch/action_groups.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml"
      - "./OpenSearch/DATA3:/usr/share/opensearch/data"
      - "./certs/root-ca.pem:/usr/share/opensearch/config/root-ca.pem"
      - "./certs/node.pem:/usr/share/opensearch/config/node.pem"
      - "./certs/node-key.pem:/usr/share/opensearch/config/node-key.pem"
      - "./certs/admin.pem:/usr/share/opensearch/config/admin.pem"
      - "./certs/admin-key.pem:/usr/share/opensearch/config/admin-key.pem"
  
  kibana.orgx.com:
    restart: always
    image: opensearchproject/opensearch-dashboards:1.3.0
    ports:
      - 5601:5601
    volumes:
      - "./certs/root-ca.pem:/usr/share/opensearch-dashboards/config/root-ca.pem"
      - "./certs/node.pem:/usr/share/opensearch-dashboards/config/node.pem"
      - "./certs/node-key.pem:/usr/share/opensearch-dashboards/config/node-key.pem"
      - "./certs/admin.pem:/usr/share/opensearch-dashboards/config/admin.pem"
      - "./certs/admin-key.pem:/usr/share/opensearch-dashboards/config/admin-key.pem"
      - "./OpenSearch-Dashboard/opensearch-dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml"
    environment:
      OPENSEARCH_HOSTS: '["https://os01.orgx.com:9200","https://os02.orgx.com:9200","https://os03.orgx.com:9200"]' # must be a string with no spaces when specified as an environment variable
      DISABLE_INSTALL_DEMO_CONFIG: "true"
2

Update :

Changed opensearch.yml by :

cluster.name: os-cluster.orgx.com
network.host: 0.0.0.0

bootstrap.memory_lock: "true" # along with the memlock settings below, disables swapping

cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 93%
cluster.routing.allocation.disk.watermark.high: 95%

plugins.security.allow_unsafe_democertificates: true
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemtrustedcas_filepath: node.pem
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false

plugins.security.authcz.admin_dn:
  - "CN=*.orgx.com,OU=X,O=XZ,ST=XT,C=XY"
plugins.security.nodes_dn:
  - "CN=*.orgx.com,OU=X,O=XZ,ST=XT,C=XY"

No more error on os01,02,03

But now i’ve got this error :

root@X:/mnt/XXX# docker-compose exec os01.orgx.com  bash -c "chmod +x plugins/opensearch-security/tools/securityadmin.sh && bash plugins/opensearch-security/tools/securityadmin.sh -cd plugins/opensearch-security/securityconfig -icl -nhnv -cacert config/root-ca.pem -cert config/admin.pem -key config/admin-key.pem -h localhost"

Security Admin v7
Will connect to localhost:9300 ... done
Connected as CN=*.orgx.com,OU=X,O=XZ,ST=XT,C=XY
ERR: Seems you use a node certificate which is also an admin certificate
     That may have worked with older OpenSearch Security versions but it indicates
     a configuration error and is therefore forbidden now.
Unable to check whether cluster is sane
ERR: An unexpected OpenSearchSecurityException occured: No user found for cluster:monitor/nodes/info
Trace:
OpenSearchSecurityException[No user found for cluster:monitor/nodes/info]
        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:287)
        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:154)
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192)
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:169)
        at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:95)
        at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:91)
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:193)
        at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:336)
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:153)
        at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:647)
        at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:118)
        at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43)
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:91)
        at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:244)
        at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:127)
        at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:109)
        at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:759)
        at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:170)
        at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:145)
        at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:110)
        at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.lang.Thread.run(Thread.java:829)
3

@Nerooz Have you tried typing the exact DN in admin_dn instead of a wildcard?

4

Hello @pablo

What do you mean by exact DN in admin_dn ?

If you refer to the subject of the wildcard :
I’ve got two possible DN :
“CN=*.orgx.com,OU=X,O=XZ,ST=XT,C=XY” “CN=orgx.com,OU=X,O=XZ,ST=XT,C=XY”

But in both case i’ve got the same message, i’ve also tried with the issuer but same error or telling me to use “CN=*.orgx.com,OU=X,O=XZ,ST=XT,C=XY” as admin_dn

5

@Nerooz What is DN in admin.pem?

6

@pablo The same as node.pem

“CN=*.orgx.com,OU=X,O=XZ,ST=XT,C=XY”

I took the bundle certificate of my wildcard

  • Intermediate + Root to get the root-ca
  • .CER to get the admin.pem and node.pem
7

After few days of research, i still do not understand.

Can we use ONE certificate for a full cluster (WILDCARD) ? with 3 node Opensearch and 1 Opensearch-Dashboard ?

The plugins.security.authcz.admin_dn do not like wildcard, does it need a special AltDNS in the wildcard ?
Why plugins.security.nodes_dn accept “*.orgx.com” but not the exact CN of the machine “os01.orgx.com” ?
Do i need to recreate a special account when i got OpenSearchSecurityException[No user found for cluster:monitor/nodes/info] ?

8

@Nerooz As per OpenSearch documentation wildcards and regular expressions can’t be used as admin_dn.
admin_dn has to be a specific DN like the one in the example.

Also, using the same certificate as node_dn and admin_dn is not supported.

9

Thanks for your reply.

So what it basically means is :

  • I won’t be able to use a WILDCARD
  • If i want HTTPS i will need specific certificates with a proper CN for each
    • I’ve tried with WILDCARD for CA/NODE and SELF-SIGNED for ADMIN , it still doesn’t work
[root@XX NAS]$  docker-compose exec os01.orgx.com bash -c "chmod +x plugins/opensearch-security/tools/securityadmin.sh && bash plugins/opensearch-security/tools/securityadmin.sh -cd plugins/opensearch-security/securityconfig -icl -nhnv -cacert config/ca.pem -cert config/admin.pem -key config/admin-key.pem -h dns-x.orgx.com.com"
Security Admin v7
Will connect to dns-x.orgx.com:9300 ... done
ERR: Cannot connect to OpenSearch. Please refer to opensearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{YLPGNzNHQeq3xz7etx4a_w}{dns-x.orgx.com}{172.X.X.80:9300}]]
        at org.opensearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:381)
        at org.opensearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:272)
        at org.opensearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:79)
        at org.opensearch.client.transport.TransportClient.doExecute(TransportClient.java:484)
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433)
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:419)
        at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:524)
        at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:157)
  • The last solution will be to use self-signed on the whole cluster and eventually use https proxy for client …

Let me know if i’m wrong here too ? thanks

10

@Nerooz You won’t be able to use the wildcard certificate as the admin client certificate. (admin_dn)

What do you mean you used a wildcard with CA/NODE? CA and node shouldn’t be the same.

11

@pablo

From the bundle certificate of my wildcard :
I got :

  • Root-Certificate
  • Intermediate-Certificate
  • CSR & Key
  • PKCS7-Certificate

My CA is composed of my Root-Certificate and Intermediate-Certificate
My NODE is a export of my PKCS7 as CER converted to PEM

12

@Nerooz Try removing -h and hostname from the command. The script will use localhost instead then.

13

I still got the same message @pablo

14

@Nerooz You get exactly the same error?

NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{YLPGNzNHQeq3xz7etx4a_w}{dns-x.orgx.com}{172.X.X.80:9300}]]
15

Yes, i’ve tried with IP/DNS too but same errors.

I went to a full self signed cluster and used a proxy https to get https over my os-dashboard.
Seems to be working. Hope it works with all cluster …