Security transport config for 2.0.0

I’m in the same boat as @bseed over here: uses http port 9200 and not transport port 9300 in OpenSearch 2.0.0 - #4 by pablo

I’m trying to launch a new OS 2.0.0 cluster in k8s (RKE). I’ll just repost what he said:

  • When I disable install demo config and point transport and http to my certs, I get an error “Transport client authentication no longer supported”
  • When I add enabled: false to security.ssl.transport, I get an error “ must be set to ‘true’”
  • When i remove every security.ssl.transport lines, I get an error
    “ or and must be set if transport ssl is requested.”

Here’s the relevant part of my config as it stands now:

            pemcert_filepath: certs/opensearch.crt.pem
            pemkey_filepath: certs/opensearch.key.pem
            pemtrustedcas_filepath: certs/ca-bundle.pem
            enforce_hostname_verification: false
            enabled: true
            pemcert_filepath: certs/opensearch.crt.pem
            pemkey_filepath: certs/opensearch.key.pem
            pemtrustedcas_filepath: certs/ca-bundle.pem
        allow_default_init_securityindex: true
            - CN=[redacted]
        audit.type: internal_opensearch
        enable_snapshot_restore_privilege: true
        check_snapshot_restore_write_privileges: true
          roles_enabled: ["all_access", "security_rest_api_access"]
          disabled_rest_categories: NONE

What do I need to do to get it to work?



Thanks for reposting, I’m still facing the same issue as of today, right now I’m using install demo config and I don’t like that

We too are facing this problem. We tried both PKCS12 keystores as well as PEM files for transport encryption, but our k8s installation too fails with “transport client authentication no longer supported”.

From a look into the source code it looks like the request cannot be identified as originating from another cluster node (looking at the if-statement surrounding the line where the exception is thrown). Apparently nodes_dn configuration is used to identify nodes - however even using a wildcard in nodes_dn does not work. Has anyone found a working configuration for nodes_dn?

Yeah I found out the hard way that it doesn’t support PKCS#1 certs, so I had to get cert-manager to issue PKCS#8 certs in PEM files. Those seem to work fine, and then I hit this hurdle.

More detailed stack when the transport lines are left in:

[2022-06-22T14:27:55,544][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-0] handshake failed for [connectToRemoteMasterNode[redacted:9300]]
org.opensearch.transport.RemoteTransportException: [opensearch-cluster-data-ingest-1][redacted:9300][internal:transport/handshake]
Caused by: org.opensearch.OpenSearchException: Transport client authentication no longer supported.
at ~[?:?]
at ~[?:?]
at ~[?:?]
at$7$1.messageReceived( ~[?:?]
at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:118) ~[?:?]
at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived( ~[?:?]
at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived( ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundHandler.handleRequest( ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundHandler.messageReceived( ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundHandler.inboundMessage( ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.TcpTransport.inboundMessage( ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundPipeline.forwardFragments( ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundPipeline.doHandleBytes( ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundPipeline.handleBytes( ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead( ~[?:?]
at ~[?:?]
at ~[?:?]
at ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead( ~[?:?]
at ~[?:?]
at ~[?:?]
at ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead( ~[?:?]
at ~[?:?]
at ~[?:?]
at ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap( ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible( ~[?:?]
at io.netty.handler.ssl.SslHandler.decode( ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection( ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode( ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead( ~[?:?]
at ~[?:?]
at ~[?:?]
at ~[?:?]
at$HeadContext.channelRead( ~[?:?]
at ~[?:?]
at ~[?:?]
at ~[?:?]
at$ ~[?:?]
at ~[?:?]
at ~[?:?]
at ~[?:?]
at ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$ ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$ ~[?:?]
at [?:?]

These lines repeat over and over. They don’t crash the pod entirely, but if I try to hit OS over :9200 I get this: OpenSearch Security not initialized.

@nik You need to add node certificate DNs in option in opensearch.yml
Use exact names instead of wildecards.



Thanks for the reply. All of my nodes get the same certificate (CN=opensearch.domain). Adding that to nodes_dn didn’t change the behavior. It still tries to bind to 9300, which should be deprecated, and then throws the same error stack I pasted above.

@nik Deprecation regards only authentication and authorization. The 9300 port will be still used.

If the node certificate is not mentioned in the nodes_dn then it is treated as an external client which is no longer supported.

@pablo Ok I misunderstood that – I thought they made all communications happen over 9200. Either way I’m still getting the same errors, and traffic over 9300 is allowed within my service mesh.

Turns out our keystore/truststore setup works if we add the full certificate chain to nodes_dn. As we use a self-signed root CA and one intermediate certificate the config looks something like this:

- "OU=...,O=company,C=...,"
- "CN=intermediate,O=company,C=..."
- "CN=root,O=companyC=..."

Can this be made to work without using the keystore setup, i.e. by using .pem files?

@powerful_clouds Yes, as per documentation.

@pablo I thought so, but I haven’t managed to make it happen (yet). :smiley:
First I generated the certs as the page you linked mentions.
Then I created a generic k8s secret called opensearch-certs, where I passed in each of those 5 files: admin.pem, admin-key.pem, node.pem, node-key.pem, root-ca.pem (although I’m not sure if I need the admin* stuff?).
I then added that secret to the secretMounts section of the opensearch.yml file:

118 secretMounts:
119   - name: opensearch-certs
120     secretName: opensearch-certs
121     path: /usr/share/opensearch/config/ssl

Finally, here’s the SSL section of opensearch.yml:

 60         ssl:
 61           transport:
 62             pemcert_filepath: ssl/node1.pem
 63             pemkey_filepath: ssl/node1-key.pem
 64             pemtrustedcas_filepath: ssl/root-ca.pem
 65             enforce_hostname_verification: false
 66           http:
 67             enabled: true
 68             pemcert_filepath: ssl/node1.pem
 69             pemkey_filepath: ssl/node1-key.pem
 70             pemtrustedcas_filepath: ssl/root-ca.pem
 71         allow_unsafe_democertificates: true
 72         allow_default_init_securityindex: true
 73         authcz:
 74           admin_dn:
 75             - "CN=AB,OU=CDE,O=Something,L=Somewhere,C=GF"
 76         nodes_dn:
 77           - "CN=AB,OU=CDE,O=Something,L=Somewhere,C=GF"

When I start up the OpenSearch pods, I get the following:

[opensearch-cluster-master-0] OpenSearchException[Transport client 
authentication no longer supported.]

And I also get

[2022-12-12T15:35:28,551][WARN ][o.o.d.HandshakingTransportAddressConnector] 
[opensearch-cluster-master-0] handshake failed for 
Caused by: org.opensearch.OpenSearchException: Transport client 
authentication no longer supported.

If I remove the transport section, I get the following error on startup:

java.lang.IllegalStateException: failed to load plugin class 
Likely root cause: OpenSearchException[ or 
and must be 
set if transport ssl is requested.]

I don’t know if this is important, but the pods are part of a k8s stateful set that’s running on a local kind cluster.

@powerful_clouds Would you mind creating a new thread and sharing the link here?

1 Like

@pablo Sure. Here’s the link to the thread. :slight_smile: