Security transport config for 2.0.0

I’m in the same boat as @bseed over here: Securityadmin.sh uses http port 9200 and not transport port 9300 in OpenSearch 2.0.0 - #4 by pablo

I’m trying to launch a new OS 2.0.0 cluster in k8s (RKE). I’ll just repost what he said:

  • When I disable install demo config and point transport and http to my certs, I get an error “Transport client authentication no longer supported”
  • When I add enabled: false to security.ssl.transport, I get an error “plugins.security.ssl.transport.enabled must be set to ‘true’”
  • When i remove every security.ssl.transport lines, I get an error
    “plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.”

Here’s the relevant part of my config as it stands now:

plugins:
      security:
        ssl:
          transport:
            pemcert_filepath: certs/opensearch.crt.pem
            pemkey_filepath: certs/opensearch.key.pem
            pemtrustedcas_filepath: certs/ca-bundle.pem
            enforce_hostname_verification: false
          http:
            enabled: true
            pemcert_filepath: certs/opensearch.crt.pem
            pemkey_filepath: certs/opensearch.key.pem
            pemtrustedcas_filepath: certs/ca-bundle.pem
        allow_default_init_securityindex: true
        authcz:
          admin_dn:
            - CN=[redacted]
        audit.type: internal_opensearch
        enable_snapshot_restore_privilege: true
        check_snapshot_restore_write_privileges: true
        restapi:
          roles_enabled: ["all_access", "security_rest_api_access"]
    opendistro_security:
      audit:
        config:
          disabled_rest_categories: NONE

What do I need to do to get it to work?

Thanks!

1 Like

Thanks for reposting, I’m still facing the same issue as of today, right now I’m using install demo config and I don’t like that

We too are facing this problem. We tried both PKCS12 keystores as well as PEM files for transport encryption, but our k8s installation too fails with “transport client authentication no longer supported”.

From a look into the source code it looks like the request cannot be identified as originating from another cluster node (looking at the if-statement surrounding the line where the exception is thrown). Apparently nodes_dn configuration is used to identify nodes - however even using a wildcard in nodes_dn does not work. Has anyone found a working configuration for nodes_dn?

Yeah I found out the hard way that it doesn’t support PKCS#1 certs, so I had to get cert-manager to issue PKCS#8 certs in PEM files. Those seem to work fine, and then I hit this hurdle.

More detailed stack when the transport lines are left in:

[2022-06-22T14:27:55,544][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-0] handshake failed for [connectToRemoteMasterNode[redacted:9300]]
org.opensearch.transport.RemoteTransportException: [opensearch-cluster-data-ingest-1][redacted:9300][internal:transport/handshake]
Caused by: org.opensearch.OpenSearchException: Transport client authentication no longer supported.
at org.opensearch.security.ssl.util.ExceptionUtils.createTransportClientNoLongerSupportedException(ExceptionUtils.java:63) ~[?:?]
at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:270) ~[?:?]
at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:153) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:651) ~[?:?]
at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:118) ~[?:?]
at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:103) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:249) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:132) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:769) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]

These lines repeat over and over. They don’t crash the pod entirely, but if I try to hit OS over :9200 I get this: OpenSearch Security not initialized.

@nik You need to add node certificate DNs in plugins.security.nodes_dn: option in opensearch.yml
Use exact names instead of wildecards.

i.e.
image

@pablo

Thanks for the reply. All of my nodes get the same certificate (CN=opensearch.domain). Adding that to nodes_dn didn’t change the behavior. It still tries to bind to 9300, which should be deprecated, and then throws the same error stack I pasted above.

@nik Deprecation regards only authentication and authorization. The 9300 port will be still used.

If the node certificate is not mentioned in the nodes_dn then it is treated as an external client which is no longer supported.

@pablo Ok I misunderstood that – I thought they made all communications happen over 9200. Either way I’m still getting the same errors, and traffic over 9300 is allowed within my service mesh.

Turns out our keystore/truststore setup works if we add the full certificate chain to nodes_dn. As we use a self-signed root CA and one intermediate certificate the config looks something like this:

nodes_dn:
- "OU=...,O=company,C=...,CN=static.node.name.something.com"
- "CN=intermediate,O=company,C=..."
- "CN=root,O=companyC=..."