Security transport config for 2.0.0

nik · June 21, 2022, 8:07pm

I’m in the same boat as @bseed over here: Securityadmin.sh uses http port 9200 and not transport port 9300 in OpenSearch 2.0.0 - #4 by pablo

I’m trying to launch a new OS 2.0.0 cluster in k8s (RKE). I’ll just repost what he said:

When I disable install demo config and point transport and http to my certs, I get an error “Transport client authentication no longer supported”
When I add enabled: false to security.ssl.transport, I get an error “plugins.security.ssl.transport.enabled must be set to ‘true’”
When i remove every security.ssl.transport lines, I get an error
“plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.”

Here’s the relevant part of my config as it stands now:

plugins:
      security:
        ssl:
          transport:
            pemcert_filepath: certs/opensearch.crt.pem
            pemkey_filepath: certs/opensearch.key.pem
            pemtrustedcas_filepath: certs/ca-bundle.pem
            enforce_hostname_verification: false
          http:
            enabled: true
            pemcert_filepath: certs/opensearch.crt.pem
            pemkey_filepath: certs/opensearch.key.pem
            pemtrustedcas_filepath: certs/ca-bundle.pem
        allow_default_init_securityindex: true
        authcz:
          admin_dn:
            - CN=[redacted]
        audit.type: internal_opensearch
        enable_snapshot_restore_privilege: true
        check_snapshot_restore_write_privileges: true
        restapi:
          roles_enabled: ["all_access", "security_rest_api_access"]
    opendistro_security:
      audit:
        config:
          disabled_rest_categories: NONE

What do I need to do to get it to work?

Thanks!

bseed · June 22, 2022, 5:10am

Thanks for reposting, I’m still facing the same issue as of today, right now I’m using install demo config and I don’t like that

m-hofmann · June 22, 2022, 2:10pm

We too are facing this problem. We tried both PKCS12 keystores as well as PEM files for transport encryption, but our k8s installation too fails with “transport client authentication no longer supported”.

From a look into the source code it looks like the request cannot be identified as originating from another cluster node (looking at the if-statement surrounding the line where the exception is thrown). Apparently nodes_dn configuration is used to identify nodes - however even using a wildcard in nodes_dn does not work. Has anyone found a working configuration for nodes_dn?

nik · June 22, 2022, 2:34pm

Yeah I found out the hard way that it doesn’t support PKCS#1 certs, so I had to get cert-manager to issue PKCS#8 certs in PEM files. Those seem to work fine, and then I hit this hurdle.

More detailed stack when the transport lines are left in:

[2022-06-22T14:27:55,544][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-0] handshake failed for [connectToRemoteMasterNode[redacted:9300]]
org.opensearch.transport.RemoteTransportException: [opensearch-cluster-data-ingest-1][redacted:9300][internal:transport/handshake]
Caused by: org.opensearch.OpenSearchException: Transport client authentication no longer supported.
at org.opensearch.security.ssl.util.ExceptionUtils.createTransportClientNoLongerSupportedException(ExceptionUtils.java:63) ~[?:?]
at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:270) ~[?:?]
at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:153) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:651) ~[?:?]
at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:118) ~[?:?]
at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:103) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:249) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:132) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:769) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]

These lines repeat over and over. They don’t crash the pod entirely, but if I try to hit OS over :9200 I get this: OpenSearch Security not initialized.

pablo · June 22, 2022, 2:49pm

@nik You need to add node certificate DNs in plugins.security.nodes_dn: option in opensearch.yml
Use exact names instead of wildecards.

i.e.

nik · June 22, 2022, 3:07pm

@pablo

Thanks for the reply. All of my nodes get the same certificate (CN=opensearch.domain). Adding that to nodes_dn didn’t change the behavior. It still tries to bind to 9300, which should be deprecated, and then throws the same error stack I pasted above.

pablo · June 22, 2022, 4:30pm

@nik Deprecation regards only authentication and authorization. The 9300 port will be still used.

If the node certificate is not mentioned in the nodes_dn then it is treated as an external client which is no longer supported.

nik · June 22, 2022, 7:06pm

@pablo Ok I misunderstood that – I thought they made all communications happen over 9200. Either way I’m still getting the same errors, and traffic over 9300 is allowed within my service mesh.

m-hofmann · June 23, 2022, 7:42am

Turns out our keystore/truststore setup works if we add the full certificate chain to nodes_dn. As we use a self-signed root CA and one intermediate certificate the config looks something like this:

nodes_dn:
- "OU=...,O=company,C=...,CN=static.node.name.something.com"
- "CN=intermediate,O=company,C=..."
- "CN=root,O=companyC=..."

powerful_clouds · December 12, 2022, 2:18pm

Can this be made to work without using the keystore setup, i.e. by using .pem files?

pablo · December 12, 2022, 3:09pm

@powerful_clouds Yes, as per documentation.

powerful_clouds · December 12, 2022, 3:41pm

@pablo I thought so, but I haven’t managed to make it happen (yet).
First I generated the certs as the page you linked mentions.
Then I created a generic k8s secret called opensearch-certs, where I passed in each of those 5 files: admin.pem, admin-key.pem, node.pem, node-key.pem, root-ca.pem (although I’m not sure if I need the admin* stuff?).
I then added that secret to the secretMounts section of the opensearch.yml file:

118 secretMounts:
119   - name: opensearch-certs
120     secretName: opensearch-certs
121     path: /usr/share/opensearch/config/ssl

Finally, here’s the SSL section of opensearch.yml:

 60         ssl:
 61           transport:
 62             pemcert_filepath: ssl/node1.pem
 63             pemkey_filepath: ssl/node1-key.pem
 64             pemtrustedcas_filepath: ssl/root-ca.pem
 65             enforce_hostname_verification: false
 66           http:
 67             enabled: true
 68             pemcert_filepath: ssl/node1.pem
 69             pemkey_filepath: ssl/node1-key.pem
 70             pemtrustedcas_filepath: ssl/root-ca.pem
 71         allow_unsafe_democertificates: true
 72         allow_default_init_securityindex: true
 73         authcz:
 74           admin_dn:
 75             - "CN=AB,OU=CDE,O=Something,L=Somewhere,C=GF"
 76         nodes_dn:
 77           - "CN=AB,OU=CDE,O=Something,L=Somewhere,C=GF"

When I start up the OpenSearch pods, I get the following:

[2022-12-12T15:35:28,556][ERROR][o.o.s.t.SecurityRequestHandler] 
[opensearch-cluster-master-0] OpenSearchException[Transport client 
authentication no longer supported.]

And I also get

[2022-12-12T15:35:28,551][WARN ][o.o.d.HandshakingTransportAddressConnector] 
[opensearch-cluster-master-0] handshake failed for 
[connectToRemoteMasterNode[<IP>:9300]]
org.opensearch.transport.RemoteTransportException: 
[opensearch-cluster-master-2][<IP>:9300][internal:transport/handshake]
Caused by: org.opensearch.OpenSearchException: Transport client 
authentication no longer supported.

If I remove the transport section, I get the following error on startup:

java.lang.IllegalStateException: failed to load plugin class 
[org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[
plugins.security.ssl.transport.keystore_filepath or 
plugins.security.ssl.transport.server.pemcert_filepath 
and plugins.security.ssl.transport.client.pemcert_filepath must be 
set if transport ssl is requested.]

I don’t know if this is important, but the pods are part of a k8s stateful set that’s running on a local kind cluster.

pablo · December 12, 2022, 4:01pm

@powerful_clouds Would you mind creating a new thread and sharing the link here?

powerful_clouds · December 12, 2022, 9:11pm

@pablo Sure. Here’s the link to the thread.

Topic		Replies	Views
TLS configuration with organizational certs Security troubleshoot , configure	2	487	January 21, 2022
Securityadmin.sh uses http port 9200 and not transport port 9300 in OpenSearch 2.0.0 Security troubleshoot , upgrade	3	2810	June 17, 2022
Opensearch security is asking for plugins.security.ssl.transport.truststore_filepath property to be set Security configure	10	4814	February 8, 2022
Tarball install security error OpenDistro	6	869	November 26, 2021
"Transport Client Authentication no longer supported" error when deploying cluster with security plugin enabled Security troubleshoot	17	7008	August 24, 2022

Security transport config for 2.0.0

Related Topics