"Transport client authentication no longer supported" in between-node transport layer communication

Security
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch image - opensearchproject/opensearch:2.11.0

Describe the issue:
Im trying to configure TLS on transport layer between nodes (1 manager node, 1 data node). TLS on REST (9200) layer works (openssl s_client -CAfile root-ca-1.pem -connect 10.0.0.70:9200), but when i start nodes i receive “Transport client authentication no longer supported” from manager node and “handshake failed for [connectToRemoteMasterNode[10.0.0.70:9300]], Transport client authentication no longer supported” from data node.

Configuration:
docker-compose of manager node (data node has the same config except certs, ip, role)

version: "3.8"
services:
  opensearch_manager1:
    network_mode: "host"
    build: .
    # image: "opensearchproject/opensearch:2.11.0"
    ulimits:
      memlock:
        hard: -1
        soft: -1
      nofile:
        soft: 262144
        hard: 262144
    environment:
      - plugins.security.nodes_dn=["CN=node1.dns.a-record,OU=OU,O=ORG,L=TORONTO,ST=ONTARIO,C=CA","CN=node2.dns.a-record,OU=OU,O=ORG,L=TORONTO,ST=ONTARIO,C=CA"]
      - plugins.security.ssl.transport.pemcert_filepath=/usr/share/opensearch/config/node1.pem
      - plugins.security.ssl.transport.pemkey_filepath=/usr/share/opensearch/config/node1-key.pem
      - plugins.security.ssl.transport.pemtrustedcas_filepath=/usr/share/opensearch/config/root-ca-1.pem
      - plugins.security.ssl.transport.enforce_hostname_verification=false
      - plugins.security.ssl.http.enabled=true
      - plugins.security.ssl.http.pemcert_filepath=/usr/share/opensearch/config/node1.pem
      - plugins.security.ssl.http.pemkey_filepath=/usr/share/opensearch/config/node1-key.pem
      - plugins.security.ssl.http.pemtrustedcas_filepath=/usr/share/opensearch/config/root-ca-1.pem
      - bootstrap.memory_lock=true
      - action.auto_create_index=true
      - plugins.security.ssl.transport.enabled=true
      - plugins.security.disabled=false
      - cluster.name=opensearch-cluster
      - node.name=opensearch-cluster-manager
      - node.roles=master
      - network.host=10.0.0.70
      - http.host=10.0.0.70
      - transport.host=10.0.0.70
      - http.port=9200
      - transport.port=9300
      - discovery.seed_hosts=10.0.0.71
      - cluster.initial_cluster_manager_nodes=opensearch-cluster-manager
      - cluster.initial_master_nodes=opensearch-cluster-manager
    volumes:
      - "os_data_manager1:/usr/share/opensearch/data"
      - ./root-ca-1.pem:/usr/share/opensearch/config/root-ca-1.pem
      - ./node1.pem:/usr/share/opensearch/config/node1.pem
      - ./node1-key.pem:/usr/share/opensearch/config/node1-key.pem
    ports:
      - 9200:9200
      - 9300:9300
    restart: "on-failure"
volumes:
  os_data_manager1:

Relevant Logs or Screenshots:

“openssl x509 -subject -nameopt RFC2253 -noout -in node1.pem” result - subject=CN=node1.dns.a-record,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA

“openssl x509 -subject -nameopt RFC2253 -noout -in node2.pem” result - subject=CN=node2.dns.a-record,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA
Also “openssl s_client -CAfile root-ca-1.pem -connect 10.0.0.70:9300” return:
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits

SSL handshake has read 1667 bytes and written 405 bytes
Verification: OK

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
80EB0495AB7F0000:error:0A000412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:ssl/record/rec_layer_s3.c:1605:SSL alert number 42

Manager logs:
application-opensearch_manager1-1 | [2023-12-08T14:56:14,806][INFO ][o.o.h.AbstractHttpServerTransport] [opensearch-cluster-manager] publish_address {10.0.0.70:9200}, bound_addresses {10.0.0.70:9200}
application-opensearch_manager1-1 | [2023-12-08T14:56:14,810][INFO ][o.o.n.Node ] [opensearch-cluster-manager] started
application-opensearch_manager1-1 | [2023-12-08T14:56:14,814][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-manager] Node started
application-opensearch_manager1-1 | [2023-12-08T14:56:14,815][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-manager] Will attempt to create index .opendistro_security and default configs if they are absent
application-opensearch_manager1-1 | [2023-12-08T14:56:14,819][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-manager] Background init thread started. Install default config?: true
application-opensearch_manager1-1 | [2023-12-08T14:56:14,820][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-manager] Wait for cluster to be available …
application-opensearch_manager1-1 | [2023-12-08T14:56:14,819][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-manager] 0 OpenSearch Security modules loaded so far:
application-opensearch_manager1-1 | [2023-12-08T14:56:14,835][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-manager] handshake failed for [connectToRemoteMasterNode[10.0.0.71:9300]]
application-opensearch_manager1-1 | org.opensearch.transport.RemoteTransportException: [opensearch-data-node-1][10.0.0.71:9300][internal:transport/handshake]
application-opensearch_manager1-1 | Caused by: org.opensearch.OpenSearchException: Transport client authentication no longer supported.
application-opensearch_manager1-1 | at org.opensearch.security.ssl.util.ExceptionUtils.createTransportClientNoLongerSupportedException(ExceptionUtils.java:68) ~[?:?]
application-opensearch_manager1-1 | at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:294) ~[?:?]
application-opensearch_manager1-1 | at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:169) ~[?:?]
application-opensearch_manager1-1 | at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:774) ~[?:?]
application-opensearch_manager1-1 | at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:113) ~[?:?]
application-opensearch_manager1-1 | at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
application-opensearch_manager1-1 | at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-2.11.0.jar:2.11.0]
application-opensearch_manager1-1 | at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:262) ~[opensearch-2.11.0.jar:2.11.0]
application-opensearch_manager1-1 | at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) ~[opensearch-2.11.0.jar:2.11.0]
application-opensearch_manager1-1 | at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) ~[opensearch-2.11.0.jar:2.11.0]
application-opensearch_manager1-1 | at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) ~[opensearch-2.11.0.jar:2.11.0]
application-opensearch_manager1-1 | at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-2.11.0.jar:2.11.0]
application-opensearch_manager1-1 | at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-2.11.0.jar:2.11.0]
application-opensearch_manager1-1 | at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-2.11.0.jar:2.11.0]
application-opensearch_manager1-1 | at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
application-opensearch_manager1-1 | at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
application-opensearch_manager1-1 | at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
application-opensearch_manager1-1 | at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1471) ~[?:?]
application-opensearch_manager1-1 | at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1334) ~[?:?]
application-opensearch_manager1-1 | at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1383) ~[?:?]
application-opensearch_manager1-1 | at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
application-opensearch_manager1-1 | at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
application-opensearch_manager1-1 | at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
application-opensearch_manager1-1 | at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
application-opensearch_manager1-1 | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
application-opensearch_manager1-1 | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
application-opensearch_manager1-1 | at java.lang.Thread.run(Thread.java:833) [?:?]
application-opensearch_manager1-1 | [2023-12-08T14:56:14,876][INFO ][o.o.c.r.a.AllocationService] [opensearch-cluster-manager] Cluster health status changed from [YELLOW] to [RED] (reason: [state recovered]).
application-opensearch_manager1-1 | [2023-12-08T14:56:14,937][INFO ][o.o.c.s.ClusterSettings ] [opensearch-cluster-manager] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
application-opensearch_manager1-1 | [2023-12-08T14:56:14,950][INFO ][o.o.a.c.HashRing ] [opensearch-cluster-manager] Node added: [BaRACysjRbOvlDqhTjlCOw]
application-opensearch_manager1-1 | [2023-12-08T14:56:14,953][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-cluster-manager] Detected cluster change event for destination migration
application-opensearch_manager1-1 | [2023-12-08T14:56:14,955][INFO ][o.o.a.c.HashRing ] [opensearch-cluster-manager] All nodes with known AD version: {BaRACysjRbOvlDqhTjlCOw=ADNodeInfo{version=2.11.0, isEligibleDataNode=false}}
application-opensearch_manager1-1 | [2023-12-08T14:56:14,956][INFO ][o.o.a.c.HashRing ] [opensearch-cluster-manager] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
application-opensearch_manager1-1 | [2023-12-08T14:56:14,957][INFO ][o.o.a.c.HashRing ] [opensearch-cluster-manager] Build AD version hash ring successfully
application-opensearch_manager1-1 | [2023-12-08T14:56:14,958][INFO ][o.o.a.c.ADClusterEventListener] [opensearch-cluster-manager] Init AD version hash ring successfully
application-opensearch_manager1-1 | [2023-12-08T14:56:15,021][INFO ][o.o.g.GatewayService ] [opensearch-cluster-manager] recovered [4] indices into cluster_state
application-opensearch_manager1-1 | [2023-12-08T14:56:15,032][WARN ][o.o.o.i.ObservabilityIndex] [opensearch-cluster-manager] message: index [.opensearch-observability/nKfRpiB6SdOaIg7-OMlvrQ] already exists
application-opensearch_manager1-1 | [2023-12-08T14:56:15,034][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [opensearch-cluster-manager] observability:createMappingTemplate ss4o_metrics_template API called
application-opensearch_manager1-1 | [2023-12-08T14:56:15,036][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-manager] Loading builtin types!
application-opensearch_manager1-1 | [2023-12-08T14:56:15,050][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-manager] Indexing [429] fieldMappingDocs from logTypes: 24
application-opensearch_manager1-1 | [2023-12-08T14:56:15,092][ERROR][o.o.s.t.SecurityRequestHandler] [opensearch-cluster-manager] OpenSearchException[Transport client authentication no longer supported.]
application-opensearch_manager1-1 | [2023-12-08T14:56:15,180][WARN ][o.o.s.SecurityAnalyticsPlugin] [opensearch-cluster-manager] Failed to initialize LogType config index and builtin log types
application-opensearch_manager1-1 | [2023-12-08T14:56:15,233][INFO ][o.o.p.PluginsService ] [opensearch-cluster-manager] PluginService:onIndexModule index:[validate-template-42s-owvktga3ehq8iqexta/EVAJfvmbRD2olvUj2jwcig]
application-opensearch_manager1-1 | [2023-12-08T14:56:15,778][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-manager] Loading builtin types!
application-opensearch_manager1-1 | [2023-12-08T14:56:15,781][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-manager] Indexing [429] fieldMappingDocs from logTypes: 24
application-opensearch_manager1-1 | [2023-12-08T14:56:15,792][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-manager] Loading builtin types!
application-opensearch_manager1-1 | [2023-12-08T14:56:15,796][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-manager] Indexing [429] fieldMappingDocs from logTypes: 24
application-opensearch_manager1-1 | [2023-12-08T14:56:15,803][INFO ][o.o.c.m.MetadataIndexTemplateService] [opensearch-cluster-manager] updating index template [ss4o_metrics_template] for index patterns [ss4o_metrics--]
application-opensearch_manager1-1 | [2023-12-08T14:56:15,868][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-cluster-manager] Detected cluster change event for destination migration
application-opensearch_manager1-1 | [2023-12-08T14:56:15,871][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [opensearch-cluster-manager] observability:Mapping Template ss4o_metrics_template creation Acknowledged
application-opensearch_manager1-1 | [2023-12-08T14:56:15,871][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [opensearch-cluster-manager] observability:createMappingTemplate ss4o_traces_template API called
application-opensearch_manager1-1 | [2023-12-08T14:56:15,877][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-manager] Index .opendistro_security already exists
application-opensearch_manager1-1 | [2023-12-08T14:56:15,877][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-manager] Node started, try to initialize it. Wait for at least yellow cluster state…
application-opensearch_manager1-1 | [2023-12-08T14:56:15,888][INFO ][o.o.p.PluginsService ] [opensearch-cluster-manager] PluginService:onIndexModule index:[validate-template–vrzjaearo6sgd1us8rfwa/zAhoasWwQQOycB9WdluwHg]
application-opensearch_manager1-1 | [2023-12-08T14:56:15,937][INFO ][o.o.c.m.MetadataIndexTemplateService] [opensearch-cluster-manager] updating index template [ss4o_traces_template] for index patterns [ss4o_traces--]
application-opensearch_manager1-1 | [2023-12-08T14:56:15,968][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch-cluster-manager] Detected cluster change event for destination migration
application-opensearch_manager1-1 | [2023-12-08T14:56:15,970][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [opensearch-cluster-manager] observability:Mapping Template ss4o_traces_template creation Acknowledged
application-opensearch_manager1-1 | [2023-12-08T14:56:16,060][ERROR][o.o.s.t.SecurityRequestHandler] [opensearch-cluster-manager] OpenSearchException[Transport client authentication no longer supported.]

Data logs:
application-opensearch_data1-1 | [2023-12-08T12:57:40,196][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-data-node-1] Wait for cluster to be available …
application-opensearch_data1-1 | [2023-12-08T12:57:41,049][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-data-node-1] handshake failed for [connectToRemoteMasterNode[10.0.0.70:9300]]
application-opensearch_data1-1 | org.opensearch.transport.RemoteTransportException: [opensearch-cluster-manager][10.0.0.70:9300][internal:transport/handshake]
application-opensearch_data1-1 | Caused by: org.opensearch.OpenSearchException: Transport client authentication no longer supported.
application-opensearch_data1-1 | at org.opensearch.security.ssl.util.ExceptionUtils.createTransportClientNoLongerSupportedException(ExceptionUtils.java:68) ~[?:?]
application-opensearch_data1-1 | at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:294) ~[?:?]
application-opensearch_data1-1 | at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:169) ~[?:?]
application-opensearch_data1-1 | at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:774) ~[?:?]
application-opensearch_data1-1 | at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:113) ~[?:?]
application-opensearch_data1-1 | at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
application-opensearch_data1-1 | at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-2.11.0.jar:2.11.0]
application-opensearch_data1-1 | at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:262) ~[opensearch-2.11.0.jar:2.11.0]
application-opensearch_data1-1 | at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:140) ~[opensearch-2.11.0.jar:2.11.0]
application-opensearch_data1-1 | at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123) ~[opensearch-2.11.0.jar:2.11.0]
application-opensearch_data1-1 | at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770) ~[opensearch-2.11.0.jar:2.11.0]
application-opensearch_data1-1 | at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-2.11.0.jar:2.11.0]
application-opensearch_data1-1 | at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-2.11.0.jar:2.11.0]
application-opensearch_data1-1 | at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-2.11.0.jar:2.11.0]
application-opensearch_data1-1 | at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
application-opensearch_data1-1 | at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
application-opensearch_data1-1 | at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
application-opensearch_data1-1 | at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1471) ~[?:?]
application-opensearch_data1-1 | at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1334) ~[?:?]
application-opensearch_data1-1 | at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1383) ~[?:?]
application-opensearch_data1-1 | at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
application-opensearch_data1-1 | at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
application-opensearch_data1-1 | at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
application-opensearch_data1-1 | at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
application-opensearch_data1-1 | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
application-opensearch_data1-1 | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
application-opensearch_data1-1 | at java.lang.Thread.run(Thread.java:833) [?:?]

2

Hi @rsen

Have you read the following topic? If not, please check the link below:

3

Hi! @Eugene7
Thank you for the response. Yes, i’ve read it, as i understood, securityadmin.sh requires HTTP connection - port 9200 and it was fixed here. I’m running 2.11 version where these bugs should have been fixed, but I still get them. Maybe i have missed something?

4

Hi @rsen ,

Is that OU=OU a correct value did you mean to :

?

If so, could you try adjusting your plugins.security.nodes_dn accordingly and let me know if it fixed the issues?

Thanks,
Mantas

5

Hi, @Mantas
Thank you for the response. I’ve fixed that, but it did not change anything. I have disabled demo certificates and recieve another errors from security plugin. I think if i fix them it will give me more space to interact with security plugin.
Thanks

6

Could you share these errors here, please?

Please note: disabling the demo certificates will switch your cluster to production mode, “before using OpenSearch in a production environment, you must configure the Security plugin manually with your own certificates, authentication method, users, and passwords.”

best,
Mantas

7

Hi, @Mantas , @Eugene7 !
I have fixed all the errors by disabling demo config, setting correct transport and http host, setting correct network driver in docker-compose file ( network_mode: “host”), added admin certificates, moving certificate configs to opensearch.yml instead of env variables in docker-compose, setting correct opensearch config parameters:

plugins.security.allow_default_init_securityindex: true
plugins.security.ssl.http.clientauth_mode: OPTIONAL

and modifying config.yml:

---

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: false
        order: 0
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
    authz: {}

I don’t really know what exactly fixed the error, but all these things together work well. I continue to secure opensearch cluster and close this topic. Thank you!

1 Like