OpenSearchException: Transport client authentication no longer supported

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 2.3.0

Describe the issue:
I’m getting the following message in the logs, and I’m not sure how to troubleshoot it or even where to look. The documentation says: “TLS is optional for the REST layer and mandatory for the transport layer.”. So, an error message saying that the mandatory requirement is no longer supported seem kind of stupid?

[2022-11-01T15:07:23,402][ERROR][o.o.s.t.SecurityRequestHandler] [test-cluster-master-1] OpenSearchException[Transport client authentication no longer supported.]
[2022-11-01T15:07:23,581][ERROR][o.o.s.t.SecurityRequestHandler] [test-cluster-master-1] OpenSearchException[Transport client authentication no longer supported.]
[2022-11-01T15:07:23,848][WARN ][o.o.d.HandshakingTransportAddressConnector] [test-cluster-master-1] handshake failed for [connectToRemoteMasterNode[]]
org.opensearch.transport.RemoteTransportException: [test-cluster-master-2][][internal:transport/handshake]
Caused by: org.opensearch.OpenSearchException: Transport client authentication no longer supported.
        at ~[?:?]
        at ~[?:?]
        at ~[?:?]
        at$7$1.messageReceived( ~[?:?]
        at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:100) ~[?:?]
        at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived( ~[?:?]
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived( ~[opensearch-2.3.0.jar:2.3.0]
        at org.opensearch.transport.InboundHandler.handleRequest( ~[opensearch-2.3.0.jar:2.3.0]
        at org.opensearch.transport.InboundHandler.messageReceived( ~[opensearch-2.3.0.jar:2.3.0]
        at org.opensearch.transport.InboundHandler.inboundMessage( ~[opensearch-2.3.0.jar:2.3.0]
        at org.opensearch.transport.TcpTransport.inboundMessage( ~[opensearch-2.3.0.jar:2.3.0]
        at org.opensearch.transport.InboundPipeline.forwardFragments( ~[opensearch-2.3.0.jar:2.3.0]
        at org.opensearch.transport.InboundPipeline.doHandleBytes( ~[opensearch-2.3.0.jar:2.3.0]
        at org.opensearch.transport.InboundPipeline.handleBytes( ~[opensearch-2.3.0.jar:2.3.0]
        at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead( ~[?:?]
        at ~[?:?]
        at ~[?:?]
        at ~[?:?]
        at io.netty.handler.logging.LoggingHandler.channelRead( ~[?:?]
        at ~[?:?]
        at ~[?:?]
        at ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead( ~[?:?]
        at ~[?:?]
        at ~[?:?]
        at ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap( ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible( ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode( ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection( ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode( ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead( ~[?:?]
        at ~[?:?]
        at ~[?:?]
        at ~[?:?]
        at$HeadContext.channelRead( ~[?:?]
        at ~[?:?]
        at ~[?:?]
        at ~[?:?]
        at$ ~[?:?]
        at ~[?:?]
        at ~[?:?]
        at ~[?:?]
        at ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$ ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$ ~[?:?]
        at [?:?]

I want to assume that this is just a case of confusing terminology, but I can’t seem to find anything about transport client configuration anywhere else than the settings, which as it states in the documentation is mandatory.

And the transport_enabled setting in the security config, but disabling that doesn’t help (and it’s enabled in the example configuration in the repo)


        # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
        # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
        # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
        #filtered_alias_mode: warn
        #do_not_fail_on_forbidden: false
          # Kibana multitenancy
          multitenancy_enabled: true
          server_username: kibanaserver
          index: '.kibana'
          anonymous_auth_enabled: false
            enabled: false
            #internalProxies: '10\.80\.\d+\.\d+' # regex pattern - Load balancer subnets.
            internalProxies: '.*' # trust all internal proxies, regex pattern
            remoteIpHeader:  'X-Forwarded-For'
            ###### see for regex help
            ###### more information about XFF
            ###### and here
            ###### and
            description: "Authenticate via HTTP Basic against internal users database"
            http_enabled: true
            transport_enabled: false
            order: 0
              type: basic
              challenge: false
              type: intern

clusterName: "test-cluster"
nodeGroup: "master"
masterService: "test-cluster-master"
replicas: 3
  - master

opensearchJavaOpts: "-Xmx512M -Xms512M"
    cpu: "100m"
    memory: "768Mi"

  enabled: true
  enableInitChown: false
  storageClass: "default"
  size: 8Gi
    - ReadWriteOnce
# Allows you to add any config files in {{ .Values.opensearchHome }}/config
opensearchHome: /usr/share/opensearch
# such as opensearch.yml and
  opensearch.yml: | test-cluster
    # Bind to all interfaces because we don't know what IP address Docker will assign to us.
            pemcert_filepath: certs/node.crt
            pemkey_filepath: certs/node.key
            pemtrustedcas_filepath: certs/ca.crt
            enforce_hostname_verification: false
            enabled: true
            pemcert_filepath: certs/node.crt
            pemkey_filepath: certs/node.key
            pemtrustedcas_filepath: certs/ca.crt
        allow_unsafe_democertificates: false
        allow_default_init_securityindex: true
          admin_dn: # Generated by
            - CN=admin,OU=test-cluster
        audit.type: internal_opensearch
        enable_snapshot_restore_privilege: true
        check_snapshot_restore_write_privileges: true
          roles_enabled: ["all_access", "security_rest_api_access"]
          enabled: true
    ######## End OpenSearch Security Demo Configuration ########

  - secretName: test-cluster-certs
    name: cluster-certs
    path: /usr/share/opensearch/config/certs

    value: "true"

    securityConfigSecret: test-cluster-securityconfig

**Relevant Logs or Screenshots**:

@albgus I think you’ve faced the scenario described in this thread.

This issue is not present when demo certificates are in use. However, when you migrate to production then you must configure

Any node that won’t be in that list, will produce the reported error in OpenSearch logs.
Please remember that the values of the have to match the node certificate.

The error says that TLS is optional for the REST. That is correct. The REST connection is on port 9200. This can be either secured or not. However, since transport layer auth/auth is no longer supported, has moved from port 9300 to 9200. That forces port 9200 to be secured for running script.

TLS always was and is mandatory for the transport layer (ports 9300 - 9400).

To follow up on this, the root cause ended up being an error in the certificates. I had based the configuration on a setup that relied in setting subjectAltName=RID: on the certs instead of defining Which is apparently valid and works but wasn’t clear.

Still, that error message is really bad, as it explicitly says that a required configuration has been removed and no longer supported.