Config Issue Opensearch not running after "Admin password set successfully"

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch v2.14.0 using helm chart

Describe the issue:
Once pods is provisioned it get stuck in the msg “Admin password set successfully.” post that container is not running. it keeps restart

Configuration:
This my values.yaml file.

---
clusterName: "opensearch-cluster"
nodeGroup: "master"

roles:
  - master
  - remote_cluster_client

replicas: 2

config:
  opensearch.yml: |
    cluster.name: opensearch-cluster

    # Bind to all interfaces because we don't know what IP address Docker will assign to us.
    network.host: 0.0.0.0

    # Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
    # Implicitly done if ".singleNode" is set to "true".
    # discovery.type: single-node

    # Start OpenSearch Security Demo Configuration
    # WARNING: revise all the lines below before you go into production
    plugins:
      security:
        ssl:
          transport:
            pemcert_filepath: esnode.pem
            pemkey_filepath: esnode-key.pem
            pemtrustedcas_filepath: root-ca.pem
            enforce_hostname_verification: false
          http:
            enabled: true
            anonymous_auth_enabled: true
            pemcert_filepath: esnode.pem
            pemkey_filepath: esnode-key.pem
            pemtrustedcas_filepath: root-ca.pem
        allow_unsafe_democertificates: true
        allow_default_init_securityindex: true
        authcz:
          admin_dn:
            - CN=kirk,OU=client,O=client,L=test,C=de
        audit.type: internal_opensearch
        enable_snapshot_restore_privilege: true
        check_snapshot_restore_write_privileges: true
        restapi:
          roles_enabled: ["all_access", "security_rest_api_access"]
        system_indices:
          enabled: true
          indices:
            [
              ".opendistro-alerting-config",
              ".opendistro-alerting-alert*",
              ".opendistro-anomaly-results*",
              ".opendistro-anomaly-detector*",
              ".opendistro-anomaly-checkpoints",
              ".opendistro-anomaly-detection-state",
              ".opendistro-reports-*",
              ".opendistro-notifications-*",
              ".opendistro-notebooks",
              ".opendistro-asynchronous-search-response*",
            ]
    ######## End OpenSearch Security Demo Configuration ########
  # log4j2.properties:

Relevant Logs or Screenshots:

This option is valid for config.yml as per OpenSearch documentation. Please remove it from opensearch.yml and try to deploy the cluster again.

@pablo Thankyou. Yes I have removed the complete opensearch config file now I have a script like below. Now pods are running but with SSL error. I have attached my script and the error

##Value,yaml.

---
clusterName: "opensearch-cluster"
nodeGroup: "master"

roles:
  - master
  - remote_cluster_client

replicas: 2

extraEnvs:
  - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
    value: rggwee
  - name: AWS_WEB_IDENTITY_TOKEN_FILE
    value: '/usr/share/opensearch/config/repository-s3/aws-web-identity-token-file'
  - name: AWS_ROLE_ARN
    value: ${AWS_ROLE_ARN}
  - name: anonymous_auth_enabled
    value: 'true'


Error

Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f5f6e6f6465733f66696c7465725f706174683d6e6f6465732e2a2e76657273696f6e2532436e6f6465732e2a2e687474702e7075626c6973685f616464726573732532436e6f6465732e2a2e697020485454502f312e310d0a617574686f72697a6174696f6e3a2042617369632061326c695957356863325679646d56794f6d7470596d467559584e6c636e5a6c63673d3d0d0a757365722d6167656e743a206f70656e7365617263682d6a732f312e312e3020286c696e757820362e312e39302d39392e3137332e616d7a6e323032332e7838365f36342d7836343b204e6f64652e6a73207631382e31392e30290d0a782d6f70656e7365617263682d70726f647563742d6f726967696e3a206f70656e7365617263682d64617368626f617264730d0a486f73743a206f70656e7365617263682d636c75737465722d6d61737465723a393230300d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a0d0a

at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1314) ~[netty-handler-4.1.109.Final.jar:4.1.109.Final]

at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[netty-handler-4.1.109.Final.jar:4.1.109.Final]

at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[netty-codec-4.1.109.Final.jar:4.1.109.Final]

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[netty-codec-4.1.109.Final.jar:4.1.109.Final]

... 16 more

[2024-06-19T05:35:13,370][ERROR][o.o.h.n.s.SecureNetty4HttpServerTransport] [opensearch-cluster-master-1] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f5f6e6f6465733f66696c7465725f706174683d6e6f6465732e2a2e76657273696f6e2532436e6f6465732e2a2e687474702e7075626c6973685f616464726573732532436e6f6465732e2a2e697020485454502f312e310d0a617574686f72697a6174696f6e3a2042617369632061326c695957356863325679646d56794f6d7470596d467559584e6c636e5a6c63673d3d0d0a757365722d6167656e743a206f70656e7365617263682d6a732f312e312e3020286c696e757820362e312e39302d39392e3137332e616d7a6e323032332e7838365f36342d7836343b204e6f64652e6a73207631382e31392e30290d0a782d6f70656e7365617263682d70726f647563742d6f726967696e3a206f70656e7365617263682d64617368626f617264730d0a486f73743a206f70656e7365617263682d636c75737465722d6d61737465723a393230300d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a0d0a

io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f5f6e6f6465733f66696c7465725f706174683d6e6f6465732e2a2e76657273696f6e2532436e6f6465732e2a2e687474702e7075626c6973685f616464726573732532436e6f6465732e2a2e697020485454502f312e310d0a617574686f72697a6174696f6e3a2042617369632061326c695957356863325679646d56794f6d7470596d467559584e6c636e5a6c63673d3d0d0a757365722d6167656e743a206f70656e7365617263682d6a732f312e312e3020286c696e757820362e312e39302d39392e3137332e616d7a6e323032332e7838365f36342d7836343b204e6f64652e6a73207631382e31392e30290d0a782d6f70656e7365617263682d70726f647563742d6f726967696e3a206f70656e7365617263682d64617368626f617264730d0a486f73743a206f70656e7365617263682d636c75737465722d6d61737465723a393230300d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a0d0a

at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1314) ~[netty-handler-4.1.109.Final.jar:4.1.109.Final]

at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[netty-handler-4.1.109.Final.jar:4.1.109.Final]

at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[netty-codec-4.1.109.Final.jar:4.1.109.Final]

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[netty-codec-4.1.109.Final.jar:4.1.109.Final]

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.109.Final.jar:4.1.109.Final]

at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.109.Final.jar:4.1.109.Final]

at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

[2024-06-19T05:35:13,371][WARN ][o.o.h.AbstractHttpServerTransport] [opensearch-cluster-master-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.122.145.48:9200, remoteAddress=/10.122.145.110:43046}

io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f5f6e6f6465733f66696c7465725f706174683d6e6f6465732e2a2e76657273696f6e2532436e6f6465732e2a2e687474702e7075626c6973685f616464726573732532436e6f6465732e2a2e697020485454502f312e310d0a617574686f72697a6174696f6e3a2042617369632061326c695957356863325679646d56794f6d7470596d467559584e6c636e5a6c63673d3d0d0a757365722d6167656e743a206f70656e7365617263682d6a732f312e312e3020286c696e757820362e312e39302d39392e3137332e616d7a6e323032332e7838365f36342d7836343b204e6f64652e6a73207631382e31392e30290d0a782d6f70656e7365617263682d70726f647563742d6f726967696e3a206f70656e7365617263682d64617368626f617264730d0a486f73743a206f70656e7365617263682d636c75737465722d6d61737465723a393230300d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a0d0a

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:500) ~[netty-codec-4.1.109.Final.jar:4.1.109.Final]

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.109.Final.jar:4.1.109.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.109.Final.jar:4.1.109.Final]

at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.109.Final.jar:4.1.109.Final]

at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f5f6e6f6465733f66696c7465725f706174683d6e6f6465732e2a2e76657273696f6e2532436e6f6465732e2a2e687474702e7075626c6973685f616464726573732532436e6f6465732e2a2e697020485454502f312e310d0a617574686f72697a6174696f6e3a2042617369632061326c695957356863325679646d56794f6d7470596d467559584e6c636e5a6c63673d3d0d0a757365722d6167656e743a206f70656e7365617263682d6a732f312e312e3020286c696e757820362e312e39302d39392e3137332e616d7a6e323032332e7838365f36342d7836343b204e6f64652e6a73207631382e31392e30290d0a782d6f70656e7365617263682d70726f647563742d6f726967696e3a206f70656e7365617263682d64617368626f617264730d0a486f73743a206f70656e7365617263682d636c75737465722d6d61737465723a393230300d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a0d0a

at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1314) ~[netty-handler-4.1.109.Final.jar:4.1.109.Final]

at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[netty-handler-4.1.109.Final.jar:4.1.109.Final]

at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[netty-codec-4.1.109.Final.jar:4.1.109.Final]

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[netty-codec-4.1.109.Final.jar:4.1.109.Final]

@learnawsnaresh Only the anonymous part was incorrect. The remaining part of the opensearch.yml is required to configure OpenSearch security plugin.
This plugin is enabled by default and SSL connection at the Transport layer is mandatory.

@pablo If I use the “plugins.security.disabled: true” under the opensearch.yaml is that SSL/TSL mandatory ? I have attached the Warn and pods status for master node, and data nodes and dashboard config.

values.yaml

config:
opensearch.yml: |
cluster.name: opensearch-cluster

network.host: 0.0.0.0

######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.disabled: true
# plugins.security.ssl.transport.pemcert_filepath: esnode.pem
# plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
# plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
# plugins.security.ssl.transport.enforce_hostname_verification: false
# plugins.security.ssl.http.enabled: true
# plugins.security.ssl.http.pemcert_filepath: esnode.pem
# plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
# plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
# plugins.security.allow_unsafe_democertificates: true
# plugins.security.allow_default_init_securityindex: true
# plugins.security.authcz.admin_dn: ['CN=kirk,OU=client,O=client,L=test,C=de']
# plugins.security.audit.type: internal_opensearch
# plugins.security.enable_snapshot_restore_privilege: true
# plugins.security.check_snapshot_restore_write_privileges: true
# plugins.security.restapi.roles_enabled: [all_access, security_rest_api_access]
# plugins.security.system_indices.enabled: true
# plugins.security.system_indices.indices: [.plugins-ml-agent, .plugins-ml-config, .plugins-ml-connector,
#   .plugins-ml-controller, .plugins-ml-model-group, .plugins-ml-model, .plugins-ml-task,
#   .plugins-ml-conversation-meta, .plugins-ml-conversation-interactions, .plugins-ml-memory-meta,
#   .plugins-ml-memory-message, .plugins-ml-stop-words, .opendistro-alerting-config,
#   .opendistro-alerting-alert*, .opendistro-anomaly-results*, .opendistro-anomaly-detector*,
#   .opendistro-anomaly-checkpoints, .opendistro-anomaly-detection-state, .opendistro-reports-*,
#   .opensearch-notifications-*, .opensearch-notebooks, .opensearch-observability, .ql-datasources,
#   .opendistro-asynchronous-search-response*, .replication-metadata-store, .opensearch-knn-models,
#   .geospatial-ip2geo-data*, .plugins-flow-framework-config, .plugins-flow-framework-templates,
#   .plugins-flow-framework-state]
# node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########

master node console:

[2024-06-19T10:40:17,917][WARN ][o.o.t.TcpTransport       ] [opensearch-cluster-master-1] SSL/TLS request received but SSL/TLS is not enabled on this node, got (16,3,3,1), [Netty4TcpChannel{localAddress=/10.122.145.34:9300, remoteAddress=/10.122.145.110:37078}], closing connection
[2024-06-19T10:40:18,859][WARN ][o.o.t.TcpTransport       ] [opensearch-cluster-master-1] SSL/TLS request received but SSL/TLS is not enabled on this node, got (16,3,3,1), [Netty4TcpChannel{localAddress=/10.122.145.34:9300, remoteAddress=/10.122.145.253:54902}], closing connection
[2024-06-19T10:40:18,916][WARN ][o.o.t.TcpTransport       ] [opensearch-cluster-master-1] SSL/TLS request received but SSL/TLS is not enabled on this node, got (16,3,3,1), [Netty4TcpChannel{localAddress=/10.122.145.34:9300,

You’ve placed this value correctly in the master nodes - under opensearch.yml but you didn’t repeat that for data nodes.
As a result data nodes have security plugin enabled and trying to connect over transport layer with other nodes using SSL.

@pablo Thankyou. Now after setting the value “plugins.security.disabled: true” in the data the master node now I got the opensearch login page. We will use opensearch internally and we have our own authentication system using the AWS cognito, now I am facing another challenge like Opensearch dashboard asking for a username and password. I want to disable the authentication.

I am giving my final script of data, master, and dashboard here. Guide me where and what changes I have to do further.

@learnawsnaresh You need to disable security plugin in OpenSearch Dashboards.

Hi, I am using only 3 values in the opensearch_dashboard.yaml file remaining all I have commented as below. But still getting the login page.

server.host: '0.0.0.0'
server.name: opensearch-dashboards
opensearch.hosts: [http://opensearch-cluster-master:9200]

@learnawsnaresh The login page appears as the security plugin for OpenSearch Dashboards is still installed. You need to either remove it or disable it by adding the below section.

extraEnvs:
  - name: "DISABLE_SECURITY_DASHBOARDS_PLUGIN"
    value: "true"

@pablo Thankyou. it worked.

1 Like