Opensearch-2.3.0 multinode SSL setup issue

bimlesh_singh · December 1, 2022, 12:56pm

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 2.3.0

Describe the issue: We were trying to evaluate Opensearch 2.3.0 for replacing Elasticsearch service.
The setup steps go below-

We have two VMs-xx-xx-xx-123 and xx-xx-xx-456
Extracted tar files on both VMs with sudo user account
Ran command opensearch-tar-install.sh which configured default TLS and default SSL files and other security related setup.
multi-node setup worked fine for one cluster-manger node and another as data node by making small changes in opensearch.yml file.
Now generated self-signed certificates like-ca-cert.pem,admin.pem,admin-key.pem,node1-pem,node1-key.pem
Updated opensearch.yml files with generated ones (self-signed ones)
added cluster.initial_cluster_manager_nodes: [“cluster-manager-node”] and node.roles: [cluster_manager] on cluster manager VM
On Data VM, added node.roles: [data] in opensearch.yml
imported ca-cert.pem in opensearch’s cacerts file in internal jdk on both VMs
9.Started the services on both VMs but nodes are started showing error after using self-signed certificate.

Configuration: 1 node as cluster-manager and another node as data node.

Relevant Logs or Screenshots:
[2022-12-01T17:37:20,132][WARN ][o.o.c.c.ClusterFormationFailureHelper] [data-node1] cluster-manager not discovered yet: have discovered [{data-node1}{u8O7kcb9RtOYZeFugHB17g}{whMAJN4KRv6coxLQd-9XeA}{10.46.34.100}{10.46.34.100:9301}{d}{shard_indexing_pressure_enabled=true}]; discovery will continue using [10.46.34.60:9301] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-12-01T17:37:25,291][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [data-node1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.3.0.jar:2.3.0]
at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.3.0.jar:2.3.0]
at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.3.0.jar:2.3.0]
at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.3.0.jar:2.3.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.3.0.jar:2.3.0]
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.3.0.0.jar:2.3.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.3.0.jar:2.3.0]
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.3.0.0.jar:2.3.0.0]

pablo · December 1, 2022, 1:14pm

@bimlesh_singh Please share your config.yml and opensearch.yml from both OS nodes.

bimlesh_singh · December 1, 2022, 1:52pm

Hi Pablo,
We have not modified config.yml on any VM so its contains default value.
And attached opensearch.yml content from both vm’s.
We have generated self-signed certificates with the same name as default ones.

opensearch.yml[cluster-manager - node]

node.name: cluster-manager-node
cluster.name: es-cluster
indices.query.bool.max_clause_count: 10000
transport.compress: true
transport.port: 9300
transport.tcp.port: 9300
node.roles: [cluster_manager]
network.host: XX.XX.XX.60
http.port: 9200
discovery.seed_hosts: [“XX.XX.XX.60:9300”, “XX.XX.XX.100:9300”]
cluster.initial_cluster_manager_nodes: [“cluster-manager-node”]
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

CN=XXX,OU=XXX,O=XXX,L=XXX, C=XX

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [“.plugins-ml-model”, “.plugins-ml-task”, “.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opensearch-notifications-”, “.opensearch-notebooks”, “.opensearch-observability”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”]
node.max_local_storage_nodes: 3

opensearch.yml[data - node]

node.name: data-node
cluster.name: es-cluster
indices.query.bool.max_clause_count: 10000
transport.compress: true
transport.port: 9300
transport.tcp.port: 9300
node.roles: [data]
network.host: XX.XX.XX.100
http.port: 9200
discovery.seed_hosts: [“XX.XX.XX.60:9300”, “XX.XX.XX.100:9300”]
cluster.initial_cluster_manager_nodes: [“cluster-manager-node”]
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

CN=XXX,OU=XXX,O=XXX,L=XXX, C=XX

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [“.plugins-ml-model”, “.plugins-ml-task”, “.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opensearch-notifications-”, “.opensearch-notebooks”, “.opensearch-observability”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”]
node.max_local_storage_nodes: 3

bimlesh_singh · December 1, 2022, 3:11pm

Hi Pablo, we are doing this step on standalone VMs and both VMs are pingable from each other. When we use self-signed cert on nodes, the discovery of cluster-manager fails on data node.
Also, if you could help us in determining whether Opensearch multi cluster node is possible over On-Premises Linux VMs (without AWS hosted Linux servers)?

pablo · December 1, 2022, 7:23pm

@bimlesh_singh Did you copy all certificates to both nodes? What is the path to the certs in the OpenSearch filesystem?

bimlesh_singh · December 2, 2022, 8:07am

Hi Pablo, we have copied respective self-signed certificates in their respective <OPENSEARCH_HOME>/config/ on both VMs.
We have updatd cacert file with ssl certifcate and ca-certifcate as well.
Log from cluster-manager VM-
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/home/elastic/opensearch-2.3.0/lib/opensearch-2.3.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/home/elastic/opensearch-2.3.0/lib/opensearch-2.3.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2022-12-02T13:26:28,701][INFO ][o.o.n.Node ] [cluster-manager-node] version[2.3.0], pid[12437], build[tar/6f6e84ebc54af31a976f53af36a5c69d474a5140/2022-09-09T00:07:12.137133581Z], OS[Linux/3.10.0-1160.62.1.el7.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.4/17.0.4+8]
[2022-12-02T13:26:28,703][INFO ][o.o.n.Node ] [cluster-manager-node] JVM home [/home/elastic/opensearch-2.3.0/jdk], using bundled JDK [true]
[2022-12-02T13:26:28,704][INFO ][o.o.n.Node ] [cluster-manager-node] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-5366195585817922009, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/home/elastic/opensearch-2.3.0/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/home/elastic/opensearch-2.3.0, -Dopensearch.path.conf=/home/elastic/opensearch-2.3.0/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2022-12-02T13:26:31,780][WARN ][stderr ] [cluster-manager-node] SLF4J: Failed to load class “org.slf4j.impl.StaticLoggerBinder”.
[2022-12-02T13:26:31,781][WARN ][stderr ] [cluster-manager-node] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2022-12-02T13:26:31,781][WARN ][stderr ] [cluster-manager-node] SLF4J: See SLF4J Error Codes for further details.
[2022-12-02T13:26:31,806][INFO ][o.o.s.s.t.SSLConfig ] [cluster-manager-node] SSL dual mode is disabled
[2022-12-02T13:26:31,806][INFO ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] OpenSearch Config path is /home/elastic/opensearch-2.3.0/config
[2022-12-02T13:26:32,511][INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-manager-node] JVM supports TLSv1.3
[2022-12-02T13:26:32,515][INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-manager-node] Config directory is /home/elastic/opensearch-2.3.0/config/, from there the key- and truststore files are resolved relatively
[2022-12-02T13:26:33,349][INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-manager-node] TLS Transport Client Provider : JDK
[2022-12-02T13:26:33,350][INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-manager-node] TLS Transport Server Provider : JDK
[2022-12-02T13:26:33,351][INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-manager-node] TLS HTTP Provider : JDK
[2022-12-02T13:26:33,351][INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-manager-node] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2022-12-02T13:26:33,351][INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-manager-node] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[2022-12-02T13:26:33,907][INFO ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] Clustername: es-cluster
[2022-12-02T13:26:33,925][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] Directory /home/elastic/opensearch-2.3.0/config has insecure file permissions (should be 0700)
[2022-12-02T13:26:33,925][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/kirk.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,926][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/esnode.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,926][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/esnode-key.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,926][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/kirk-key.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,927][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/root-ca_old.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,927][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/admin.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,927][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/admin-key.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,928][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/node1.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,928][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/node1-key.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,928][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/node2.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,929][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/root-ca.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:35,343][INFO ][o.o.p.c.PluginSettings ] [cluster-manager-node] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2022-12-02T13:26:36,612][INFO ][o.o.i.r.ReindexPlugin ] [cluster-manager-node] ReindexPlugin reloadSPI called
[2022-12-02T13:26:36,614][INFO ][o.o.i.r.ReindexPlugin ] [cluster-manager-node] Unable to find any implementation for RemoteReindexExtension
[2022-12-02T13:26:36,709][INFO ][o.o.j.JobSchedulerPlugin ] [cluster-manager-node] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2022-12-02T13:26:36,777][INFO ][o.o.j.JobSchedulerPlugin ] [cluster-manager-node] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2022-12-02T13:26:36,780][INFO ][o.o.j.JobSchedulerPlugin ] [cluster-manager-node] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-12-02T13:26:36,800][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [aggs-matrix-stats]
[2022-12-02T13:26:36,801][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [analysis-common]
[2022-12-02T13:26:36,801][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [geo]
[2022-12-02T13:26:36,801][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [ingest-common]
[2022-12-02T13:26:36,802][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [ingest-geoip]
[2022-12-02T13:26:36,802][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [ingest-user-agent]
[2022-12-02T13:26:36,802][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [lang-expression]
[2022-12-02T13:26:36,803][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [lang-mustache]
[2022-12-02T13:26:36,803][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [lang-painless]
[2022-12-02T13:26:36,803][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [mapper-extras]
[2022-12-02T13:26:36,803][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [opensearch-dashboards]
[2022-12-02T13:26:36,804][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [parent-join]
[2022-12-02T13:26:36,804][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [percolator]
[2022-12-02T13:26:36,818][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [rank-eval]
[2022-12-02T13:26:36,818][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [reindex]
[2022-12-02T13:26:36,819][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [repository-url]
[2022-12-02T13:26:36,819][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [systemd]
[2022-12-02T13:26:36,819][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [transport-netty4]
[2022-12-02T13:26:36,820][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-alerting]
[2022-12-02T13:26:36,820][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-anomaly-detection]
[2022-12-02T13:26:36,821][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-asynchronous-search]
[2022-12-02T13:26:36,821][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-cross-cluster-replication]
[2022-12-02T13:26:36,821][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-geospatial]
[2022-12-02T13:26:36,821][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-index-management]
[2022-12-02T13:26:36,822][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-job-scheduler]
[2022-12-02T13:26:36,822][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-knn]
[2022-12-02T13:26:36,822][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-ml]
[2022-12-02T13:26:36,823][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-notifications]
[2022-12-02T13:26:36,823][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-notifications-core]
[2022-12-02T13:26:36,823][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-observability]
[2022-12-02T13:26:36,823][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-performance-analyzer]
[2022-12-02T13:26:36,824][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-reports-scheduler]
[2022-12-02T13:26:36,824][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-security]
[2022-12-02T13:26:36,824][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-sql]
[2022-12-02T13:26:36,886][INFO ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting ‘http.compression: true’ in opensearch.yml
[2022-12-02T13:26:36,943][INFO ][o.o.e.NodeEnvironment ] [cluster-manager-node] using [1] data paths, mounts [[/ (/dev/mapper/centos-root)]], net usable_space [22.6gb], net total_space [143.4gb], types [xfs]
[2022-12-02T13:26:36,944][INFO ][o.o.e.NodeEnvironment ] [cluster-manager-node] heap size [1gb], compressed ordinary object pointers [true]
[2022-12-02T13:26:37,224][INFO ][o.o.n.Node ] [cluster-manager-node] node name [cluster-manager-node], node ID [7nRIqu1xT4uLuVk0e-5byA], cluster name [es-cluster], roles [cluster_manager]
[2022-12-02T13:26:45,617][WARN ][o.o.s.c.Salt ] [cluster-manager-node] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2022-12-02T13:26:45,734][INFO ][o.o.s.a.i.AuditLogImpl ] [cluster-manager-node] Message routing enabled: true
[2022-12-02T13:26:45,772][INFO ][o.o.s.f.SecurityFilter ] [cluster-manager-node] indices are made immutable.
[2022-12-02T13:26:46,657][INFO ][o.o.a.b.ADCircuitBreakerService] [cluster-manager-node] Registered memory breaker.
[2022-12-02T13:26:47,471][INFO ][o.o.m.c.b.MLCircuitBreakerService] [cluster-manager-node] Registered ML memory breaker.
[2022-12-02T13:26:48,874][INFO ][o.o.t.NettyAllocator ] [cluster-manager-node] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2022-12-02T13:26:49,136][INFO ][o.o.d.DiscoveryModule ] [cluster-manager-node] using discovery type [zen] and seed hosts providers [settings]
[2022-12-02T13:26:50,511][WARN ][o.o.g.DanglingIndicesState] [cluster-manager-node] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-12-02T13:26:51,581][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [cluster-manager-node] PerformanceAnalyzer Enabled: false
[2022-12-02T13:26:51,626][INFO ][o.o.n.Node ] [cluster-manager-node] initialized
[2022-12-02T13:26:51,626][INFO ][o.o.n.Node ] [cluster-manager-node] starting …
[2022-12-02T13:26:51,834][INFO ][o.o.t.TransportService ] [cluster-manager-node] publish_address {XX.XX.XX.100:9300}, bound_addresses {XX.XX.XX.100:9300}
[2022-12-02T13:26:52,286][INFO ][o.o.b.BootstrapChecks ] [cluster-manager-node] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-12-02T13:26:52,304][INFO ][o.o.c.c.Coordinator ] [cluster-manager-node] cluster UUID [r2Ho1zFbQVG5Tw7b1D9Xow]
[2022-12-02T13:26:52,522][INFO ][o.o.c.s.MasterService ] [cluster-manager-node] elected-as-cluster-manager ([1] nodes joined)[{cluster-manager-node}{7nRIqu1xT4uLuVk0e-5byA}{Y5gDvq8xQSe5FjWV5PnfCw}{XX.XX.XX.100}{XX.XX.XX.100:9300}{m}{shard_indexing_pressure_enabled=true} elect leader, BECOME_CLUSTER_MANAGER_TASK, FINISH_ELECTION], term: 20, version: 68, delta: cluster-manager node changed {previous , current [{cluster-manager-node}{7nRIqu1xT4uLuVk0e-5byA}{Y5gDvq8xQSe5FjWV5PnfCw}{XX.XX.XX.100}{XX.XX.XX.100:9300}{m}{shard_indexing_pressure_enabled=true}]}
[2022-12-02T13:26:52,617][INFO ][o.o.c.s.ClusterApplierService] [cluster-manager-node] cluster-manager node changed {previous , current [{cluster-manager-node}{7nRIqu1xT4uLuVk0e-5byA}{Y5gDvq8xQSe5FjWV5PnfCw}{XX.XX.XX.100}{XX.XX.XX.100:9300}{m}{shard_indexing_pressure_enabled=true}]}, term: 20, version: 68, reason: Publication{term=20, version=68}
[2022-12-02T13:26:52,640][INFO ][o.o.a.c.ADClusterEventListener] [cluster-manager-node] Cluster is not recovered yet.
[2022-12-02T13:26:52,647][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [cluster-manager-node] Detected cluster change event for destination migration
[2022-12-02T13:26:52,687][INFO ][o.o.i.i.ManagedIndexCoordinator] [cluster-manager-node] Cache cluster manager node onClusterManager time: 1669967812687
[2022-12-02T13:26:52,692][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [cluster-manager-node] Config override setting update called with empty string. Ignoring.
[2022-12-02T13:26:52,722][INFO ][o.o.h.AbstractHttpServerTransport] [cluster-manager-node] publish_address {XX.XX.XX.100:9200}, bound_addresses {XX.XX.XX.100:9200}
[2022-12-02T13:26:52,723][INFO ][o.o.n.Node ] [cluster-manager-node] started
[2022-12-02T13:26:52,723][INFO ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] Node started
[2022-12-02T13:26:52,724][INFO ][o.o.s.c.ConfigurationRepository] [cluster-manager-node] Will attempt to create index .opendistro_security and default configs if they are absent
[2022-12-02T13:26:52,762][INFO ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] 0 OpenSearch Security modules loaded so far:
[2022-12-02T13:26:52,817][INFO ][o.o.s.c.ConfigurationRepository] [cluster-manager-node] Background init thread started. Install default config?: true
[2022-12-02T13:26:52,874][INFO ][o.o.c.s.ClusterSettings ] [cluster-manager-node] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
[2022-12-02T13:26:52,883][INFO ][o.o.a.c.HashRing ] [cluster-manager-node] Node added: [7nRIqu1xT4uLuVk0e-5byA]
[2022-12-02T13:26:52,886][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [cluster-manager-node] Detected cluster change event for destination migration
[2022-12-02T13:26:52,891][INFO ][o.o.a.c.HashRing ] [cluster-manager-node] All nodes with known AD version: {7nRIqu1xT4uLuVk0e-5byA=ADNodeInfo{version=2.3.0, isEligibleDataNode=false}}
[2022-12-02T13:26:52,892][INFO ][o.o.a.c.HashRing ] [cluster-manager-node] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
[2022-12-02T13:26:52,892][INFO ][o.o.a.c.HashRing ] [cluster-manager-node] Build AD version hash ring successfully
[2022-12-02T13:26:52,893][INFO ][o.o.a.c.ADClusterEventListener] [cluster-manager-node] Init AD version hash ring successfully
[2022-12-02T13:26:52,935][INFO ][o.o.g.GatewayService ] [cluster-manager-node] recovered [3] indices into cluster_state
[2022-12-02T13:26:52,939][INFO ][o.o.s.c.ConfigurationRepository] [cluster-manager-node] Index .opendistro_security already exists
[2022-12-02T13:26:52,939][INFO ][o.o.s.c.ConfigurationRepository] [cluster-manager-node] Node started, try to initialize it. Wait for at least yellow cluster state…
[2022-12-02T13:27:52,693][INFO ][o.o.i.i.ManagedIndexCoordinator] [cluster-manager-node] Performing move cluster state metadata.
[2022-12-02T13:27:52,697][INFO ][o.o.i.i.MetadataService ] [cluster-manager-node] ISM config index not exist, so we cancel the metadata migration job.
[2022-12-02T13:28:25,620][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [cluster-manager-node] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?]
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.79.Final.jar:4.1.79.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.79.Final.jar:4.1.79.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-12-02T13:28:25,636][WARN ][o.o.t.TcpTransport ] [cluster-manager-node] exception caught on transport layer [Netty4TcpChannel{localAddress=/XX.XX.XX.100:9300, remoteAddress=/XX.XX.XX.60:32854}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.79.Final.jar:4.1.79.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.79.Final.jar:4.1.79.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?]
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
… 16 more
[2022-12-02T13:28:26,213][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [cluster-manager-node] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

Gsmitt · December 6, 2022, 12:00am

Hello @bimlesh_singh

First, Its really hard to read the log file you posted, SO taking initiative and here is your log file cleaned up so its easier to read.

WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/home/elastic/opensearch-2.3.0/lib/opensearch-2.3.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/home/elastic/opensearch-2.3.0/lib/opensearch-2.3.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2022-12-02T13:26:28,701][INFO ][o.o.n.Node ] [cluster-manager-node] version[2.3.0], pid[12437], build[tar/6f6e84ebc54af31a976f53af36a5c69d474a5140/2022-09-09T00:07:12.137133581Z], OS[Linux/3.10.0-1160.62.1.el7.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.4/17.0.4+8]
[2022-12-02T13:26:28,703][INFO ][o.o.n.Node ] [cluster-manager-node] JVM home [/home/elastic/opensearch-2.3.0/jdk], using bundled JDK [true]
[2022-12-02T13:26:28,704][INFO ][o.o.n.Node ] [cluster-manager-node] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-5366195585817922009, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/home/elastic/opensearch-2.3.0/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/home/elastic/opensearch-2.3.0, -Dopensearch.path.conf=/home/elastic/opensearch-2.3.0/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2022-12-02T13:26:31,780][WARN ][stderr ] [cluster-manager-node] SLF4J: Failed to load class “org.slf4j.impl.StaticLoggerBinder”.
[2022-12-02T13:26:31,781][WARN ][stderr ] [cluster-manager-node] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2022-12-02T13:26:31,781][WARN ][stderr ] [cluster-manager-node] SLF4J: See SLF4J Error Codes for further details.
[2022-12-02T13:26:31,806][INFO ][o.o.s.s.t.SSLConfig ] [cluster-manager-node] SSL dual mode is disabled
[2022-12-02T13:26:31,806][INFO ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] OpenSearch Config path is /home/elastic/opensearch-2.3.0/config
[2022-12-02T13:26:32,511][INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-manager-node] JVM supports TLSv1.3
[2022-12-02T13:26:32,515][INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-manager-node] Config directory is /home/elastic/opensearch-2.3.0/config/, from there the key- and truststore files are resolved relatively
[2022-12-02T13:26:33,349][INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-manager-node] TLS Transport Client Provider : JDK
[2022-12-02T13:26:33,350][INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-manager-node] TLS Transport Server Provider : JDK
[2022-12-02T13:26:33,351][INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-manager-node] TLS HTTP Provider : JDK
[2022-12-02T13:26:33,351][INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-manager-node] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2022-12-02T13:26:33,351][INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-manager-node] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[2022-12-02T13:26:33,907][INFO ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] Clustername: es-cluster
[2022-12-02T13:26:33,925][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] Directory /home/elastic/opensearch-2.3.0/config has insecure file permissions (should be 0700)
[2022-12-02T13:26:33,925][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/kirk.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,926][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/esnode.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,926][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/esnode-key.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,926][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/kirk-key.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,927][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/root-ca_old.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,927][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/admin.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,927][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/admin-key.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,928][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/node1.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,928][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/node1-key.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,928][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/node2.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:33,929][WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] File /home/elastic/opensearch-2.3.0/config/root-ca.pem has insecure file permissions (should be 0600)
[2022-12-02T13:26:35,343][INFO ][o.o.p.c.PluginSettings ] [cluster-manager-node] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2022-12-02T13:26:36,612][INFO ][o.o.i.r.ReindexPlugin ] [cluster-manager-node] ReindexPlugin reloadSPI called
[2022-12-02T13:26:36,614][INFO ][o.o.i.r.ReindexPlugin ] [cluster-manager-node] Unable to find any implementation for RemoteReindexExtension
[2022-12-02T13:26:36,709][INFO ][o.o.j.JobSchedulerPlugin ] [cluster-manager-node] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2022-12-02T13:26:36,777][INFO ][o.o.j.JobSchedulerPlugin ] [cluster-manager-node] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2022-12-02T13:26:36,780][INFO ][o.o.j.JobSchedulerPlugin ] [cluster-manager-node] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-12-02T13:26:36,800][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [aggs-matrix-stats]
[2022-12-02T13:26:36,801][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [analysis-common]
[2022-12-02T13:26:36,801][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [geo]
[2022-12-02T13:26:36,801][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [ingest-common]
[2022-12-02T13:26:36,802][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [ingest-geoip]
[2022-12-02T13:26:36,802][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [ingest-user-agent]
[2022-12-02T13:26:36,802][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [lang-expression]
[2022-12-02T13:26:36,803][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [lang-mustache]
[2022-12-02T13:26:36,803][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [lang-painless]
[2022-12-02T13:26:36,803][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [mapper-extras]
[2022-12-02T13:26:36,803][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [opensearch-dashboards]
[2022-12-02T13:26:36,804][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [parent-join]
[2022-12-02T13:26:36,804][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [percolator]
[2022-12-02T13:26:36,818][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [rank-eval]
[2022-12-02T13:26:36,818][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [reindex]
[2022-12-02T13:26:36,819][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [repository-url]
[2022-12-02T13:26:36,819][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [systemd]
[2022-12-02T13:26:36,819][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded module [transport-netty4]
[2022-12-02T13:26:36,820][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-alerting]
[2022-12-02T13:26:36,820][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-anomaly-detection]
[2022-12-02T13:26:36,821][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-asynchronous-search]
[2022-12-02T13:26:36,821][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-cross-cluster-replication]
[2022-12-02T13:26:36,821][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-geospatial]
[2022-12-02T13:26:36,821][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-index-management]
[2022-12-02T13:26:36,822][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-job-scheduler]
[2022-12-02T13:26:36,822][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-knn]
[2022-12-02T13:26:36,822][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-ml]
[2022-12-02T13:26:36,823][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-notifications]
[2022-12-02T13:26:36,823][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-notifications-core]
[2022-12-02T13:26:36,823][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-observability]
[2022-12-02T13:26:36,823][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-performance-analyzer]
[2022-12-02T13:26:36,824][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-reports-scheduler]
[2022-12-02T13:26:36,824][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-security]
[2022-12-02T13:26:36,824][INFO ][o.o.p.PluginsService ] [cluster-manager-node] loaded plugin [opensearch-sql]
[2022-12-02T13:26:36,886][INFO ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting ‘http.compression: true’ in opensearch.yml
[2022-12-02T13:26:36,943][INFO ][o.o.e.NodeEnvironment ] [cluster-manager-node] using [1] data paths, mounts [[/ (/dev/mapper/centos-root)]], net usable_space [22.6gb], net total_space [143.4gb], types [xfs]
[2022-12-02T13:26:36,944][INFO ][o.o.e.NodeEnvironment ] [cluster-manager-node] heap size [1gb], compressed ordinary object pointers [true]
[2022-12-02T13:26:37,224][INFO ][o.o.n.Node ] [cluster-manager-node] node name [cluster-manager-node], node ID [7nRIqu1xT4uLuVk0e-5byA], cluster name [es-cluster], roles [cluster_manager]
[2022-12-02T13:26:45,617][WARN ][o.o.s.c.Salt ] [cluster-manager-node] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2022-12-02T13:26:45,734][INFO ][o.o.s.a.i.AuditLogImpl ] [cluster-manager-node] Message routing enabled: true
[2022-12-02T13:26:45,772][INFO ][o.o.s.f.SecurityFilter ] [cluster-manager-node] indices are made immutable.
[2022-12-02T13:26:46,657][INFO ][o.o.a.b.ADCircuitBreakerService] [cluster-manager-node] Registered memory breaker.
[2022-12-02T13:26:47,471][INFO ][o.o.m.c.b.MLCircuitBreakerService] [cluster-manager-node] Registered ML memory breaker.
[2022-12-02T13:26:48,874][INFO ][o.o.t.NettyAllocator ] [cluster-manager-node] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2022-12-02T13:26:49,136][INFO ][o.o.d.DiscoveryModule ] [cluster-manager-node] using discovery type [zen] and seed hosts providers [settings]
[2022-12-02T13:26:50,511][WARN ][o.o.g.DanglingIndicesState] [cluster-manager-node] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-12-02T13:26:51,581][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [cluster-manager-node] PerformanceAnalyzer Enabled: false
[2022-12-02T13:26:51,626][INFO ][o.o.n.Node ] [cluster-manager-node] initialized
[2022-12-02T13:26:51,626][INFO ][o.o.n.Node ] [cluster-manager-node] starting …
[2022-12-02T13:26:51,834][INFO ][o.o.t.TransportService ] [cluster-manager-node] publish_address {XX.XX.XX.100:9300}, bound_addresses {XX.XX.XX.100:9300}
[2022-12-02T13:26:52,286][INFO ][o.o.b.BootstrapChecks ] [cluster-manager-node] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-12-02T13:26:52,304][INFO ][o.o.c.c.Coordinator ] [cluster-manager-node] cluster UUID [r2Ho1zFbQVG5Tw7b1D9Xow]
[2022-12-02T13:26:52,522][INFO ][o.o.c.s.MasterService ] [cluster-manager-node] elected-as-cluster-manager ([1] nodes joined)[{cluster-manager-node}{7nRIqu1xT4uLuVk0e-5byA}{Y5gDvq8xQSe5FjWV5PnfCw}{XX.XX.XX.100}{XX.XX.XX.100:9300}{m}{shard_indexing_pressure_enabled=true} elect leader, BECOME_CLUSTER_MANAGER_TASK, FINISH_ELECTION], term: 20, version: 68, delta: cluster-manager node changed {previous , current [{cluster-manager-node}{7nRIqu1xT4uLuVk0e-5byA}{Y5gDvq8xQSe5FjWV5PnfCw}{XX.XX.XX.100}{XX.XX.XX.100:9300}{m}{shard_indexing_pressure_enabled=true}]}
[2022-12-02T13:26:52,617][INFO ][o.o.c.s.ClusterApplierService] [cluster-manager-node] cluster-manager node changed {previous , current [{cluster-manager-node}{7nRIqu1xT4uLuVk0e-5byA}{Y5gDvq8xQSe5FjWV5PnfCw}{XX.XX.XX.100}{XX.XX.XX.100:9300}{m}{shard_indexing_pressure_enabled=true}]}, term: 20, version: 68, reason: Publication{term=20, version=68}
[2022-12-02T13:26:52,640][INFO ][o.o.a.c.ADClusterEventListener] [cluster-manager-node] Cluster is not recovered yet.
[2022-12-02T13:26:52,647][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [cluster-manager-node] Detected cluster change event for destination migration
[2022-12-02T13:26:52,687][INFO ][o.o.i.i.ManagedIndexCoordinator] [cluster-manager-node] Cache cluster manager node onClusterManager time: 1669967812687
[2022-12-02T13:26:52,692][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [cluster-manager-node] Config override setting update called with empty string. Ignoring.
[2022-12-02T13:26:52,722][INFO ][o.o.h.AbstractHttpServerTransport] [cluster-manager-node] publish_address {XX.XX.XX.100:9200}, bound_addresses {XX.XX.XX.100:9200}
[2022-12-02T13:26:52,723][INFO ][o.o.n.Node ] [cluster-manager-node] started
[2022-12-02T13:26:52,723][INFO ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] Node started
[2022-12-02T13:26:52,724][INFO ][o.o.s.c.ConfigurationRepository] [cluster-manager-node] Will attempt to create index .opendistro_security and default configs if they are absent
[2022-12-02T13:26:52,762][INFO ][o.o.s.OpenSearchSecurityPlugin] [cluster-manager-node] 0 OpenSearch Security modules loaded so far: 
[2022-12-02T13:26:52,817][INFO ][o.o.s.c.ConfigurationRepository] [cluster-manager-node] Background init thread started. Install default config?: true
[2022-12-02T13:26:52,874][INFO ][o.o.c.s.ClusterSettings ] [cluster-manager-node] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
[2022-12-02T13:26:52,883][INFO ][o.o.a.c.HashRing ] [cluster-manager-node] Node added: [7nRIqu1xT4uLuVk0e-5byA]
[2022-12-02T13:26:52,886][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [cluster-manager-node] Detected cluster change event for destination migration
[2022-12-02T13:26:52,891][INFO ][o.o.a.c.HashRing ] [cluster-manager-node] All nodes with known AD version: {7nRIqu1xT4uLuVk0e-5byA=ADNodeInfo{version=2.3.0, isEligibleDataNode=false}}
[2022-12-02T13:26:52,892][INFO ][o.o.a.c.HashRing ] [cluster-manager-node] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
[2022-12-02T13:26:52,892][INFO ][o.o.a.c.HashRing ] [cluster-manager-node] Build AD version hash ring successfully
[2022-12-02T13:26:52,893][INFO ][o.o.a.c.ADClusterEventListener] [cluster-manager-node] Init AD version hash ring successfully
[2022-12-02T13:26:52,935][INFO ][o.o.g.GatewayService ] [cluster-manager-node] recovered [3] indices into cluster_state
[2022-12-02T13:26:52,939][INFO ][o.o.s.c.ConfigurationRepository] [cluster-manager-node] Index .opendistro_security already exists
[2022-12-02T13:26:52,939][INFO ][o.o.s.c.ConfigurationRepository] [cluster-manager-node] Node started, try to initialize it. Wait for at least yellow cluster state…
[2022-12-02T13:27:52,693][INFO ][o.o.i.i.ManagedIndexCoordinator] [cluster-manager-node] Performing move cluster state metadata.
[2022-12-02T13:27:52,697][INFO ][o.o.i.i.MetadataService ] [cluster-manager-node] ISM config index not exist, so we cancel the metadata migration job.
[2022-12-02T13:28:25,620][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [cluster-manager-node] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?]
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.79.Final.jar:4.1.79.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.79.Final.jar:4.1.79.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-12-02T13:28:25,636][WARN ][o.o.t.TcpTransport ] [cluster-manager-node] exception caught on transport layer [Netty4TcpChannel{localAddress=/XX.XX.XX.100:9300, remoteAddress=/XX.XX.XX.60:32854}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.79.Final.jar:4.1.79.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.79.Final.jar:4.1.79.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.79.Final.jar:4.1.79.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?]
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
… 16 more
[2022-12-02T13:28:26,213][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [cluster-manager-node] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

Once that was completed I noticed this error

javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

I’m assuming you created you own certificates? I would assume the certificates created may not be created correct OR not in the Keystore?
I dug up my old person documents on this, I figure I share.

# Root CA
openssl genrsa -out root-ca-key.pem 2048
openssl req -new -x509 -sha256 -key root-ca-key.pem -out root-ca.pem
# Admin cert
openssl genrsa -out admin-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in admin-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out admin-key.pem
openssl req -new -key admin-key.pem -out admin.csr
openssl x509 -req -in admin.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out admin.pem
# Node cert
openssl genrsa -out node-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in node-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out node-key.pem
openssl req -new -key node-key.pem -out node.csr
openssl x509 -req -in node.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out node.pem
# Cleanup
rm admin-key-temp.pem
rm admin.csr
rm node-key-temp.pem
rm node.csr

Once those were completed I edited my YAML file to correspond and place the full path to each one of certificates.

Then I Executed /securityadmin.sh found here

Hope that helps

bimlesh_singh · December 6, 2022, 5:20am

Hi,
Thanks for input!
We have kept all certificates in config folder so didn’t use absolute path.
As per Elasticsearch Doc, if we place certificates in config folder then no need to use absolute path. Is it something needed?
We have imported ca cert and node cert both but opensearch still complains certificate_unknown issue.
Which certificates need to be updated in cacerts file?
Also, I will try again with your approach (steps) and update the same

Gsmitt · December 6, 2022, 5:27am

TBH it doesn’t hurt for troubleshoot this issue , also check permissions.

Normally its your CA.

You may want to look at this documentation here

EDIT: I forgot to mention, It has been a while since I did this, I was using Open-Distro for elasticsearch but looking back over OpenSearch documentation it does look very similar perhaps some name differences

bimlesh_singh · December 6, 2022, 6:00am

Thanks for input!
I was trying to connect with Node with java client so the cacerts file of client’s JRE must have some imported certificate so which one should be imported --ca cert or node cert?

pablo · December 6, 2022, 10:08am

@bimlesh_singh How did you create your certificates? Was it the same as @Gsmitt described?

You’ve shared the logs of the cluster-manager-node node. Would you mind sharing logs of the data-node?

According to the logs, the error appears when the manager node tries to connect with the data node over the Transport layer.

Gsmitt · December 6, 2022, 10:33pm

Hey,

After looking over the Documentation, I think you could try –ca cert ,remember when using the Default JAVA keystore make sure you make a backup. I haven’t enable TCP/TLS yet on Opensearch but it looks like the ./securityadmin.sh does this for you, Not 100% sure just looking over the documentation.

EDIT: @bimlesh_singh @pablo For example this was what I use back in the day, I think OpenSearch has the same Doc’s I have check yet.

bimlesh_singh · December 7, 2022, 9:32am

Hi Pablo/Greg,
Thanks for your inputs!
It looks like opensearch does not validate anything from certificates but Registered ID value in Subject Alternative Name (SAN). if this is present then nodes are able to communicate without certificate error. So, solution is added Registered Id in SAN of node certificate while generating certificates through OpenSSL and node setup looks good.

Gsmitt · December 10, 2022, 3:42am

I see now, my apologies like i stated it been a while.
While back I did something like that with Graylog.

Created a file named openssl-graylog.cnf with the following content for my certificates like so…

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
# Details about the issuer of the certificate
[req_distinguished_name]
C = US
ST = iowa
L = cedar rapids
O = texaco
OU = admin
CN = graylog.domain.com
[v3_req]
keyUsage = keyEncipherment, dataEncipherment,nonRepudiation
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
# IP addresses and DNS names the certificate should include
# Use IP.### for IP addresses and DNS.### for DNS names,
# with "###" being a consecutive number.
[alt_names]
IP.1 = 192.168.1.100
DNS.1 = graylog.domain.com

Then used that file in a command.

penssl req -x509 -days 1095 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem

I did notice others are using Certbot now a days or mkcert

Topic		Replies	Views
Unable to configure newly generated SSL certificates on a 3 node opensearch cluster Security configure	3	521	April 20, 2022
How to configure the opensearch cluster with self signed certificates step by step OpenSearch configure	5	573	December 19, 2023
TLS error on multi-node cluster OpenSearch	5	504	December 15, 2023
Installing tls ssl certificates Security troubleshoot , configure	3	434	June 27, 2023
3 nodes opensearch cluster, cannot start with only 2 nodes OpenSearch	20	363	March 28, 2024

Opensearch-2.3.0 multinode SSL setup issue

opensearch.yml[cluster-manager - node]

opensearch.yml[data - node]

Related Topics