OpenSearch Cluster Security not initialized

Security
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch:2.7.0
ubuntu 22.04.1

Describe the issue:
I installed opnesearch on 3 server.
Each node has a master, data rule.
Each node ran and watched logs
error 1.

Attaching to os-manager-node2
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
OpenSearch Security Demo Installer
** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-2.7.0
Detected OpenSearch Security Version: 2.7.0.0
/usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
Enabling OpenSearch Security Plugin
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.7.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.7.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2023-06-05T10:10:00,939][INFO ][o.o.n.Node               ] [os-manager-node2] version[2.7.0], pid[34], build[tar/b7a6e09e492b1e965d827525f7863b366ef0e304/2023-04-27T21:43:09.523336706Z], OS[Linux/5.15.0-43-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.6/17.0.6+10]
[2023-06-05T10:10:00,941][INFO ][o.o.n.Node               ] [os-manager-node2] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
[2023-06-05T10:10:00,941][INFO ][o.o.n.Node               ] [os-manager-node2] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-4472680479187956406, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2023-06-05T10:10:04,326][INFO ][o.o.s.s.t.SSLConfig      ] [os-manager-node2] SSL dual mode is disabled
[2023-06-05T10:10:04,326][INFO ][o.o.s.OpenSearchSecurityPlugin] [os-manager-node2] OpenSearch Config path is /usr/share/opensearch/config
[2023-06-05T10:10:04,622][INFO ][o.o.s.s.DefaultSecurityKeyStore] [os-manager-node2] JVM supports TLSv1.3
[2023-06-05T10:10:04,623][INFO ][o.o.s.s.DefaultSecurityKeyStore] [os-manager-node2] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2023-06-05T10:10:05,087][INFO ][o.o.s.s.DefaultSecurityKeyStore] [os-manager-node2] TLS Transport Client Provider : JDK
[2023-06-05T10:10:05,087][INFO ][o.o.s.s.DefaultSecurityKeyStore] [os-manager-node2] TLS Transport Server Provider : JDK
[2023-06-05T10:10:05,087][INFO ][o.o.s.s.DefaultSecurityKeyStore] [os-manager-node2] TLS HTTP Provider             : JDK
[2023-06-05T10:10:05,087][INFO ][o.o.s.s.DefaultSecurityKeyStore] [os-manager-node2] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2023-06-05T10:10:05,087][INFO ][o.o.s.s.DefaultSecurityKeyStore] [os-manager-node2] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2]
[2023-06-05T10:10:05,096][INFO ][o.o.s.OpenSearchSecurityPlugin] [os-manager-node2] Clustername: ***
[2023-06-05T10:10:05,103][WARN ][o.o.s.OpenSearchSecurityPlugin] [os-manager-node2] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[2023-06-05T10:10:05,103][WARN ][o.o.s.OpenSearchSecurityPlugin] [os-manager-node2] File /usr/share/opensearch/config/opensearch-security/config.yml has insecure file permissions (should be 0600)
[2023-06-05T10:10:05,103][WARN ][o.o.s.OpenSearchSecurityPlugin] [os-manager-node2] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
[2023-06-05T10:10:05,103][WARN ][o.o.s.OpenSearchSecurityPlugin] [os-manager-node2] File /usr/share/opensearch/config/admin.pem has insecure file permissions (should be 0600)
[2023-06-05T10:10:05,103][WARN ][o.o.s.OpenSearchSecurityPlugin] [os-manager-node2] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600)
[2023-06-05T10:10:05,103][WARN ][o.o.s.OpenSearchSecurityPlugin] [os-manager-node2] File /usr/share/opensearch/config/os-manager-node2.pem has insecure file permissions (should be 0600)
[2023-06-05T10:10:05,103][WARN ][o.o.s.OpenSearchSecurityPlugin] [os-manager-node2] File /usr/share/opensearch/config/os-manager-node2-key.pem has insecure file permissions (should be 0600)
[2023-06-05T10:10:05,104][WARN ][o.o.s.OpenSearchSecurityPlugin] [os-manager-node2] File /usr/share/opensearch/config/admin-key.pem has insecure file permissions (should be 0600)
[2023-06-05T10:10:08,316][INFO ][o.o.p.c.PluginSettings   ] [os-manager-node2] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2023-06-05T10:10:08,896][INFO ][o.o.i.r.ReindexPlugin    ] [os-manager-node2] ReindexPlugin reloadSPI called
[2023-06-05T10:10:08,897][INFO ][o.o.i.r.ReindexPlugin    ] [os-manager-node2] Unable to find any implementation for RemoteReindexExtension
[2023-06-05T10:10:08,924][INFO ][o.o.j.JobSchedulerPlugin ] [os-manager-node2] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2023-06-05T10:10:08,958][INFO ][o.o.j.JobSchedulerPlugin ] [os-manager-node2] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2023-06-05T10:10:08,959][INFO ][o.o.j.JobSchedulerPlugin ] [os-manager-node2] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2023-06-05T10:10:08,979][INFO ][o.o.j.JobSchedulerPlugin ] [os-manager-node2] Loaded scheduler extension: observability, index: .opensearch-observability-job
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [aggs-matrix-stats]
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [analysis-common]
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [geo]
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [ingest-common]
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [ingest-geoip]
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [ingest-user-agent]
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [lang-expression]
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [lang-mustache]
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [lang-painless]
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [mapper-extras]
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [opensearch-dashboards]
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [parent-join]
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [percolator]
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [rank-eval]
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [reindex]
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [repository-url]
[2023-06-05T10:10:08,984][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [search-pipeline-common]
[2023-06-05T10:10:08,985][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [systemd]
[2023-06-05T10:10:08,985][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded module [transport-netty4]
[2023-06-05T10:10:08,985][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-alerting]
[2023-06-05T10:10:08,985][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-anomaly-detection]
[2023-06-05T10:10:08,985][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-asynchronous-search]
[2023-06-05T10:10:08,985][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-cross-cluster-replication]
[2023-06-05T10:10:08,985][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-geospatial]
[2023-06-05T10:10:08,985][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-index-management]
[2023-06-05T10:10:08,985][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-job-scheduler]
[2023-06-05T10:10:08,985][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-knn]
[2023-06-05T10:10:08,985][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-ml]
[2023-06-05T10:10:08,985][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-neural-search]
[2023-06-05T10:10:08,985][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-notifications]
[2023-06-05T10:10:08,986][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-notifications-core]
[2023-06-05T10:10:08,986][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-observability]
[2023-06-05T10:10:08,986][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-performance-analyzer]
[2023-06-05T10:10:08,986][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-reports-scheduler]
[2023-06-05T10:10:08,986][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-security]
[2023-06-05T10:10:08,986][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-security-analytics]
[2023-06-05T10:10:08,986][INFO ][o.o.p.PluginsService     ] [os-manager-node2] loaded plugin [opensearch-sql]
[2023-06-05T10:10:09,021][INFO ][o.o.s.OpenSearchSecurityPlugin] [os-manager-node2] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2023-06-05T10:10:09,059][INFO ][o.o.e.NodeEnvironment    ] [os-manager-node2] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/sdc2)]], net usable_space [174.7gb], net total_space [199.8gb], types [xfs]
[2023-06-05T10:10:09,059][INFO ][o.o.e.NodeEnvironment    ] [os-manager-node2] heap size [512mb], compressed ordinary object pointers [true]
[2023-06-05T10:10:09,269][INFO ][o.o.n.Node               ] [os-manager-node2] node name [os-manager-node2], node ID [YXNrKYx6Q2mGh1AKx9aw8g], cluster name [***], roles ["data"], ["master"]
[2023-06-05T10:10:12,305][WARN ][o.o.s.c.Salt             ] [os-manager-node2] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2023-06-05T10:10:12,372][INFO ][o.o.s.a.i.AuditLogImpl   ] [os-manager-node2] Message routing enabled: true
[2023-06-05T10:10:12,393][INFO ][o.o.s.f.SecurityFilter   ] [os-manager-node2] <NONE> indices are made immutable.
[2023-06-05T10:10:12,713][INFO ][o.o.a.b.ADCircuitBreakerService] [os-manager-node2] Registered memory breaker.
[2023-06-05T10:10:13,147][INFO ][o.o.m.b.MLCircuitBreakerService] [os-manager-node2] Registered ML memory breaker.
[2023-06-05T10:10:13,147][INFO ][o.o.m.b.MLCircuitBreakerService] [os-manager-node2] Registered ML disk breaker.
[2023-06-05T10:10:13,148][INFO ][o.o.m.b.MLCircuitBreakerService] [os-manager-node2] Registered ML native memory breaker.
[2023-06-05T10:10:13,241][INFO ][o.r.Reflections          ] [os-manager-node2] Reflections took 36 ms to scan 1 urls, producing 15 keys and 37 values 
[2023-06-05T10:10:14,045][INFO ][o.o.t.NettyAllocator     ] [os-manager-node2] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
[2023-06-05T10:10:14,137][INFO ][o.o.d.DiscoveryModule    ] [os-manager-node2] using discovery type [zen] and seed hosts providers [settings]
[2023-06-05T10:10:14,613][WARN ][o.o.g.DanglingIndicesState] [os-manager-node2] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-06-05T10:10:14,747][WARN ][o.o.t.TransportService   ] [os-manager-node2] invalid action name [cluster:feature/tenancy/config/read] must start with one of: [cluster:admin, indices:data/read, indices:monitor, indices:data/write, internal:, indices:internal, cluster:monitor, cluster:internal, indices:admin]
[2023-06-05T10:10:14,755][WARN ][o.o.t.TransportService   ] [os-manager-node2] invalid action name [cluster:feature/tenancy/config/update] must start with one of: [cluster:admin, indices:data/read, indices:monitor, indices:data/write, internal:, indices:internal, cluster:monitor, cluster:internal, indices:admin]
[2023-06-05T10:10:15,136][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [os-manager-node2] PerformanceAnalyzer Enabled: false
[2023-06-05T10:10:15,243][INFO ][o.o.n.Node               ] [os-manager-node2] initialized
[2023-06-05T10:10:15,244][INFO ][o.o.n.Node               ] [os-manager-node2] starting ...
[2023-06-05T10:10:15,393][INFO ][o.o.t.TransportService   ] [os-manager-node2] publish_address {172.21.31.156:9300}, bound_addresses {0.0.0.0:9300}
[2023-06-05T10:10:15,527][INFO ][o.o.b.BootstrapChecks    ] [os-manager-node2] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-06-05T10:10:45,544][WARN ][o.o.n.Node               ] [os-manager-node2] timed out while waiting for initial discovery state - timeout: 30s
[2023-06-05T10:10:45,562][INFO ][o.o.h.AbstractHttpServerTransport] [os-manager-node2] publish_address {172.21.31.156:9200}, bound_addresses {0.0.0.0:9200}
[2023-06-05T10:10:45,562][INFO ][o.o.n.Node               ] [os-manager-node2] started
[2023-06-05T10:10:45,563][INFO ][o.o.s.OpenSearchSecurityPlugin] [os-manager-node2] Node started
[2023-06-05T10:10:45,563][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Will attempt to create index .opendistro_security and default configs if they are absent
[2023-06-05T10:10:45,563][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Background init thread started. Install default config?: true
[2023-06-05T10:10:45,564][INFO ][o.o.s.OpenSearchSecurityPlugin] [os-manager-node2] 0 OpenSearch Security modules loaded so far: []
[2023-06-05T10:10:45,564][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:10:46,564][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:10:47,564][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:10:48,565][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:10:49,565][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:10:50,565][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:10:51,566][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:10:52,566][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:10:53,566][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:10:54,567][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:10:55,567][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:10:56,567][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:10:57,568][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:10:58,568][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:10:59,569][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:00,569][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:01,569][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:02,570][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:03,570][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:04,571][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:05,571][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:06,571][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:07,572][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:08,572][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:09,573][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:10,573][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:11,573][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:12,574][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:13,574][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:14,574][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:15,575][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:15,607][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [os-manager-node2] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: ClusterManagerNotDiscoveredException[null]
    at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.7.0.jar:2.7.0]
    at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.7.0.jar:2.7.0]
    at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.7.0.jar:2.7.0]
    at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.7.0.jar:2.7.0]
    at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.7.0.jar:2.7.0]
    at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.7.0.jar:2.7.0]
    at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.7.0.jar:2.7.0]
Caused by: org.opensearch.discovery.ClusterManagerNotDiscoveredException
    at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAction.java:350) ~[opensearch-2.7.0.jar:2.7.0]
    at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) ~[opensearch-2.7.0.jar:2.7.0]
uncaught exception in thread [main]
    at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) ~[opensearch-2.7.0.jar:2.7.0]
    at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707) ~[opensearch-2.7.0.jar:2.7.0]
    at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.7.0.jar:2.7.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
    at java.lang.Thread.run(Thread.java:833) [?:?]
ClusterManagerNotDiscoveredException[null]
    at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAction.java:350)
    at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394)
    at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294)
    at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707)
    at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
For complete error details, refer to the log at /usr/share/opensearch/logs/***.log
[2023-06-05T10:11:16,575][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:17,575][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:18,576][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:19,576][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:20,577][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:21,577][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:22,577][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:23,578][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:24,578][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:25,579][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:26,579][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:27,579][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:28,580][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:29,587][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:30,587][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:31,588][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:32,588][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:33,588][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:34,589][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:35,589][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:36,590][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:37,590][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:38,591][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:39,592][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:40,592][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:41,593][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:42,593][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:43,593][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:44,594][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:45,594][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:46,594][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:47,595][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:48,595][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:49,596][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:50,596][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:51,596][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:52,597][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:53,597][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:54,598][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:55,598][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:56,599][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:57,599][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:58,599][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:11:59,600][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:12:00,600][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:12:01,600][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:12:02,601][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:12:03,601][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:12:04,601][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:12:05,602][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:12:06,602][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:12:07,603][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:12:08,603][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:12:09,603][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:12:10,604][INFO ][o.o.s.c.ConfigurationRepository] [os-manager-node2] Wait for cluster to be available ...
[2023-06-05T10:12:11

error 2.
I just keep connector with “curl”
when I connect to “curl -k https://localhost:9200”
answer-> Opensearch Security not initalized
so, I refer this page OpenSearch Security not initialized - Security - OpenSearch
but reponse is…

Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-5 [ACTIVE]. This is not an error, will keep on trying ...
  Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-5 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

How can i run a cluster correctly?

2

Hi @cucukaka
Could you please send your opensearch.yml file?

3

of course!

docker-compose.yml in os-manager-node1 server(there are 3 servers)

version: '3'
services:
  os-manager-node1:
    image: opensearchproject/opensearch:2.7.0
    container_name: ${CONTAINER_NAME}
    environment:
      - cluster.name=${CLUSTERNAME}
      - node.name=${CONTAINER_NAME}
      - node.roles=["cluster_manager", "data"] # every nodes follow this role setting
      - network.publish_host=${node1serverip}
      - network.host=0.0.0.0
      - discovery.seed_hosts=${node1serverip}:9300,${node2serverip}:9300,${node3serverip}:9300
      - cluster.initial_cluster_manager_nodes=os-manager-node1,os-manager-node2,os-manager-node3
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    volumes:
      - '/data/sas/opensearch/data:/usr/share/opensearch/data'
      - '/data/sas/opensearch/logs:/usr/share/opensearch/logs'
      - './config/root-ca.pem:/usr/share/opensearch/config/root-ca.pem'
      - './config/admin.pem:/usr/share/opensearch/config/admin.pem'
      - './config/admin-key.pem:/usr/share/opensearch/config/admin-key.pem'
      - './config/${NODE_PEM_CERT_FILE}:/usr/share/opensearch/config/${NODE_PEM_CERT_FILE}'
      - './config/${NODE_PEM_KEY_FILE}:/usr/share/opensearch/config/${NODE_PEM_KEY_FILE}'
      - './config/custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml'
      - './config/opensearch-security/config.yml:/usr/share/opensearch/config/opensearch-security/config.yml'
    ports:
      - 9200:9200
      - 9300:9300
      - 5107:9200
      - 9600:9600 # Performance Analyzer

custer-opensearch.yml

plugins.security.ssl.transport.pemcert_filepath: os-manager-node1.pem
plugins.security.ssl.transport.pemkey_filepath: os-manager-node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: os-manager-node1.pem
plugins.security.ssl.http.pemkey_filepath: os-manager-node1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=A,OU=${team},O=${company},L=${location},ST=${state},C=KR

plugins.security.nodes_dn:
  - 'CN=os-manager-node1,OU=${team},O=${company},L=${location},ST=${state},C=KR'
  - 'CN=os-manager-node2,OU=${team},O=${company},L=${location},ST=${state},C=KR'
  - 'CN=os-manager-node3,OU=${team},O=${company},L=${location},ST=${state},C=KR'
plugins.security.audit.type: internal_opensearch

plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
opendistro_security.audit.config.disabled_rest_categories: NONE
opendistro_security.audit.config.disabled_transport_categories: NONE

and also each server connected with each other
image
image

and also I try to connect with this script

network.host: <IP address of the node>

but it can’t connect…

4

filnally I solved this error!
network.host → network.bind_host is correct!
Also, when I create pem files, we carefully write “CN” and “subjectAltName”
I wrote all the values ​​the same like this

openssl req -new -key os-manager-node1-key.pem -subj "/C=KR/ST=**/L=**/O=**/OU=**/CN=os-manager-node1" -out os-manager-node1.csr
echo 'subjectAltName=DNS:os-manager-node1' > os-manager-node1.ext
1 Like
5

sorry, Do you can use perform analyzer with these configs? and how to check this?

6

@pdcuong27 Yes, you can. The above config exposes port 9600 which provides access to the Performance Analyzer.

Please follow the OpenSearch Documentation.

7

oh, i would check. But how can use performance analyzer with opensearch operator?

8

@pdcuong27 Could you open a new thread where we could continue this discussion?

1 Like