Versions
OpenSearch - 2.15.0
Describe the issue:
I created an opensearch cluster on different physical servers. Opensearch works in Docker. And when I run the curl -XGET "http://localhost:9200/_cluster/health" command I get an error
{“error”:{“root_cause”:[{“type”:“cluster_manager_not_discovered_exception”,“reason”:null}],“type”:“cluster_manager_not_discovered_exception”,“reason”:null},“status”:503}
my docker-compose.yml
version: '3'
services:
opensearch-node1:
image: opensearch:2.15.0
container_name: opensearch-node01
environment:
- cluster.name=opensearch-cluster # Name the cluster
- node.name=node01 # Name the node that will run in this container
- discovery.seed_hosts=node01,node02,node03 # Nodes to look for when discovering the cluster
- cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2,opensearch-node3 # Nodes eligibile to serve as cluster manager
- bootstrap.memory_lock=true # Disable JVM heap memory swapping
- "OPENSEARCH_JAVA_OPTS=-Xms8192m -Xmx8192m" # Set min and max JVM heap sizes to at least 50% of system RAM
- "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
- "DISABLE_SECURITY_PLUGIN=true" # Disables Security plugin
ulimits:
memlock:
soft: -1 # Set memlock to unlimited (no soft or hard limit)
hard: -1
nofile:
soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
hard: 65536
volumes:
- /data/opensearch:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
# expose:
# - 9200
# - 9600
ports:
- 9200:9200 # REST API
- 9600:9600 # Performance Analyzer
# networks:
# - opensearch-net # All of the containers will join the same Docker bridge network
network_mode: host
opensearch-dashboards:
image: opensearch-dashboards:2.15.0
container_name: opensearch-dashboards
ports:
- 5601:5601 # Map host port 5601 to container port 5601
expose:
- "5601" # Expose port 5601 for web access to OpenSearch Dashboards
environment:
- 'OPENSEARCH_HOSTS=["http://node01:9200","http://node02:9200","http://node03:9200"]'
- "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
# networks:
# - opensearch-net
network_mode: host
#volumes:
# opensearch-data1:
# opensearch-data2:
#networks:
# opensearch-net:
Configuration:
Docker
Docker-compose
Opensearch
Relevant Logs or Screenshots:
docker logs "container id"
Disabling OpenSearch Security Plugin
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.15.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
Jul 10, 2024 10:47:25 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.15.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2024-07-10T10:47:27,051][INFO ][o.o.n.Node ] [node01] version[2.15.0], pid[1], build[tar/61dbcd0795c9bfe9b81e5762175414bc38bbcadf/2024-06-20T03:26:49.193630411Z], OS[Linux/5.15.0-204.147.6.2.el8uek.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.3/21.0.3+9-LTS]
[2024-07-10T10:47:27,055][INFO ][o.o.n.Node ] [node01] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2024-07-10T10:47:27,055][INFO ][o.o.n.Node ] [node01] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-9282092181321069659, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xms8192m, -Xmx8192m, -XX:MaxDirectMemorySize=4294967296, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2024-07-10T10:47:29,032][INFO ][o.o.s.s.t.SSLConfig ] [node01] SSL dual mode is disabled
[2024-07-10T10:47:29,033][WARN ][o.o.s.OpenSearchSecurityPlugin] [node01] OpenSearch Security plugin installed but disabled. This can expose your configuration (including passwords) to the public.
[2024-07-10T10:47:29,846][INFO ][o.o.p.c.c.PluginSettings ] [node01] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2024-07-10T10:47:30,565][INFO ][o.o.i.r.ReindexPlugin ] [node01] ReindexPlugin reloadSPI called
[2024-07-10T10:47:30,566][INFO ][o.o.i.r.ReindexPlugin ] [node01] Unable to find any implementation for RemoteReindexExtension
[2024-07-10T10:47:30,627][INFO ][o.o.j.JobSchedulerPlugin ] [node01] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2024-07-10T10:47:30,628][INFO ][o.o.j.JobSchedulerPlugin ] [node01] Loaded scheduler extension: opensearch_time_series_analytics, index: .opendistro-anomaly-detector-jobs
[2024-07-10T10:47:30,629][INFO ][o.o.j.JobSchedulerPlugin ] [node01] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2024-07-10T10:47:30,630][INFO ][o.o.j.JobSchedulerPlugin ] [node01] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
[2024-07-10T10:47:30,632][INFO ][o.o.j.JobSchedulerPlugin ] [node01] Loaded scheduler extension: opensearch_sap_job, index: .opensearch-sap--job
[2024-07-10T10:47:30,650][INFO ][o.o.p.PluginsService ] [node01] loaded module [aggs-matrix-stats]
[2024-07-10T10:47:30,650][INFO ][o.o.p.PluginsService ] [node01] loaded module [analysis-common]
[2024-07-10T10:47:30,650][INFO ][o.o.p.PluginsService ] [node01] loaded module [cache-common]
[2024-07-10T10:47:30,650][INFO ][o.o.p.PluginsService ] [node01] loaded module [geo]
[2024-07-10T10:47:30,650][INFO ][o.o.p.PluginsService ] [node01] loaded module [ingest-common]
[2024-07-10T10:47:30,651][INFO ][o.o.p.PluginsService ] [node01] loaded module [ingest-geoip]
[2024-07-10T10:47:30,651][INFO ][o.o.p.PluginsService ] [node01] loaded module [ingest-user-agent]
[2024-07-10T10:47:30,651][INFO ][o.o.p.PluginsService ] [node01] loaded module [lang-expression]
[2024-07-10T10:47:30,651][INFO ][o.o.p.PluginsService ] [node01] loaded module [lang-mustache]
[2024-07-10T10:47:30,651][INFO ][o.o.p.PluginsService ] [node01] loaded module [lang-painless]
[2024-07-10T10:47:30,651][INFO ][o.o.p.PluginsService ] [node01] loaded module [mapper-extras]
[2024-07-10T10:47:30,651][INFO ][o.o.p.PluginsService ] [node01] loaded module [opensearch-dashboards]
[2024-07-10T10:47:30,651][INFO ][o.o.p.PluginsService ] [node01] loaded module [parent-join]
[2024-07-10T10:47:30,651][INFO ][o.o.p.PluginsService ] [node01] loaded module [percolator]
[2024-07-10T10:47:30,651][INFO ][o.o.p.PluginsService ] [node01] loaded module [rank-eval]
[2024-07-10T10:47:30,652][INFO ][o.o.p.PluginsService ] [node01] loaded module [reindex]
[2024-07-10T10:47:30,652][INFO ][o.o.p.PluginsService ] [node01] loaded module [repository-url]
[2024-07-10T10:47:30,652][INFO ][o.o.p.PluginsService ] [node01] loaded module [search-pipeline-common]
[2024-07-10T10:47:30,652][INFO ][o.o.p.PluginsService ] [node01] loaded module [systemd]
[2024-07-10T10:47:30,652][INFO ][o.o.p.PluginsService ] [node01] loaded module [transport-netty4]
[2024-07-10T10:47:30,652][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-alerting]
[2024-07-10T10:47:30,653][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-anomaly-detection]
[2024-07-10T10:47:30,653][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-asynchronous-search]
[2024-07-10T10:47:30,653][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-cross-cluster-replication]
[2024-07-10T10:47:30,653][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-custom-codecs]
[2024-07-10T10:47:30,653][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-flow-framework]
[2024-07-10T10:47:30,653][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-geospatial]
[2024-07-10T10:47:30,653][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-index-management]
[2024-07-10T10:47:30,653][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-job-scheduler]
[2024-07-10T10:47:30,654][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-knn]
[2024-07-10T10:47:30,654][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-ml]
[2024-07-10T10:47:30,654][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-neural-search]
[2024-07-10T10:47:30,654][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-notifications]
[2024-07-10T10:47:30,654][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-notifications-core]
[2024-07-10T10:47:30,654][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-observability]
[2024-07-10T10:47:30,654][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-performance-analyzer]
[2024-07-10T10:47:30,654][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-reports-scheduler]
[2024-07-10T10:47:30,654][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-security]
[2024-07-10T10:47:30,654][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-security-analytics]
[2024-07-10T10:47:30,654][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-skills]
[2024-07-10T10:47:30,655][INFO ][o.o.p.PluginsService ] [node01] loaded plugin [opensearch-sql]
[2024-07-10T10:47:30,734][INFO ][o.o.e.ExtensionsManager ] [node01] ExtensionsManager initialized
[2024-07-10T10:47:30,748][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] [node01] Using MemorySegmentIndexInput with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
[2024-07-10T10:47:30,757][INFO ][o.o.e.NodeEnvironment ] [node01] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/mapper/ol-root)]], net usable_space [255.7gb], net total_space [292.1gb], types [ext4]
[2024-07-10T10:47:30,757][INFO ][o.o.e.NodeEnvironment ] [node01] heap size [8gb], compressed ordinary object pointers [true]
[2024-07-10T10:47:30,851][INFO ][o.o.n.Node ] [node01] node name [node01], node ID [zj4riFT1QV-czKxuUQsSRQ], cluster name [opensearch-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
[2024-07-10T10:47:33,188][DEPRECATION][o.o.d.c.s.Settings ] [node01] [index.store.hybrid.mmap.extensions] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2024-07-10T10:47:33,822][INFO ][o.o.n.p.NeuralSearch ] [node01] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled]
[2024-07-10T10:47:34,666][INFO ][o.o.t.b.CircuitBreakerService] [node01] Registered memory breaker.
[2024-07-10T10:47:35,167][INFO ][o.o.m.b.MLCircuitBreakerService] [node01] Registered ML memory breaker.
[2024-07-10T10:47:35,168][INFO ][o.o.m.b.MLCircuitBreakerService] [node01] Registered ML disk breaker.
[2024-07-10T10:47:35,168][INFO ][o.o.m.b.MLCircuitBreakerService] [node01] Registered ML native memory breaker.
[2024-07-10T10:47:35,288][INFO ][o.r.Reflections ] [node01] Reflections took 56 ms to scan 1 urls, producing 26 keys and 67 values
[2024-07-10T10:47:35,397][WARN ][o.o.s.p.SQLPlugin ] [node01] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
[2024-07-10T10:47:36,351][INFO ][o.o.t.NettyAllocator ] [node01] creating NettyAllocator with the following configs: [name=opensearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={opensearch.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2024-07-10T10:47:36,470][INFO ][o.o.d.DiscoveryModule ] [node01] using discovery type [zen] and seed hosts providers [settings]
[2024-07-10T10:47:37,152][WARN ][o.o.g.DanglingIndicesState] [node01] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-07-10T10:47:38,177][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [node01] PerformanceAnalyzer Enabled: false
[2024-07-10T10:47:38,223][INFO ][o.o.n.Node ] [node01] initialized
[2024-07-10T10:47:38,224][INFO ][o.o.n.Node ] [node01] starting ...
[2024-07-10T10:47:38,258][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [windows_logtype.json] log type
[2024-07-10T10:47:38,259][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [waf_logtype.json] log type
[2024-07-10T10:47:38,260][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [vpcflow_logtype.json] log type
[2024-07-10T10:47:38,261][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [test_windows_logtype.json] log type
[2024-07-10T10:47:38,262][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [s3_logtype.json] log type
[2024-07-10T10:47:38,262][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [others_web_logtype.json] log type
[2024-07-10T10:47:38,262][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [others_proxy_logtype.json] log type
[2024-07-10T10:47:38,263][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [others_macos_logtype.json] log type
[2024-07-10T10:47:38,263][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [others_compliance_logtype.json] log type
[2024-07-10T10:47:38,264][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [others_cloud_logtype.json] log type
[2024-07-10T10:47:38,264][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [others_apt_logtype.json] log type
[2024-07-10T10:47:38,265][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [others_application_logtype.json] log type
[2024-07-10T10:47:38,265][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [okta_logtype.json] log type
[2024-07-10T10:47:38,266][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [network_logtype.json] log type
[2024-07-10T10:47:38,266][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [netflow_logtype.json] log type
[2024-07-10T10:47:38,267][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [m365_logtype.json] log type
[2024-07-10T10:47:38,267][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [linux_logtype.json] log type
[2024-07-10T10:47:38,268][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [gworkspace_logtype.json] log type
[2024-07-10T10:47:38,268][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [github_logtype.json] log type
[2024-07-10T10:47:38,269][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [dns_logtype.json] log type
[2024-07-10T10:47:38,270][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [cloudtrail_logtype.json] log type
[2024-07-10T10:47:38,270][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [azure_logtype.json] log type
[2024-07-10T10:47:38,271][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [apache_access_logtype.json] log type
[2024-07-10T10:47:38,271][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node01] Loaded [ad_ldap_logtype.json] log type
[2024-07-10T10:47:38,399][INFO ][o.o.t.TransportService ] [node01] publish_address {192.168.0.66:9300}, bound_addresses {[::]:9300}
[2024-07-10T10:47:38,401][INFO ][o.o.t.TransportService ] [node01] Remote clusters initialized successfully.
[2024-07-10T10:47:38,568][INFO ][o.o.b.BootstrapChecks ] [node01] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2024-07-10T10:47:38,581][INFO ][o.o.c.c.ClusterBootstrapService] [node01] skipping cluster bootstrapping as local node does not match bootstrap requirements: [opensearch-node1, opensearch-node2, opensearch-node3]
[2024-07-10T10:47:48,585][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node01] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [opensearch-node1, opensearch-node2, opensearch-node3] to bootstrap a cluster: have discovered [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node02}{5RPTs2DHQIG1lJPHxRRBKA}{5y-UdC1YTlSD1C9CD2p71Q}{192.168.0.67}{192.168.0.67:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node03}{qGvT8QxISguFBRdFXrOu-Q}{71jE8PGGTG2Pp-U41WnyJw}{192.168.0.68}{192.168.0.68:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:9300, 192.168.0.67:9300, 192.168.0.68:9300] from hosts providers and [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2024-07-10T10:47:58,587][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node01] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [opensearch-node1, opensearch-node2, opensearch-node3] to bootstrap a cluster: have discovered [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node02}{5RPTs2DHQIG1lJPHxRRBKA}{5y-UdC1YTlSD1C9CD2p71Q}{192.168.0.67}{192.168.0.67:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node03}{qGvT8QxISguFBRdFXrOu-Q}{71jE8PGGTG2Pp-U41WnyJw}{192.168.0.68}{192.168.0.68:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:9300, 192.168.0.67:9300, 192.168.0.68:9300] from hosts providers and [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2024-07-10T10:48:08,587][WARN ][o.o.n.Node ] [node01] timed out while waiting for initial discovery state - timeout: 30s
[2024-07-10T10:48:08,589][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node01] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [opensearch-node1, opensearch-node2, opensearch-node3] to bootstrap a cluster: have discovered [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node02}{5RPTs2DHQIG1lJPHxRRBKA}{5y-UdC1YTlSD1C9CD2p71Q}{192.168.0.67}{192.168.0.67:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node03}{qGvT8QxISguFBRdFXrOu-Q}{71jE8PGGTG2Pp-U41WnyJw}{192.168.0.68}{192.168.0.68:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:9300, 192.168.0.67:9300, 192.168.0.68:9300] from hosts providers and [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2024-07-10T10:48:08,596][INFO ][o.o.h.AbstractHttpServerTransport] [node01] publish_address {192.168.0.66:9200}, bound_addresses {[::]:9200}
[2024-07-10T10:48:08,596][INFO ][o.o.n.Node ] [node01] started
[2024-07-10T10:48:08,597][INFO ][o.o.s.OpenSearchSecurityPlugin] [node01] 0 OpenSearch Security modules loaded so far: []
[2024-07-10T10:48:18,590][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node01] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [opensearch-node1, opensearch-node2, opensearch-node3] to bootstrap a cluster: have discovered [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node02}{5RPTs2DHQIG1lJPHxRRBKA}{5y-UdC1YTlSD1C9CD2p71Q}{192.168.0.67}{192.168.0.67:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node03}{qGvT8QxISguFBRdFXrOu-Q}{71jE8PGGTG2Pp-U41WnyJw}{192.168.0.68}{192.168.0.68:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:9300, 192.168.0.67:9300, 192.168.0.68:9300] from hosts providers and [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2024-07-10T10:48:28,592][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node01] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [opensearch-node1, opensearch-node2, opensearch-node3] to bootstrap a cluster: have discovered [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node02}{5RPTs2DHQIG1lJPHxRRBKA}{5y-UdC1YTlSD1C9CD2p71Q}{192.168.0.67}{192.168.0.67:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node03}{qGvT8QxISguFBRdFXrOu-Q}{71jE8PGGTG2Pp-U41WnyJw}{192.168.0.68}{192.168.0.68:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:9300, 192.168.0.67:9300, 192.168.0.68:9300] from hosts providers and [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2024-07-10T10:48:38,593][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node01] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [opensearch-node1, opensearch-node2, opensearch-node3] to bootstrap a cluster: have discovered [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node02}{5RPTs2DHQIG1lJPHxRRBKA}{5y-UdC1YTlSD1C9CD2p71Q}{192.168.0.67}{192.168.0.67:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node03}{qGvT8QxISguFBRdFXrOu-Q}{71jE8PGGTG2Pp-U41WnyJw}{192.168.0.68}{192.168.0.68:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:9300, 192.168.0.67:9300, 192.168.0.68:9300] from hosts providers and [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2024-07-10T10:48:38,604][ERROR][o.o.s.l.LogTypeService ] [node01] Failed creating LOG_TYPE_INDEX
org.opensearch.discovery.ClusterManagerNotDiscoveredException: null
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$1.onTimeout(TransportClusterManagerNodeAction.java:356) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:737) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:882) [opensearch-2.15.0.jar:2.15.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
[2024-07-10T10:48:38,609][WARN ][o.o.s.SecurityAnalyticsPlugin] [node01] Failed to initialize LogType config index and builtin log types
[2024-07-10T10:48:38,620][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node01] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: ClusterManagerNotDiscoveredException[null]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.15.0.jar:2.15.0]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.15.0.jar:2.15.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.15.0.jar:2.15.0]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.15.0.jar:2.15.0]
Caused by: org.opensearch.discovery.ClusterManagerNotDiscoveredException
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$1.onTimeout(TransportClusterManagerNodeAction.java:356) ~[opensearch-2.15.0.jar:2.15.0]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) ~[opensearch-2.15.0.jar:2.15.0]
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) ~[opensearch-2.15.0.jar:2.15.0]
at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:737) ~[opensearch-2.15.0.jar:2.15.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:882) ~[opensearch-2.15.0.jar:2.15.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
uncaught exception in thread [main]
ClusterManagerNotDiscoveredException[null]
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$1.onTimeout(TransportClusterManagerNodeAction.java:356)
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394)
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294)
at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:737)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:882)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch-cluster.log
[2024-07-10T10:48:45,426][WARN ][r.suppressed ] [node01] path: /_cluster/health, params: {}
org.opensearch.discovery.ClusterManagerNotDiscoveredException: null
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$1.onTimeout(TransportClusterManagerNodeAction.java:356) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:737) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:882) [opensearch-2.15.0.jar:2.15.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
[2024-07-10T10:48:48,595][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node01] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [opensearch-node1, opensearch-node2, opensearch-node3] to bootstrap a cluster: have discovered [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node02}{5RPTs2DHQIG1lJPHxRRBKA}{5y-UdC1YTlSD1C9CD2p71Q}{192.168.0.67}{192.168.0.67:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node03}{qGvT8QxISguFBRdFXrOu-Q}{71jE8PGGTG2Pp-U41WnyJw}{192.168.0.68}{192.168.0.68:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:9300, 192.168.0.67:9300, 192.168.0.68:9300] from hosts providers and [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2024-07-10T10:48:58,596][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node01] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [opensearch-node1, opensearch-node2, opensearch-node3] to bootstrap a cluster: have discovered [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node02}{5RPTs2DHQIG1lJPHxRRBKA}{5y-UdC1YTlSD1C9CD2p71Q}{192.168.0.67}{192.168.0.67:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node03}{qGvT8QxISguFBRdFXrOu-Q}{71jE8PGGTG2Pp-U41WnyJw}{192.168.0.68}{192.168.0.68:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:9300, 192.168.0.67:9300, 192.168.0.68:9300] from hosts providers and [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2024-07-10T10:49:08,597][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node01] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [opensearch-node1, opensearch-node2, opensearch-node3] to bootstrap a cluster: have discovered [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node02}{5RPTs2DHQIG1lJPHxRRBKA}{5y-UdC1YTlSD1C9CD2p71Q}{192.168.0.67}{192.168.0.67:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node03}{qGvT8QxISguFBRdFXrOu-Q}{71jE8PGGTG2Pp-U41WnyJw}{192.168.0.68}{192.168.0.68:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:9300, 192.168.0.67:9300, 192.168.0.68:9300] from hosts providers and [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2024-07-10T10:49:18,598][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node01] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [opensearch-node1, opensearch-node2, opensearch-node3] to bootstrap a cluster: have discovered [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node02}{5RPTs2DHQIG1lJPHxRRBKA}{5y-UdC1YTlSD1C9CD2p71Q}{192.168.0.67}{192.168.0.67:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node03}{qGvT8QxISguFBRdFXrOu-Q}{71jE8PGGTG2Pp-U41WnyJw}{192.168.0.68}{192.168.0.68:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:9300, 192.168.0.67:9300, 192.168.0.68:9300] from hosts providers and [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2024-07-10T10:49:28,599][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node01] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [opensearch-node1, opensearch-node2, opensearch-node3] to bootstrap a cluster: have discovered [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node02}{5RPTs2DHQIG1lJPHxRRBKA}{5y-UdC1YTlSD1C9CD2p71Q}{192.168.0.67}{192.168.0.67:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node03}{qGvT8QxISguFBRdFXrOu-Q}{71jE8PGGTG2Pp-U41WnyJw}{192.168.0.68}{192.168.0.68:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:9300, 192.168.0.67:9300, 192.168.0.68:9300] from hosts providers and [{node01}{zj4riFT1QV-czKxuUQsSRQ}{-8c40b82RDCDciEnZq9tLQ}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0