Opensearch cluster_Manger node startup failed

[Di_Di]

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch-2.8.0
Host OS: Windows10, Windows Server 2012/2019

Describe the issue:
If I set my node rule as “Cluster_Manager”, it can not startup.
Following error messages is from my opensearch windows console output:

*[2023-09-20T15:23:13,594][INFO ][o.o.b.BootstrapChecks    ] [Data-Center] bound or publishing to a non-loopback address, enforcing bootstrap checks*

[2023-09-20T15:23:43,634][WARN ][o.o.n.Node ] [Data-Center] timed out while waiting for initial discovery state - timeout: 30s
[2023-09-20T15:23:43,629][WARN ][o.c.o.c.s.ConfigSyncService] [Data-Center] Could not start ConfigFileUpdater. Retrying to start it.
org.opensearch.discovery.ClusterManagerNotDiscoveredException: null

•    at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAct*
  

ion.java:350) [opensearch-2.8.0.jar:2.8.0]

•    at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) [opensearch-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) [opensearch-2.8.0.jar:2.8*
  

.0]

•    at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707) [opensearch-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [opensearch-2.8.0.jar:2.8.0]*
  

•    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]*
  

•    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]*
  

•    at java.lang.Thread.run(Thread.java:1589) [?:?]*
  

[2023-09-20T15:23:43,645][INFO ][o.o.h.AbstractHttpServerTransport] [Data-Center] publish_address {192.168.10.225:9200}, bound_addresses {192.168.10.2
25:9200}
[2023-09-20T15:23:43,647][INFO ][o.o.n.Node ] [Data-Center] started
[2023-09-20T15:23:43,648][INFO ][o.o.s.OpenSearchSecurityPlugin] [Data-Center] Node started
[2023-09-20T15:23:43,650][INFO ][o.o.s.OpenSearchSecurityPlugin] [Data-Center] 0 OpenSearch Security modules loaded so far: []
[2023-09-20T15:24:13,677][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [Data-Center] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: ClusterManagerNotDiscoveredException[null]

•    at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.8.0.jar:2.8.0]*
  

Caused by: org.opensearch.discovery.ClusterManagerNotDiscoveredException

•    at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAct*
  

ion.java:350) ~[opensearch-2.8.0.jar:2.8.0]

•    at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) ~[opensearch-2.8.0.jar:2.8.0*
  

]

•    at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) ~[opensearch-2.8.0.jar:2.*
  

8.0]

•    at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707) ~[opensearch-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.8.0.jar:2.8.0]*
  

•    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]*
  

•    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]*
  

•    at java.lang.Thread.run(Thread.java:1589) [?:?]*
  

uncaught exception in thread [main]
ClusterManagerNotDiscoveredException[null]

•    at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAct*
  

ion.java:350)

•    at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394)*
  

•    at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294)*
  

•    at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707)*
  

•    at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)*
  

•    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)*
  

•    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)*
  

•    at java.base/java.lang.Thread.run(Thread.java:1589)*
  

For complete error details, refer to the log at F:\fess_cluster\opensearch-2.8.0\logs\cluster-tester.log
[2023-09-20T15:24:13,703][INFO ][o.o.n.Node ] [Data-Center] stopping …
[2023-09-20T15:24:13,707][WARN ][o.c.o.c.s.ConfigSyncService] [Data-Center] Could not start ConfigFileUpdater. Retrying to start it.
org.opensearch.node.NodeClosedException: node closed {Data-Center}{c1XwkbktS2qE-rokLy7rTQ}{HJB-Io6JT3y52da5WfLR5Q}{192.168.10.225}{192.168.10.225:9300
}{cluster_nanager}{shard_indexing_pressure_enabled=true}

•    at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onClusterServiceClose(TransportClusterMa*
  

nagerNodeAction.java:341) [opensearch-2.8.0.jar:2.8.0]

•    at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:387) [opensearch-2.8.*
  

0.jar:2.8.0]

•    at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:276) [opensearch-2.8.0.jar:2.8.0*
  

]

•    at org.opensearch.cluster.service.ClusterApplierService.doStop(ClusterApplierService.java:195) [opensearch-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:97) [opensearch-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.cluster.service.ClusterService.doStop(ClusterService.java:141) [opensearch-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:97) [opensearch-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.node.Node.stop(Node.java:1377) [opensearch-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.node.Node.close(Node.java:1403) [opensearch-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) [opensearch-common-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) [opensearch-common-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:81) [opensearch-common-2.8.0.jar:2.8.0]*
  

•    at org.opensearch.bootstrap.Bootstrap$4.run(Bootstrap.java:206) [opensearch-2.8.0.jar:2.8.0]*
  

[2023-09-20T15:24:13,732][INFO ][o.o.n.Node ] [Data-Center] stopped
[2023-09-20T15:24:13,734][INFO ][o.o.n.Node ] [Data-Center] closing …
[2023-09-20T15:24:13,743][INFO ][o.o.n.Node ] [Data-Center] closed

But if I comment out node role setting line:
node.roles: [ cluster_nanager ]
Then it startup succeed.
Why?

Configuration:

Relevant Logs or Screenshots:

In the “opensearch.yml” file, if added node roles that the node is set as cluster_manager:
node.roles: [ cluster_nanager ]
The OPS can not succeed to start, unless I removed node roles as “Cluster_Manager” setting line.

What’s wrong with my cluster-manager node setting?

[radu.gheorghe]

Can you post your opensearch.yml here?

org.opensearch.discovery.ClusterManagerNotDiscoveredException seems to imply that there’s no existing cluster manager. If this is the first cluster manager, it has to also be in the list of discovery.seed_hosts as well as cluster.initial_cluster_manager_nodes.

[Di_Di]

Thank for your reply.

Yes, this is the first cluster-manager node in my opensear cluster, also is the first start-up node.
Here is my opensearch.yml file content:

=

======================== OpenSearch Configuration =========================

NOTE: OpenSearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please consult the documentation for further information on configuration options:

https://www.opensearch.org

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

#cluster.name: my-application
cluster.name: cluster-tester

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

#node.name: node-1
node.name: Data-Center
node.roles: [ cluster_nanager ]

Add custom attributes to the node:

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

#path.data: /path/to/data

Path to log files:

#path.logs: /path/to/logs

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

#bootstrap.memory_lock: true

Make sure that the heap size is set to about half the memory available

on the system and that the owner of the process is allowed to use this

limit.

OpenSearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

#network.host: 192.168.0.1
network.host: 192.168.10.225

Set a custom port for HTTP:

#http.port: 9200

For more information, consult the network module documentation.

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when this node is started:

The default list of hosts is [“127.0.0.1”, “[::1]”]

#discovery.seed_hosts: [“host1”, “host2”]
discovery.seed_hosts: [“192.168.10.225”,“192.168.10.226”]

Bootstrap the cluster using an initial set of cluster-manager-eligible nodes:

#cluster.initial_cluster_manager_nodes: [“node-1”, “node-2”]
cluster.initial_cluster_manager_nodes: [“192.168.10.225”]

For more information, consult the discovery and cluster formation module documentation.

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

#gateway.recover_after_nodes: 3

For more information, consult the gateway module documentation.

---------------------------------- Various -----------------------------------

Require explicit names when deleting indices:

#action.destructive_requires_name: true

---------------------------------- Remote Store -----------------------------------

Controls whether cluster imposes index creation only with remote store enabled

cluster.remote_store.enabled: true

Repository to use for segment upload while enforcing remote store for an index

cluster.remote_store.repository: my-repo-1

Controls whether cluster imposes index creation only with translog remote store enabled

cluster.remote_store.translog.enabled: true

Repository to use for translog upload while enforcing remote store for an index

cluster.remote_store.translog.repository: my-repo-1

---------------------------------- Experimental Features -----------------------------------

Gates the visibility of the experimental segment replication features until they are production ready.

OpenSearch.experimental.feature.segment_replication_experimental.enabled: false

Gates the visibility of the index setting that allows persisting data to remote store along with local disk.

Once the feature is ready for production release, this feature flag can be removed.

OpenSearch.experimental.feature.remote_store.enabled: false

Gates the functionality of a new parameter to the snapshot restore API

that allows for creation of a new index type that searches a snapshot

directly in a remote repository without restoring all index data to disk

ahead of time.

OpenSearch.experimental.feature.searchable_snapshot.enabled: false

Gates the functionality of enabling extensions to work with OpenSearch.

This feature enables applications to extend features of OpenSearch outside of

the core.

OpenSearch.experimental.feature.extensions.enabled: false

Gates the search pipeline feature. This feature enables configurable processors

for search requests and search responses, similar to ingest pipelines.

OpenSearch.experimental.feature.search_pipeline.enabled: false

Added by Yan

configsync.config_path: F:/fess_cluster/opensearch-2.8.0/data/config/
plugins.security.disabled: true

============================End of Oepnsearch.yml =============

After I started it, failed with below error nessages;
=================== Error Messages Ouput ====================

[2023-09-22T08:22:43,857][INFO ][o.o.m.b.MLCircuitBreakerService] [Data-Center] Registered ML memory breaker.
[2023-09-22T08:22:43,860][INFO ][o.o.m.b.MLCircuitBreakerService] [Data-Center] Registered ML disk breaker.
[2023-09-22T08:22:43,861][INFO ][o.o.m.b.MLCircuitBreakerService] [Data-Center] Registered ML native memory breaker.
[2023-09-22T08:22:43,979][INFO ][o.r.Reflections ] [Data-Center] Reflections took 47 ms to scan 1 urls, producing 15 keys and 37 values
[2023-09-22T08:22:44,820][INFO ][o.o.t.NettyAllocator ] [Data-Center] creating NettyAllocator with the following configs: [name=opensearch_configu
red, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={opensearch.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1g
c_region_size=32mb}]
[2023-09-22T08:22:44,900][INFO ][o.o.d.DiscoveryModule ] [Data-Center] using discovery type [zen] and seed hosts providers [settings]
[2023-09-22T08:22:45,416][WARN ][o.o.g.DanglingIndicesState] [Data-Center] gateway.auto_import_dangling_indices is disabled, dangling indices will not
be automatically detected or imported and must be managed manually
[2023-09-22T08:22:46,119][INFO ][o.o.n.Node ] [Data-Center] initialized
[2023-09-22T08:22:46,121][INFO ][o.o.n.Node ] [Data-Center] starting …
[2023-09-22T08:22:46,284][INFO ][o.o.t.TransportService ] [Data-Center] publish_address {192.168.10.225:9300}, bound_addresses {192.168.10.225:9300}

[2023-09-22T08:22:46,447][INFO ][o.o.b.BootstrapChecks ] [Data-Center] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-09-22T08:23:16,474][WARN ][o.c.o.c.s.ConfigSyncService] [Data-Center] Could not start ConfigFileUpdater. Retrying to start it.
org.opensearch.discovery.ClusterManagerNotDiscoveredException: null
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAct
ion.java:350) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) [opensearch-2.8.0.jar:2.8.0]

    at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) [opensearch-2.8.0.jar:2.8

.0]
at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [opensearch-2.8.0.jar:2.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
at java.lang.Thread.run(Thread.java:1589) [?:?]
[2023-09-22T08:23:16,481][WARN ][o.o.n.Node ] [Data-Center] timed out while waiting for initial discovery state - timeout: 30s
[2023-09-22T08:23:16,499][INFO ][o.o.h.AbstractHttpServerTransport] [Data-Center] publish_address {192.168.10.225:9200}, bound_addresses {192.168.10.2
25:9200}
[2023-09-22T08:23:16,501][INFO ][o.o.n.Node ] [Data-Center] started
[2023-09-22T08:23:16,502][INFO ][o.o.s.OpenSearchSecurityPlugin] [Data-Center] Node started
[2023-09-22T08:23:16,504][INFO ][o.o.s.OpenSearchSecurityPlugin] [Data-Center] 0 OpenSearch Security modules loaded so far:
[2023-09-22T08:23:46,526][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [Data-Center] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: ClusterManagerNotDiscoveredException[null]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.8.0.jar:2.8.0]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.8.0.jar:2.8.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.8.0.jar:2.8.0]
Caused by: org.opensearch.discovery.ClusterManagerNotDiscoveredException
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAct
ion.java:350) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) ~[opensearch-2.8.0.jar:2.8.0
]
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) ~[opensearch-2.8.0.jar:2.
8.0]
at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.8.0.jar:2.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) [?:?]
uncaught exception in thread [main]
ClusterManagerNotDiscoveredException[null]
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAct
ion.java:350)
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394)
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294)
at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1589)
For complete error details, refer to the log at F:\fess_cluster\opensearch-2.8.0\logs\cluster-tester.log
[2023-09-22T08:23:46,553][INFO ][o.o.n.Node ] [Data-Center] stopping …
[2023-09-22T08:23:46,556][WARN ][o.c.o.c.s.ConfigSyncService] [Data-Center] Could not start ConfigFileUpdater. Retrying to start it.
org.opensearch.node.NodeClosedException: node closed {Data-Center}{RWwljk6cQn6LLd3hnt9YmQ}{GkWjHTOyTHWaLby_Y2j6oA}{192.168.10.225}{192.168.10.225:9300
}{cluster_nanager}{shard_indexing_pressure_enabled=true}
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onClusterServiceClose(TransportClusterMa
nagerNodeAction.java:341) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:387) [opensearch-2.8.
0.jar:2.8.0]
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:276) [opensearch-2.8.0.jar:2.8.0
]
at org.opensearch.cluster.service.ClusterApplierService.doStop(ClusterApplierService.java:195) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:97) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.cluster.service.ClusterService.doStop(ClusterService.java:141) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:97) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.node.Node.stop(Node.java:1377) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.node.Node.close(Node.java:1403) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) [opensearch-common-2.8.0.jar:2.8.0]
at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) [opensearch-common-2.8.0.jar:2.8.0]
at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:81) [opensearch-common-2.8.0.jar:2.8.0]
at org.opensearch.bootstrap.Bootstrap$4.run(Bootstrap.java:206) [opensearch-2.8.0.jar:2.8.0]
[2023-09-22T08:23:46,581][INFO ][o.o.n.Node ] [Data-Center] stopped
[2023-09-22T08:23:46,583][INFO ][o.o.n.Node ] [Data-Center] closing …
[2023-09-22T08:23:46,592][INFO ][o.o.n.Node ] [Data-Center] closed

==================== End of Error Messages Output =================

Please teach me where am I wrong?

[Di_Di]

Thanks your reply.
Yes, it the only cluster-manager node in my cluster, and is the first node strt-up.
Following is my opensearch.yml settings I modified:

cluster.name: cluster-tester

node.roles: [ cluster_nanager ]
node.name: Data-Center
network.host: 192.168.10.225
discovery.seed_hosts: [“192.168.10.225”,“192.168.10.226”]
cluster.initial_cluster_manager_nodes: [“192.168.10.225”]
configsync.config_path: F:/fess_cluster/opensearch-2.8.0/data/config/
plugins.security.disabled: true

I tried again, but failed once more.
Some error messages:
[2023-09-22T08:22:46,284][INFO ][o.o.t.TransportService ] [Data-Center] publish_address {192.168.10.225:9300}, bound_addresses {192.168.10.225:9300}

[2023-09-22T08:22:46,447][INFO ][o.o.b.BootstrapChecks ] [Data-Center] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-09-22T08:23:16,474][WARN ][o.c.o.c.s.ConfigSyncService] [Data-Center] Could not start ConfigFileUpdater. Retrying to start it.
org.opensearch.discovery.ClusterManagerNotDiscoveredException: null
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAct
ion.java:350) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) [opensearch-2.8.0.jar:2.8.0]

[Di_Di]

More error messages: output:

[2023-09-22T08:23:16,481][WARN ][o.o.n.Node ] [Data-Center] timed out while waiting for initial discovery state - timeout: 30s
[2023-09-22T08:23:16,499][INFO ][o.o.h.AbstractHttpServerTransport] [Data-Center] publish_address {192.168.10.225:9200}, bound_addresses {192.168.10.2
25:9200}
[2023-09-22T08:23:16,501][INFO ][o.o.n.Node ] [Data-Center] started
[2023-09-22T08:23:16,502][INFO ][o.o.s.OpenSearchSecurityPlugin] [Data-Center] Node started
[2023-09-22T08:23:16,504][INFO ][o.o.s.OpenSearchSecurityPlugin] [Data-Center] 0 OpenSearch Security modules loaded so far:
[2023-09-22T08:23:46,526][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [Data-Center] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: ClusterManagerNotDiscoveredException[null]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.8.0.jar:2.8.0]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.8.0.jar:2.8.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.8.0.jar:2.8.0]
Caused by: org.opensearch.discovery.ClusterManagerNotDiscoveredException
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAct
ion.java:350) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) ~[opensearch-2.8.0.jar:2.8.0
]
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) ~[opensearch-2.8.0.jar:2.
8.0]
at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.8.0.jar:2.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) [?:?]
uncaught exception in thread [main]
ClusterManagerNotDiscoveredException[null]
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAct
ion.java:350)
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394)
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294)
at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1589)
For complete error details, refer to the log at F:\fess_cluster\opensearch-2.8.0\logs\cluster-tester.log
[2023-09-22T08:23:46,553][INFO ][o.o.n.Node ] [Data-Center] stopping …
[2023-09-22T08:23:46,556][WARN ][o.c.o.c.s.ConfigSyncService] [Data-Center] Could not start ConfigFileUpdater. Retrying to start it.
org.opensearch.node.NodeClosedException: node closed {Data-Center}{RWwljk6cQn6LLd3hnt9YmQ}{GkWjHTOyTHWaLby_Y2j6oA}{192.168.10.225}{192.168.10.225:9300
}{cluster_nanager}{shard_indexing_pressure_enabled=true}
at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onClusterServiceClose(TransportClusterMa
nagerNodeAction.java:341) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:387) [opensearch-2.8.
0.jar:2.8.0]
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:276) [opensearch-2.8.0.jar:2.8.0
]
at org.opensearch.cluster.service.ClusterApplierService.doStop(ClusterApplierService.java:195) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:97) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.cluster.service.ClusterService.doStop(ClusterService.java:141) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:97) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.node.Node.stop(Node.java:1377) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.node.Node.close(Node.java:1403) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) [opensearch-common-2.8.0.jar:2.8.0]
at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) [opensearch-common-2.8.0.jar:2.8.0]
at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:81) [opensearch-common-2.8.0.jar:2.8.0]
at org.opensearch.bootstrap.Bootstrap$4.run(Bootstrap.java:206) [opensearch-2.8.0.jar:2.8.0]
[2023-09-22T08:23:46,581][INFO ][o.o.n.Node ] [Data-Center] stopped
[2023-09-22T08:23:46,583][INFO ][o.o.n.Node ] [Data-Center] closing …
[2023-09-22T08:23:46,592][INFO ][o.o.n.Node ] [Data-Center] closed

[radu.gheorghe]

@Di_Di

node.roles: [ cluster_nanager ]

It looks like you have a typo here.

[Di_Di]

Ah! How foolish I am!
However, even I corrected to [" cluster_manager" ], my cluster manager still can not start-up normally.
Anyway, new errors messages occurred:
Windows PowerShell
Copyright (C) 2014 Microsoft Corporation. All rights reserved.

PS C:\Users\Administrator> f:
PS F:> cd .\fess_cluster\opensearch-2.8.0\bin
PS F:\fess_cluster\opensearch-2.8.0\bin> .\opensearch.bat
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/F:/fess_cluster/opensearch-2.8.0/lib/opensearch-2.8.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/F:/fess_cluster/opensearch-2.8.0/lib/opensearch-2.8.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2023-09-28T09:43:08,321][INFO ][o.o.n.Node ] [Data-Center] version[2.8.0], pid[5444], build[zip/db90a415ff2fd428b4f7b3f800a51dc229287cb4/2023-06-03T06:29:28.546587700Z], OS[Windows Server 2012 R2/6.3/amd64], JVM[Oracle Corporation/Java HotSpot™ 64-Bit Server VM/19/19+36-2238]
[2023-09-28T09:43:08,324][INFO ][o.o.n.Node ] [Data-Center] JVM home [C:\Program Files\Java\jdk-19], using bundled JDK [false]
[2023-09-28T09:43:08,325][INFO ][o.o.n.Node ] [Data-Center] JVM arguments [-Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms96g, -Xmx96g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\2\opensearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -XX:MaxDirectMemorySize=51539607552, -Dopensearch, -Dopensearch.path.home=F:\fess_cluster\opensearch-2.8.0, -Dopensearch.path.conf=F:\fess_cluster\opensearch-2.8.0\config, -Dopensearch.distribution.type=zip, -Dopensearch.
bundled_jdk=true]
[2023-09-28T09:43:10,021][INFO ][o.o.s.s.t.SSLConfig ] [Data-Center] SSL dual mode is disabled
[2023-09-28T09:43:10,023][WARN ][o.o.s.OpenSearchSecurityPlugin] [Data-Center] OpenSearch Security plugin installed but disabled. This can expose your configuration (including passwords) to the public.
[2023-09-28T09:43:11,039][INFO ][o.o.i.r.ReindexPlugin ] [Data-Center] ReindexPlugin reloadSPI called
[2023-09-28T09:43:11,042][INFO ][o.o.i.r.ReindexPlugin ] [Data-Center] Unable to find any implementation for RemoteReindexExtension
[2023-09-28T09:43:11,088][INFO ][o.o.j.JobSchedulerPlugin ] [Data-Center] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2023-09-28T09:43:11,126][INFO ][o.o.j.JobSchedulerPlugin ] [Data-Center] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2023-09-28T09:43:11,128][INFO ][o.o.j.JobSchedulerPlugin ] [Data-Center] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2023-09-28T09:43:11,199][INFO ][o.o.j.JobSchedulerPlugin ] [Data-Center] Loaded scheduler extension: observability, index: .opensearch-observability-job
[2023-09-28T09:43:11,211][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [aggs-matrix-stats]
[2023-09-28T09:43:11,212][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [analysis-common]
[2023-09-28T09:43:11,213][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [geo]
[2023-09-28T09:43:11,214][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [ingest-common]
[2023-09-28T09:43:11,215][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [ingest-geoip]
[2023-09-28T09:43:11,217][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [ingest-user-agent]
[2023-09-28T09:43:11,218][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [lang-expression]
[2023-09-28T09:43:11,219][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [lang-mustache]
[2023-09-28T09:43:11,220][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [lang-painless]
[2023-09-28T09:43:11,221][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [mapper-extras]
[2023-09-28T09:43:11,222][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [opensearch-dashboards]
[2023-09-28T09:43:11,223][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [parent-join]
[2023-09-28T09:43:11,225][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [percolator]
[2023-09-28T09:43:11,226][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [rank-eval]
[2023-09-28T09:43:11,227][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [reindex]
[2023-09-28T09:43:11,228][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [repository-url]
[2023-09-28T09:43:11,229][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [search-pipeline-common]
[2023-09-28T09:43:11,230][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [transport-netty4]
[2023-09-28T09:43:11,233][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [analysis-extension]
[2023-09-28T09:43:11,234][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [analysis-fess]
[2023-09-28T09:43:11,235][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [configsync]
[2023-09-28T09:43:11,236][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [minhash]
[2023-09-28T09:43:11,238][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-alerting]
[2023-09-28T09:43:11,239][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-anomaly-detection]
[2023-09-28T09:43:11,240][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-asynchronous-search]
[2023-09-28T09:43:11,241][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-cross-cluster-replication]
[2023-09-28T09:43:11,242][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-geospatial]
[2023-09-28T09:43:11,243][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-index-management]
[2023-09-28T09:43:11,244][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-job-scheduler]
[2023-09-28T09:43:11,246][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-knn]
[2023-09-28T09:43:11,247][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-ml]
[2023-09-28T09:43:11,248][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-neural-search]
[2023-09-28T09:43:11,249][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-notifications]
[2023-09-28T09:43:11,250][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-notifications-core]
[2023-09-28T09:43:11,251][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-observability]
[2023-09-28T09:43:11,252][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-reports-scheduler]
[2023-09-28T09:43:11,254][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-security]
[2023-09-28T09:43:11,255][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-security-analytics]
[2023-09-28T09:43:11,256][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-sql]
[2023-09-28T09:43:11,326][INFO ][o.o.e.ExtensionsManager ] [Data-Center] ExtensionsManager initialized
[2023-09-28T09:43:11,344][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] [Data-Center] Using MemorySegmentIndexInput with Java 19; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
[2023-09-28T09:43:11,358][INFO ][o.o.e.NodeEnvironment ] [Data-Center] using [1] data paths, mounts [[繝懊Μ繝･繝ｼ繝 (F:)]], net usable_space [937.4gb], net total_space [1023.9gb], types [NTFS]
[2023-09-28T09:43:11,361][INFO ][o.o.e.NodeEnvironment ] [Data-Center] heap size [96gb], compressed ordinary object pointers [false]
[2023-09-28T09:43:11,435][INFO ][o.o.n.Node ] [Data-Center] node name [Data-Center], node ID [0-TzzDksSXqkHtkktGRnCQ], cluster name [cluster-tester], roles [cluster_manager]
[2023-09-28T09:43:15,743][INFO ][o.o.a.b.ADCircuitBreakerService] [Data-Center] Registered memory breaker.
[2023-09-28T09:43:16,264][INFO ][o.o.m.b.MLCircuitBreakerService] [Data-Center] Registered ML memory breaker.
[2023-09-28T09:43:16,266][INFO ][o.o.m.b.MLCircuitBreakerService] [Data-Center] Registered ML disk breaker.
[2023-09-28T09:43:16,268][INFO ][o.o.m.b.MLCircuitBreakerService] [Data-Center] Registered ML native memory breaker.
[2023-09-28T09:43:16,385][INFO ][o.r.Reflections ] [Data-Center] Reflections took 48 ms to scan 1 urls, producing 15 keys and 37 values
[2023-09-28T09:43:17,293][INFO ][o.o.t.NettyAllocator ] [Data-Center] creating NettyAllocator with the following configs: [name=opensearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={opensearch.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=32mb}]
[2023-09-28T09:43:17,374][INFO ][o.o.d.DiscoveryModule ] [Data-Center] using discovery type [zen] and seed hosts providers [settings]
[2023-09-28T09:43:17,910][WARN ][o.o.g.DanglingIndicesState] [Data-Center] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-09-28T09:43:18,655][INFO ][o.o.n.Node ] [Data-Center] initialized
[2023-09-28T09:43:18,656][INFO ][o.o.n.Node ] [Data-Center] starting …
[2023-09-28T09:43:18,788][INFO ][o.o.t.TransportService ] [Data-Center] publish_address {192.168.10.225:9300}, bound_addresses {192.168.10.225:9300}
[2023-09-28T09:43:18,974][INFO ][o.o.b.BootstrapChecks ] [Data-Center] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-09-28T09:43:19,011][INFO ][o.o.c.c.Coordinator ] [Data-Center] setting initial configuration to VotingConfiguration{0-TzzDksSXqkHtkktGRnCQ}
[2023-09-28T09:43:19,194][INFO ][o.o.c.s.MasterService ] [Data-Center] elected-as-cluster-manager ([1] nodes joined)[{Data-Center}{0-TzzDksSXqkHtkktGRnCQ}{k4cWLKcaRyOQgAwtKqs_yQ}{192.168.10.225}{192.168.10.225:9300}{m}{shard_indexing_pressure_enabled=true} elect leader, BECOME_CLUSTER_MANAGER_TASK, FINISH_ELECTION], term: 1, version: 1, delta: cluster-manager node changed {previous , current [{Data-Center}{0-TzzDksSXqkHtkktGRnCQ}{k4cWLKcaRyOQgAwtKqs_yQ}{192.168.10.225}{192.168.10.225:9300}{m}{shard_indexing_pressure_enabled=true}]}
[2023-09-28T09:43:19,244][INFO ][o.o.c.c.CoordinationState] [Data-Center] cluster UUID set to [EUoZDm1mRceZeVjKkDQLIw]
[2023-09-28T09:43:19,279][INFO ][o.o.c.s.ClusterApplierService] [Data-Center] cluster-manager node changed {previous , current [{Data-Center}{0-TzzDksSXqkHtkktGRnCQ}{k4cWLKcaRyOQgAwtKqs_yQ}{192.168.10.225}{192.168.10.225:9300}{m}{shard_indexing_pressure_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2023-09-28T09:43:19,289][INFO ][o.o.a.c.ADClusterEventListener] [Data-Center] Cluster is not recovered yet.
[2023-09-28T09:43:19,299][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration
[2023-09-28T09:43:19,331][INFO ][o.o.i.i.ManagedIndexCoordinator] [Data-Center] Cache cluster manager node onClusterManager time: 1695861799330
[2023-09-28T09:43:19,350][INFO ][o.o.h.AbstractHttpServerTransport] [Data-Center] publish_address {192.168.10.225:9200}, bound_addresses {192.168.10.225:9200}
[2023-09-28T09:43:19,351][INFO ][o.o.n.Node ] [Data-Center] started
[2023-09-28T09:43:19,352][INFO ][o.o.d.PeerFinder ] [Data-Center] setting findPeersInterval to [1s] as node commission status = [true] for local node [{Data-Center}{0-TzzDksSXqkHtkktGRnCQ}{k4cWLKcaRyOQgAwtKqs_yQ}{192.168.10.225}{192.168.10.225:9300}{m}{shard_indexing_pressure_enabled=true}]
[2023-09-28T09:43:19,353][INFO ][o.o.s.OpenSearchSecurityPlugin] [Data-Center] Node started
[2023-09-28T09:43:19,356][INFO ][o.o.s.OpenSearchSecurityPlugin] [Data-Center] 0 OpenSearch Security modules loaded so far:
[2023-09-28T09:43:19,368][INFO ][o.o.a.c.HashRing ] [Data-Center] Node added: [0-TzzDksSXqkHtkktGRnCQ]
[2023-09-28T09:43:19,408][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration
[2023-09-28T09:43:19,410][INFO ][o.o.a.c.HashRing ] [Data-Center] All nodes with known AD version: {0-TzzDksSXqkHtkktGRnCQ=ADNodeInfo{version=2.8.0, isEligibleDataNode=false}}
[2023-09-28T09:43:19,411][INFO ][o.o.a.c.HashRing ] [Data-Center] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
[2023-09-28T09:43:19,413][INFO ][o.o.a.c.HashRing ] [Data-Center] Build AD version hash ring successfully
[2023-09-28T09:43:19,415][INFO ][o.o.a.c.ADClusterEventListener] [Data-Center] Init AD version hash ring successfully
[2023-09-28T09:43:19,449][INFO ][o.o.g.GatewayService ] [Data-Center] recovered [0] indices into cluster_state
[2023-09-28T09:43:19,483][INFO ][o.o.p.PluginsService ] [Data-Center] PluginService:onIndexModule index:[.opensearch-observability/kZhLfgTFTyGhhLLSppIzKQ]
[2023-09-28T09:43:19,675][INFO ][o.o.c.m.MetadataCreateIndexService] [Data-Center] [.opensearch-observability] creating index, cause [api], templates , shards [1]/[0]
[2023-09-28T09:43:19,690][INFO ][o.o.c.r.a.AllocationService] [Data-Center] Cluster health status changed from [YELLOW] to [RED] (reason: [index [.opensearch-observability] created]).
[2023-09-28T09:43:19,733][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration
[2023-09-28T09:43:19,739][INFO ][o.o.p.PluginsService ] [Data-Center] PluginService:onIndexModule index:[configsync/xrAbC-1mSAyDrUcY-Si-gQ]
[2023-09-28T09:43:19,775][INFO ][o.o.c.m.MetadataCreateIndexService] [Data-Center] [configsync] creating index, cause [api], templates , shards [1]/[0]
[2023-09-28T09:43:19,809][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration
[2023-09-28T09:43:49,739][INFO ][o.o.o.i.ObservabilityIndex] [Data-Center] observability:Index .opensearch-observability creation Acknowledged
[2023-09-28T09:43:49,741][INFO ][o.o.o.i.ObservabilityTracesIndex] [Data-Center] observability:createMappingTemplate ss4o_trace_template API called
[2023-09-28T09:43:49,754][INFO ][o.o.p.PluginsService ] [Data-Center] PluginService:onIndexModule index:[validate-template-mpybpllmqeyryoqby6hheg/C8QD1lPPRB6wHL29ezTnsg]
[2023-09-28T09:43:49,798][INFO ][o.o.c.m.MetadataIndexTemplateService] [Data-Center] adding index template [ss4o_trace_template] for index patterns [ss4o_traces--]
[2023-09-28T09:43:49,834][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration
[2023-09-28T09:43:49,836][INFO ][o.o.o.i.ObservabilityTracesIndex] [Data-Center] observability:Mapping Template ss4o_trace_template creation Acknowledged
[2023-09-28T09:43:49,837][INFO ][o.o.o.i.ObservabilityMetricsIndex] [Data-Center] observability:createMappingTemplate ss4o_metric_template API called
[2023-09-28T09:43:49,843][INFO ][o.o.p.PluginsService ] [Data-Center] PluginService:onIndexModule index:[validate-template-xnt4tqitrlmq4ajz6qesyq/7c1BxMX3SiiiB2BVGpwEDA]
[2023-09-28T09:43:49,879][INFO ][o.o.c.m.MetadataIndexTemplateService] [Data-Center] adding index template [ss4o_metric_template] for index patterns [ss4o_metrics--]
[2023-09-28T09:43:49,914][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration
[2023-09-28T09:43:49,916][INFO ][o.o.o.i.ObservabilityMetricsIndex] [Data-Center] observability:Mapping Template ss4o_metric_template creation Acknowledged
[2023-09-28T09:44:19,335][INFO ][o.o.i.i.ManagedIndexCoordinator] [Data-Center] Performing move cluster state metadata.
[2023-09-28T09:44:19,340][INFO ][o.o.i.i.ManagedIndexCoordinator] [Data-Center] Performing ISM template migration.
[2023-09-28T09:44:19,342][INFO ][o.o.i.i.MetadataService ] [Data-Center] ISM config index not exist, so we cancel the metadata migration job.
[2023-09-28T09:44:19,344][INFO ][o.o.i.i.m.ISMTemplateService] [Data-Center] Doing ISM template migration 1 time.
[2023-09-28T09:44:19,346][INFO ][o.o.i.i.m.ISMTemplateService] [Data-Center] Use 2023-09-27T23:43:19.330Z as migrating ISM template last_updated_time
[2023-09-28T09:44:19,351][INFO ][o.o.p.PluginsService ] [Data-Center] PluginService:onIndexModule index:[simulate_template_index_aooapnokt1gq94ycbvimqa/7MFo1FMgRS634SqEQ10WHQ]
[2023-09-28T09:44:19,379][INFO ][o.o.p.PluginsService ] [Data-Center] PluginService:onIndexModule index:[simulate_template_index_aooapnokt1gq94ycbvimqa/7MFo1FMgRS634SqEQ10WHQ]
[2023-09-28T09:44:19,418][INFO ][o.o.p.PluginsService ] [Data-Center] PluginService:onIndexModule index:[simulate_template_index_ipmmgiu7ttyvnoamacp3bq/7xjncY6yR92aQ0ZyLhIMow]
[2023-09-28T09:44:19,445][INFO ][o.o.p.PluginsService ] [Data-Center] PluginService:onIndexModule index:[simulate_template_index_ipmmgiu7ttyvnoamacp3bq/7xjncY6yR92aQ0ZyLhIMow]
[2023-09-28T09:44:19,476][INFO ][o.o.i.i.m.ISMTemplateService] [Data-Center] ISM templates: {=[ISMTemplate(indexPatterns=[ss4o_traces--], priority=1, lastUpdatedTime=2023-09-27T23:43:19.330Z), ISMTemplate(indexPatterns=[ss4o_metrics--], priority=1, lastUpdatedTime=2023-09-27T23:43:19.330Z)]}
[2023-09-28T09:44:19,482][INFO ][o.o.i.i.m.ISMTemplateService] [Data-Center] Policies to update:
[2023-09-28T09:44:19,487][INFO ][o.o.i.i.m.ISMTemplateService] [Data-Center] Failure experienced when migrating ISM Template and update ISM policies: {}
[2023-09-28T09:44:19,541][INFO ][o.o.c.s.ClusterSettings ] [Data-Center] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
[2023-09-28T09:44:19,543][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration
[2023-09-28T09:44:19,547][INFO ][o.o.i.i.m.ISMTemplateService] [Data-Center] Successfully update template migration setting
[2023-09-28T09:44:19,828][INFO ][o.c.o.c.s.ConfigSyncService] [Data-Center] ConfigFileUpdater is started at 1m intervals.
[2023-09-28T09:45:19,338][INFO ][o.o.i.i.ManagedIndexCoordinator] [Data-Center] Cancel background move metadata process.
[2023-09-28T09:45:19,340][INFO ][o.o.i.i.ManagedIndexCoordinator] [Data-Center] Performing move cluster state metadata.
[2023-09-28T09:45:19,341][INFO ][o.o.i.i.MetadataService ] [Data-Center] Move metadata has finished.
[2023-09-28T09:45:19,856][ERROR][o.c.o.c.s.ConfigSyncService] [Data-Center] Failed to process ConfigFileUpdater.
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:664) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:372) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:699) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:472) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:273) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:350) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]*****
***** at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]*****
***** at java.lang.Thread.run(Thread.java:1589) [?:?]*****
[2023-09-28T09:46:19,875][ERROR][o.c.o.c.s.ConfigSyncService] [Data-Center] Failed to process ConfigFileUpdater.
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:664) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:372) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:699) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:472) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:273) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:350) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]*****
***** at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]*****
***** at java.lang.Thread.run(Thread.java:1589) [?:?]*****
[2023-09-28T09:47:19,890][ERROR][o.c.o.c.s.ConfigSyncService] [Data-Center] Failed to process ConfigFileUpdater.
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:664) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:372) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:699) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:472) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:273) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:350) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]*****
***** at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]*****
***** at java.lang.Thread.run(Thread.java:1589) [?:?]*****
[2023-09-28T09:48:18,989][INFO ][o.o.j.s.JobSweeper ] [Data-Center] Running full sweep
[2023-09-28T09:48:19,341][INFO ][o.o.i.i.PluginVersionSweepCoordinator] [Data-Center] Canceling sweep ism plugin version job
[2023-09-28T09:48:19,909][ERROR][o.c.o.c.s.ConfigSyncService] [Data-Center] Failed to process ConfigFileUpdater.
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:664) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:372) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:699) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:472) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:273) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:350) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]*****
***** at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]*****
***** at java.lang.Thread.run(Thread.java:1589) [?:?]*****
[2023-09-28T09:49:19,924][ERROR][o.c.o.c.s.ConfigSyncService] [Data-Center] Failed to process ConfigFileUpdater.
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:664) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:372) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:699) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:472) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:273) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:350) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]*****
***** at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]*****
***** at java.lang.Thread.run(Thread.java:1589) [?:?]*****
[2023-09-28T09:50:19,943][ERROR][o.c.o.c.s.ConfigSyncService] [Data-Center] Failed to process ConfigFileUpdater.
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:664) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:372) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:699) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:472) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:273) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:350) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.8.0.jar:2.8.0]*****
***** at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]*****
***** at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]*****
***** at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]*****
***** at java.lang.Thread.run(Thread.java:1589) [?:?]*****

```

What’s wrong now?
Thanks in advanced.

[radu.gheorghe]

@Di_Di

[2023-09-28T09:50:19,943][ERROR][o.c.o.c.s.ConfigSyncService] [Data-Center] Failed to process ConfigFileUpdater.

I have no idea where this comes from. Is this a custom plugin?

[pablo]

@Di_Di Is this related to Opensearch launch failed on cluster manager ?

If yes, please follow the response.

Opensearch launch failed on cluster manager

@Di_Di The OpenSearch node with only the cluster_manager node role can’t hold any data (indices). It requires a data node role to store the data. I’d suggest starting a master node with a data node.

[Di_Di]

Sorry, my late reply for all of aboves.

I’ve already understand what wrong with it.
Thanks for everyone help.