# Opensearch launch failed on cluster manager **URL:** \ **Category:** OpenSearch\ **Created:** [October 2, 2023, 1:08am UTC](https://forum.opensearch.org/t/opensearch-launch-failed-on-cluster-manager/16096 "2023-10-02T01:08:55Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Di\_Di](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/di_di/32/6127_2.png) [@Di\_Di](https://forum.opensearch.org/u/Di_Di)\ **Post date:** [October 2, 2023, 1:08am UTC](https://forum.opensearch.org/t/opensearch-launch-failed-on-cluster-manager/16096/1 "2023-10-02T01:08:56Z")
**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser): **Describe the issue** : I’m trying to create an opensearch cluster, in which contains a “cluster manager” node, and tothers data nodes. But when I tried to start opensearch on the cluster manager node which is the first node to start in my cluster, but start-up process aborted with errors , the main error is as below: _ **org.opensearch.action.search.SearchPhaseExecutionException: all shards failed** _ However, if I removed “node.roles: [cluster\_manager]” setting, opensearch starting process succeed without any problem. **Configuration** : Following is my opensearch.yml: # ======================== OpenSearch Configuration ========================= # # NOTE: OpenSearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # [https://www.opensearch.org](https://www.opensearch.org) # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # #cluster.name: my-application _ **cluster.name: cluster-tester** _ # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # #node.name: node-1 _ **node.name: Data-Center** _ _**node.roles: [cluster\_manager]**_ # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # #path.data: /path/to/data # # Path to log files: # #path.logs: /path/to/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory\_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # OpenSearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # #network.host: 192.168.0.1 _ **network.host: 192.168.10.225** _ # # Set a custom port for HTTP: # _ **http.port: 9200** _ # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is [“127.0.0.1”, “[::1]”] # #discovery.seed\_hosts: [“host1”, “host2”] _**discovery.seed\_hosts: [“192.168.10.225”,“192.168.10.226”]**_ # # Bootstrap the cluster using an initial set of cluster-manager-eligible nodes: # #cluster.initial\_cluster\_manager\_nodes: [“node-1”, “node-2”] _**cluster.initial\_cluster\_manager\_nodes: [“192.168.10.225”]**_ # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover\_after\_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive\_requires\_name: true # # ---------------------------------- Remote Store ----------------------------------- # Controls whether cluster imposes index creation only with remote store enabled # cluster.remote\_store.enabled: true # # Repository to use for segment upload while enforcing remote store for an index # cluster.remote\_store.repository: my-repo-1 # # Controls whether cluster imposes index creation only with translog remote store enabled # cluster.remote\_store.translog.enabled: true # # Repository to use for translog upload while enforcing remote store for an index # cluster.remote\_store.translog.repository: my-repo-1 # # ---------------------------------- Experimental Features ----------------------------------- # # Gates the visibility of the experimental segment replication features until they are production ready. # #OpenSearch.experimental.feature.segment\_replication\_experimental.enabled: false # # # Gates the visibility of the index setting that allows persisting data to remote store along with local disk. # Once the feature is ready for production release, this feature flag can be removed. # #OpenSearch.experimental.feature.remote\_store.enabled: false # # # Gates the functionality of a new parameter to the snapshot restore API # that allows for creation of a new index type that searches a snapshot # directly in a remote repository without restoring all index data to disk # ahead of time. # #OpenSearch.experimental.feature.searchable\_snapshot.enabled: false # # # Gates the functionality of enabling extensions to work with OpenSearch. # This feature enables applications to extend features of OpenSearch outside of # the core. # #OpenSearch.experimental.feature.extensions.enabled: false # # # Gates the search pipeline feature. This feature enables configurable processors # for search requests and search responses, similar to ingest pipelines. #OpenSearch.experimental.feature.search\_pipeline.enabled: false # Added by Yan configsync.config\_path: F:/fess\_cluster/opensearch-2.8.0/data/config/ plugins.security.disabled: true **Relevant Logs or Screenshots** : # Below is all output info: in my console: PS F:\fess\_cluster\opensearch-2.8.0\bin\> .\opensearch.bat WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/F:/fess\_cluster/opensearch-2.8.0/lib/opensearch-2.8.0.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/F:/fess\_cluster/opensearch-2.8.0/lib/opensearch-2.8.0.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security WARNING: System::setSecurityManager will be removed in a future release [2023-10-02T09:39:54,169][INFO][o.o.n.Node] [Data-Center] version[2.8.0], pid[7000], build[zip/db90a415ff2fd428b4f7b3f800a51dc229287cb4/2023-06-03T06:29:28.546587700Z], OS[Windows Server 2012 R2/6.3/amd64], JVM[Oracle Corporation/Java HotSpot™ 64-Bit Server VM/19/19+36-2238] [2023-10-02T09:39:54,173][INFO][o.o.n.Node] [Data-Center] JVM home [C:\Program Files\Java\jdk-19], using bundled JDK [false] [2023-10-02T09:39:54,174][INFO][o.o.n.Node] [Data-Center] JVM arguments [-Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms96g, -Xmx96g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\2\opensearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs\_err\_pid%p.log, -Xlog:gc\*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -XX:MaxDirectMemorySize=51539607552, -Dopensearch, -Dopensearch.path.home=F:\fess\_cluster\opensearch-2.8.0, -Dopensearch.path.conf=F:\fess\_cluster\opensearch-2.8.0\config, -Dopensearch.distribution.type=zip, -Dopensearch. bundled\_jdk=true] [2023-10-02T09:39:55,390][INFO][o.o.s.s.t.SSLConfig] [Data-Center] SSL dual mode is disabled [2023-10-02T09:39:55,392][WARN][o.o.s.OpenSearchSecurityPlugin] [Data-Center] OpenSearch Security plugin installed but disabled. This can expose your configuration (including passwords) to the public. [2023-10-02T09:39:56,237][INFO][o.o.i.r.ReindexPlugin] [Data-Center] ReindexPlugin reloadSPI called [2023-10-02T09:39:56,240][INFO][o.o.i.r.ReindexPlugin] [Data-Center] Unable to find any implementation for RemoteReindexExtension [2023-10-02T09:39:56,284][INFO][o.o.j.JobSchedulerPlugin] [Data-Center] Loaded scheduler extension: opendistro\_anomaly\_detector, index: .opendistro-anomaly-detector-jobs [2023-10-02T09:39:56,323][INFO][o.o.j.JobSchedulerPlugin] [Data-Center] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions [2023-10-02T09:39:56,326][INFO][o.o.j.JobSchedulerPlugin] [Data-Center] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config [2023-10-02T09:39:56,411][INFO][o.o.j.JobSchedulerPlugin] [Data-Center] Loaded scheduler extension: observability, index: .opensearch-observability-job [2023-10-02T09:39:56,421][INFO][o.o.p.PluginsService] [Data-Center] loaded module [aggs-matrix-stats] [2023-10-02T09:39:56,422][INFO][o.o.p.PluginsService] [Data-Center] loaded module [analysis-common] [2023-10-02T09:39:56,424][INFO][o.o.p.PluginsService] [Data-Center] loaded module [geo] [2023-10-02T09:39:56,425][INFO][o.o.p.PluginsService] [Data-Center] loaded module [ingest-common] [2023-10-02T09:39:56,426][INFO][o.o.p.PluginsService] [Data-Center] loaded module [ingest-geoip] [2023-10-02T09:39:56,427][INFO][o.o.p.PluginsService] [Data-Center] loaded module [ingest-user-agent] [2023-10-02T09:39:56,428][INFO][o.o.p.PluginsService] [Data-Center] loaded module [lang-expression] [2023-10-02T09:39:56,429][INFO][o.o.p.PluginsService] [Data-Center] loaded module [lang-mustache] [2023-10-02T09:39:56,430][INFO][o.o.p.PluginsService] [Data-Center] loaded module [lang-painless] [2023-10-02T09:39:56,431][INFO][o.o.p.PluginsService] [Data-Center] loaded module [mapper-extras] [2023-10-02T09:39:56,432][INFO][o.o.p.PluginsService] [Data-Center] loaded module [opensearch-dashboards] [2023-10-02T09:39:56,433][INFO][o.o.p.PluginsService] [Data-Center] loaded module [parent-join] [2023-10-02T09:39:56,434][INFO][o.o.p.PluginsService] [Data-Center] loaded module [percolator] [2023-10-02T09:39:56,435][INFO][o.o.p.PluginsService] [Data-Center] loaded module [rank-eval] [2023-10-02T09:39:56,436][INFO][o.o.p.PluginsService] [Data-Center] loaded module [reindex] [2023-10-02T09:39:56,437][INFO][o.o.p.PluginsService] [Data-Center] loaded module [repository-url] [2023-10-02T09:39:56,438][INFO][o.o.p.PluginsService] [Data-Center] loaded module [search-pipeline-common] [2023-10-02T09:39:56,440][INFO][o.o.p.PluginsService] [Data-Center] loaded module [transport-netty4] [2023-10-02T09:39:56,442][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [analysis-extension] [2023-10-02T09:39:56,443][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [analysis-fess] [2023-10-02T09:39:56,444][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [configsync] [2023-10-02T09:39:56,445][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [minhash] [2023-10-02T09:39:56,446][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-alerting] [2023-10-02T09:39:56,447][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-anomaly-detection] [2023-10-02T09:39:56,449][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-asynchronous-search] [2023-10-02T09:39:56,450][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-cross-cluster-replication] [2023-10-02T09:39:56,451][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-geospatial] [2023-10-02T09:39:56,452][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-index-management] [2023-10-02T09:39:56,453][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-job-scheduler] [2023-10-02T09:39:56,454][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-knn] [2023-10-02T09:39:56,455][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-ml] [2023-10-02T09:39:56,456][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-neural-search] [2023-10-02T09:39:56,457][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-notifications] [2023-10-02T09:39:56,458][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-notifications-core] [2023-10-02T09:39:56,459][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-observability] [2023-10-02T09:39:56,460][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-reports-scheduler] [2023-10-02T09:39:56,462][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-security] [2023-10-02T09:39:56,463][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-security-analytics] [2023-10-02T09:39:56,464][INFO][o.o.p.PluginsService] [Data-Center] loaded plugin [opensearch-sql] [2023-10-02T09:39:56,529][INFO][o.o.e.ExtensionsManager] [Data-Center] ExtensionsManager initialized [2023-10-02T09:39:56,544][INFO][o.a.l.s.MemorySegmentIndexInputProvider] [Data-Center] Using MemorySegmentIndexInput with Java 19; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false [2023-10-02T09:39:56,557][INFO][o.o.e.NodeEnvironment] [Data-Center] using [1] data paths, mounts [[繝懊Μ繝･繝ｼ繝 (F:)]], net usable\_space [937.4gb], net total\_space [1023.9gb], types [NTFS] [2023-10-02T09:39:56,563][INFO][o.o.e.NodeEnvironment] [Data-Center] heap size [96gb], compressed ordinary object pointers [false] [2023-10-02T09:39:56,628][INFO][o.o.n.Node] [Data-Center] node name [Data-Center], node ID [2piMCxDbToSoDWsj2QpncA], cluster name [cluster-tester], roles [cluster\_manager] [2023-10-02T09:40:00,434][INFO][o.o.a.b.ADCircuitBreakerService] [Data-Center] Registered memory breaker. [2023-10-02T09:40:00,883][INFO][o.o.m.b.MLCircuitBreakerService] [Data-Center] Registered ML memory breaker. [2023-10-02T09:40:00,885][INFO][o.o.m.b.MLCircuitBreakerService] [Data-Center] Registered ML disk breaker. [2023-10-02T09:40:00,887][INFO][o.o.m.b.MLCircuitBreakerService] [Data-Center] Registered ML native memory breaker. [2023-10-02T09:40:01,009][INFO][o.r.Reflections] [Data-Center] Reflections took 49 ms to scan 1 urls, producing 15 keys and 37 values [2023-10-02T09:40:01,938][INFO][o.o.t.NettyAllocator] [Data-Center] creating NettyAllocator with the following configs: [name=opensearch\_configured, chunk\_size=1mb, suggested\_max\_allocation\_size=1mb, factors={opensearch.unsafe.use\_netty\_default\_chunk\_and\_page\_size=false, g1gc\_enabled=true, g1gc\_region\_size=32mb}] [2023-10-02T09:40:02,020][INFO][o.o.d.DiscoveryModule] [Data-Center] using discovery type [zen] and seed hosts providers [settings] [2023-10-02T09:40:02,544][WARN][o.o.g.DanglingIndicesState] [Data-Center] gateway.auto\_import\_dangling\_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2023-10-02T09:40:03,289][INFO][o.o.n.Node] [Data-Center] initialized [2023-10-02T09:40:03,290][INFO][o.o.n.Node] [Data-Center] starting … [2023-10-02T09:40:03,427][INFO][o.o.t.TransportService] [Data-Center] publish\_address {192.168.10.225:9300}, bound\_addresses {192.168.10.225:9300} [2023-10-02T09:40:03,596][INFO][o.o.b.BootstrapChecks] [Data-Center] bound or publishing to a non-loopback address, enforcing bootstrap checks [2023-10-02T09:40:03,624][INFO][o.o.c.c.Coordinator] [Data-Center] setting initial configuration to VotingConfiguration{2piMCxDbToSoDWsj2QpncA} [2023-10-02T09:40:03,745][INFO][o.o.c.s.MasterService] [Data-Center] elected-as-cluster-manager ([1] nodes joined)[{Data-Center}{2piMCxDbToSoDWsj2QpncA}{CIq7xakKT\_eGfWuc38rbJQ}{192.168.10.225}{192.168.10.225:9300}{m}{shard\_indexing\_pressure\_enabled=true} elect leader, _BECOME\_CLUSTER\_MANAGER\_TASK_, _FINISH\_ELECTION_], term: 1, version: 1, delta: cluster-manager node changed {previous , current [{Data-Center}{2piMCxDbToSoDWsj2QpncA}{CIq7xakKT\_eGfWuc38rbJQ}{192.168.10.225}{192.168.10.225:9300}{m}{shard\_indexing\_pressure\_enabled=true}]} [2023-10-02T09:40:03,793][INFO][o.o.c.c.CoordinationState] [Data-Center] cluster UUID set to [p2inlSMCR0ScARpimD5sLg] [2023-10-02T09:40:03,830][INFO][o.o.c.s.ClusterApplierService] [Data-Center] cluster-manager node changed {previous , current [{Data-Center}{2piMCxDbToSoDWsj2QpncA}{CIq7xakKT\_eGfWuc38rbJQ}{192.168.10.225}{192.168.10.225:9300}{m}{shard\_indexing\_pressure\_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1} [2023-10-02T09:40:03,839][INFO][o.o.a.c.ADClusterEventListener] [Data-Center] Cluster is not recovered yet. [2023-10-02T09:40:03,847][INFO][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration [2023-10-02T09:40:03,870][INFO][o.o.i.i.ManagedIndexCoordinator] [Data-Center] Cache cluster manager node onClusterManager time: 1696207203869 [2023-10-02T09:40:03,891][INFO][o.o.d.PeerFinder] [Data-Center] setting findPeersInterval to [1s] as node commission status = [true] for local node [{Data-Center}{2piMCxDbToSoDWsj2QpncA}{CIq7xakKT\_eGfWuc38rbJQ}{192.168.10.225}{192.168.10.225:9300}{m}{shard\_indexing\_pressure\_enabled=true}] [2023-10-02T09:40:03,895][INFO][o.o.h.AbstractHttpServerTransport] [Data-Center] publish\_address {192.168.10.225:9200}, bound\_addresses {192.168.10.225:9200} [2023-10-02T09:40:03,896][INFO][o.o.n.Node] [Data-Center] started [2023-10-02T09:40:03,897][INFO][o.o.s.OpenSearchSecurityPlugin] [Data-Center] Node started [2023-10-02T09:40:03,899][INFO][o.o.s.OpenSearchSecurityPlugin] [Data-Center] 0 OpenSearch Security modules loaded so far: [2023-10-02T09:40:03,906][INFO][o.o.a.c.HashRing] [Data-Center] Node added: [2piMCxDbToSoDWsj2QpncA] [2023-10-02T09:40:03,911][INFO][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration [2023-10-02T09:40:03,912][INFO][o.o.a.c.HashRing] [Data-Center] All nodes with known AD version: {2piMCxDbToSoDWsj2QpncA=ADNodeInfo{version=2.8.0, isEligibleDataNode=false}} [2023-10-02T09:40:03,914][INFO][o.o.a.c.HashRing] [Data-Center] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0 [2023-10-02T09:40:03,915][INFO][o.o.a.c.HashRing] [Data-Center] Build AD version hash ring successfully [2023-10-02T09:40:03,916][INFO][o.o.a.c.ADClusterEventListener] [Data-Center] Init AD version hash ring successfully [2023-10-02T09:40:03,938][INFO][o.o.g.GatewayService] [Data-Center] recovered [0] indices into cluster\_state [2023-10-02T09:40:03,962][INFO][o.o.p.PluginsService] [Data-Center] PluginService:onIndexModule index:[.opensearch-observability/wC2BZrlwR5K8Ix3FhDc0Fw] [2023-10-02T09:40:04,200][INFO][o.o.c.m.MetadataCreateIndexService] [Data-Center] [.opensearch-observability] creating index, cause [api], templates , shards [1]/[0] [2023-10-02T09:40:04,213][INFO][o.o.c.r.a.AllocationService] [Data-Center] Cluster health status changed from [YELLOW] to [RED] (reason: [index [.opensearch-observability] created]). [2023-10-02T09:40:04,257][INFO][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration [2023-10-02T09:40:04,263][INFO][o.o.p.PluginsService] [Data-Center] PluginService:onIndexModule index:[configsync/wh7N0siISpa\_97AvFGjAEA] [2023-10-02T09:40:04,298][INFO][o.o.c.m.MetadataCreateIndexService] [Data-Center] [configsync] creating index, cause [api], templates , shards [1]/[0] [2023-10-02T09:40:04,331][INFO][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration [2023-10-02T09:40:34,263][INFO][o.o.o.i.ObservabilityIndex] [Data-Center] observability:Index .opensearch-observability creation Acknowledged [2023-10-02T09:40:34,265][INFO][o.o.o.i.ObservabilityTracesIndex] [Data-Center] observability:createMappingTemplate ss4o\_trace\_template API called [2023-10-02T09:40:34,279][INFO][o.o.p.PluginsService] [Data-Center] PluginService:onIndexModule index:[validate-template-cvszzmtmtykrtblkdmmldg/4drnWvAbTxqcna5TmNvLcA] [2023-10-02T09:40:34,325][INFO][o.o.c.m.MetadataIndexTemplateService] [Data-Center] adding index template [ss4o\_trace\_template] for index patterns [ss4o\_traces-_-_] [2023-10-02T09:40:34,361][INFO][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration [2023-10-02T09:40:34,363][INFO][o.o.o.i.ObservabilityTracesIndex] [Data-Center] observability:Mapping Template ss4o\_trace\_template creation Acknowledged [2023-10-02T09:40:34,364][INFO][o.o.o.i.ObservabilityMetricsIndex] [Data-Center] observability:createMappingTemplate ss4o\_metric\_template API called [2023-10-02T09:40:34,370][INFO][o.o.p.PluginsService] [Data-Center] PluginService:onIndexModule index:[validate-template-hxgme5vorpivv8wf62e4pq/7-XLbscQToe4cTrq26tTaw] [2023-10-02T09:40:34,408][INFO][o.o.c.m.MetadataIndexTemplateService] [Data-Center] adding index template [ss4o\_metric\_template] for index patterns [ss4o\_metrics-_-_] [2023-10-02T09:40:34,445][INFO][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration [2023-10-02T09:40:34,447][INFO][o.o.o.i.ObservabilityMetricsIndex] [Data-Center] observability:Mapping Template ss4o\_metric\_template creation Acknowledged [2023-10-02T09:41:03,875][INFO][o.o.i.i.ManagedIndexCoordinator] [Data-Center] Performing move cluster state metadata. [2023-10-02T09:41:03,877][INFO][o.o.i.i.MetadataService] [Data-Center] ISM config index not exist, so we cancel the metadata migration job. [2023-10-02T09:41:03,878][INFO][o.o.i.i.ManagedIndexCoordinator] [Data-Center] Performing ISM template migration. [2023-10-02T09:41:03,880][INFO][o.o.i.i.m.ISMTemplateService] [Data-Center] Doing ISM template migration 1 time. [2023-10-02T09:41:03,881][INFO][o.o.i.i.m.ISMTemplateService] [Data-Center] Use 2023-10-01T23:40:03.869Z as migrating ISM template last\_updated\_time [2023-10-02T09:41:03,886][INFO][o.o.p.PluginsService] [Data-Center] PluginService:onIndexModule index:[simulate\_template\_index\_ljihwhiasrahi4uv1ct\_gq/hnnZnH6iQBWZezm-dzbqig] [2023-10-02T09:41:03,914][INFO][o.o.p.PluginsService] [Data-Center] PluginService:onIndexModule index:[simulate\_template\_index\_ljihwhiasrahi4uv1ct\_gq/hnnZnH6iQBWZezm-dzbqig] [2023-10-02T09:41:03,950][INFO][o.o.p.PluginsService] [Data-Center] PluginService:onIndexModule index:[simulate\_template\_index\_k-6kn0fhqou8aysqgoyvfq/hvIJQtjZSe-dR0oRiA9C2g] [2023-10-02T09:41:03,979][INFO][o.o.p.PluginsService] [Data-Center] PluginService:onIndexModule index:[simulate\_template\_index\_k-6kn0fhqou8aysqgoyvfq/hvIJQtjZSe-dR0oRiA9C2g] [2023-10-02T09:41:04,012][INFO][o.o.i.i.m.ISMTemplateService] [Data-Center] ISM templates: {=[ISMTemplate(indexPatterns=[ss4o\_traces-_-_], priority=1, lastUpdatedTime=2023-10-01T23:40:03.869Z), ISMTemplate(indexPatterns=[ss4o\_metrics-_-_], priority=1, lastUpdatedTime=2023-10-01T23:40:03.869Z)]} [2023-10-02T09:41:04,017][INFO][o.o.i.i.m.ISMTemplateService] [Data-Center] Policies to update: [2023-10-02T09:41:04,021][INFO][o.o.i.i.m.ISMTemplateService] [Data-Center] Failure experienced when migrating ISM Template and update ISM policies: {} [2023-10-02T09:41:04,082][INFO][o.o.c.s.ClusterSettings] [Data-Center] updating [plugins.index\_state\_management.template\_migration.control] from [0] to [-1] [2023-10-02T09:41:04,084][INFO][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration [2023-10-02T09:41:04,088][INFO][o.o.i.i.m.ISMTemplateService] [Data-Center] Successfully update template migration setting [2023-10-02T09:41:04,338][INFO][o.c.o.c.s.ConfigSyncService] [Data-Center] ConfigFileUpdater is started at 1m intervals. [2023-10-02T09:42:03,876][INFO][o.o.i.i.ManagedIndexCoordinator] [Data-Center] Cancel background move metadata process. [2023-10-02T09:42:03,877][INFO][o.o.i.i.ManagedIndexCoordinator] [Data-Center] Performing move cluster state metadata. [2023-10-02T09:42:03,878][INFO][o.o.i.i.MetadataService] [Data-Center] Move metadata has finished. [2023-10-02T09:42:04,365][ERROR][o.c.o.c.s.ConfigSyncService] [Data-Center] Failed to process ConfigFileUpdater. org.opensearch.action.search.SearchPhaseExecutionException: all shards failed at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:664) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:372) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:699) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:472) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:273) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:350) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] at java.lang.Thread.run(Thread.java:1589) [?:?] [2023-10-02T09:43:04,382][ERROR][o.c.o.c.s.ConfigSyncService] [Data-Center] Failed to process ConfigFileUpdater. _ **org.opensearch.action.search.SearchPhaseExecutionException: all shards failed** _ at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:664) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:372) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:699) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:472) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:273) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:350) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.8.0.jar:2.8.0] at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] at java.lang.Thread.run(Thread.java:1589) [?:?] [2023-10-02T09:44:04,396][ERROR][o.c.o.c.s.ConfigSyncService] [Data-Center] Failed to process ConfigFileUpdater. org.opensearch.action.search.SearchPhaseExecutionException: all shards failed ========================================================= From the error: “org.opensearch.action.search.SearchPhaseExecutionException: all shards failed” that my opensearch can not create initialized shards, however, I don’t know how to do next. Any suggestion is very appreciated. ---
**Author:** ![Gsmitt](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/gsmitt/32/1718_2.png) [@Gsmitt](https://forum.opensearch.org/u/Gsmitt)\ **Post date:** [October 4, 2023, 12:27am UTC](https://forum.opensearch.org/t/opensearch-launch-failed-on-cluster-manager/16096/2 "2023-10-04T00:27:47Z")
Hey @Di_Di Is it possible to make your Openearch config file more readable? Example: ```auto root@ansible:/var/log/opensearch# cat /etc/opensearch/opensearch.yml # ======================== OpenSearch Configuration ========================= # # NOTE: OpenSearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.opensearch.org # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: openlab # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # #node.name: node-1 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /var/lib/opensearch ``` ---
**Author:** ![Di\_Di](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/di_di/32/6127_2.png) [@Di\_Di](https://forum.opensearch.org/u/Di_Di)\ **Post date:** [October 4, 2023, 3:03am UTC](https://forum.opensearch.org/t/opensearch-launch-failed-on-cluster-manager/16096/5 "2023-10-04T03:03:21Z")
Sorry for your inconvenience. Finnlly I figure out why my normal text post, but showing is so terrible. Here is my new post, hope you can read it clearly. ================= =========== Start ==================================== #- #- Please consult the documentation for further information on configuration options: #- [https://www.opensearch.org](https://www.opensearch.org) #- #- ---------------------------------- Cluster ----------------------------------- #- #- Use a descriptive name for your cluster: #- #-cluster.name: my-application **cluster.name: cluster-tester** #- #- ------------------------------------ Node ------------------------------------ #- #- Use a descriptive name for the node: #- **node.name: Data-Center** **node.roles: [cluster\_manager]** #- #- Add custom attributes to the node: #- #-node.attr.rack: r1 #- #- ----------------------------------- Paths ------------------------------------ #- #- Path to directory where to store the data (separate multiple locations by comma): #- #-path.data: /path/to/data #- #- Path to log files: #- #-path.logs: /path/to/logs #- #- ----------------------------------- Memory ----------------------------------- #- #- Lock the memory on startup: #- #-bootstrap.memory\_lock: true #- #- Make sure that the heap size is set to about half the memory available #- on the system and that the owner of the process is allowed to use this #- limit. #- #- OpenSearch performs poorly when the system is swapping the memory. #- #- ---------------------------------- Network ----------------------------------- #- #- Set the bind address to a specific IP (IPv4 or IPv6): #- #-network.host: 192.168.0.1 **network.host: 192.168.10.225** #- #- Set a custom port for HTTP: #- http.port: 9200 #- #- For more information, consult the network module documentation. #- #- --------------------------------- Discovery ---------------------------------- #- #- Pass an initial list of hosts to perform discovery when this node is started: #- The default list of hosts is [“127.0.0.1”, “[::1]”] #- #-discovery.seed\_hosts: [“host1”, “host2”] **discovery.seed\_hosts: [“192.168.10.225”,“192.168.10.226”]** #- #- Bootstrap the cluster using an initial set of cluster-manager-eligible nodes: #- #-cluster.initial\_cluster\_manager\_nodes: [“node-1”, “node-2”] **cluster.initial\_cluster\_manager\_nodes: [“192.168.10.225”]** #- #- For more information, consult the discovery and cluster formation module documentation. #- #- ---------------------------------- Gateway ----------------------------------- #- #- Block initial recovery after a full cluster restart until N nodes are started: #- #-gateway.recover\_after\_nodes: 3 #- #- For more information, consult the gateway module documentation. #- #- ---------------------------------- Various ----------------------------------- #- #- Require explicit names when deleting indices: #- #-action.destructive\_requires\_name: true #- #- ---------------------------------- Remote Store ----------------------------------- #- Controls whether cluster imposes index creation only with remote store enabled #- cluster.remote\_store.enabled: true #- #- Repository to use for segment upload while enforcing remote store for an index #- cluster.remote\_store.repository: my-repo-1 #- #- Controls whether cluster imposes index creation only with translog remote store enabled #- cluster.remote\_store.translog.enabled: true #- #- Repository to use for translog upload while enforcing remote store for an index #- cluster.remote\_store.translog.repository: my-repo-1 #- #- ---------------------------------- Experimental Features ----------------------------------- #- #- Gates the visibility of the experimental segment replication features until they are production ready. #- #-opensearch.experimental.feature.segment\_replication\_experimental.enabled: false #- #- #- Gates the visibility of the index setting that allows persisting data to remote store along with local disk. #- Once the feature is ready for production release, this feature flag can be removed. #- #-opensearch.experimental.feature.remote\_store.enabled: false #- #- #- Gates the functionality of a new parameter to the snapshot restore API #- that allows for creation of a new index type that searches a snapshot #- directly in a remote repository without restoring all index data to disk #- ahead of time. #- #-opensearch.experimental.feature.searchable\_snapshot.enabled: false #- #- #- Gates the functionality of enabling extensions to work with OpenSearch. #- This feature enables applications to extend features of OpenSearch outside of #- the core. #- #-opensearch.experimental.feature.extensions.enabled: false #- #- #- Gates the search pipeline feature. This feature enables configurable processors #- for search requests and search responses, similar to ingest pipelines. #-opensearch.experimental.feature.search\_pipeline.enabled: false #- Added configsync.config\_path: F:/fess\_cluster/opensearch-2.8.0/data/config/ plugins.security.disabled: true ================= =========== END ==================================== ---
**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\ **Post date:** [October 4, 2023, 7:04pm UTC](https://forum.opensearch.org/t/opensearch-launch-failed-on-cluster-manager/16096/6 "2023-10-04T19:04:51Z")
@Di_Di The OpenSearch node with only the cluster\_manager node role can’t hold any data (indices). It requires a data node role to store the data. I’d suggest starting a master node with a data node. ---
**Author:** ![Di\_Di](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/di_di/32/6127_2.png) [@Di\_Di](https://forum.opensearch.org/u/Di_Di)\ **Post date:** [October 4, 2023, 11:43pm UTC](https://forum.opensearch.org/t/opensearch-launch-failed-on-cluster-manager/16096/7 "2023-10-04T23:43:03Z")
That’s right! My opensearch cluster manager node startup succeed. Thank for your help.