Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Describe the issue:
I’m trying to create an opensearch cluster, in which contains
a “cluster manager” node, and tothers data nodes.
But when I tried to start opensearch on the cluster manager node which is the first node to start in my cluster, but start-up process aborted with errors , the main error is as below:
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
However, if I removed “node.roles: [ cluster_manager ]” setting, opensearch starting process succeed without any problem.
Configuration:
Following is my opensearch.yml:
======================== OpenSearch Configuration =========================
NOTE: OpenSearch comes with reasonable defaults for most settings.
Before you set out to tweak and tune the configuration, make sure you
understand what are you trying to accomplish and the consequences.
The primary way of configuring a node is via this file. This template lists
the most important settings you may want to configure for a production cluster.
Please consult the documentation for further information on configuration options:
https://www.opensearch.org
---------------------------------- Cluster -----------------------------------
Use a descriptive name for your cluster:
#cluster.name: my-application
cluster.name: cluster-tester
------------------------------------ Node ------------------------------------
Use a descriptive name for the node:
#node.name: node-1
node.name: Data-Center
node.roles: [ cluster_manager ]
Add custom attributes to the node:
#node.attr.rack: r1
----------------------------------- Paths ------------------------------------
Path to directory where to store the data (separate multiple locations by comma):
#path.data: /path/to/data
Path to log files:
#path.logs: /path/to/logs
----------------------------------- Memory -----------------------------------
Lock the memory on startup:
#bootstrap.memory_lock: true
Make sure that the heap size is set to about half the memory available
on the system and that the owner of the process is allowed to use this
limit.
OpenSearch performs poorly when the system is swapping the memory.
---------------------------------- Network -----------------------------------
Set the bind address to a specific IP (IPv4 or IPv6):
#network.host: 192.168.0.1
network.host: 192.168.10.225
Set a custom port for HTTP:
http.port: 9200
For more information, consult the network module documentation.
--------------------------------- Discovery ----------------------------------
Pass an initial list of hosts to perform discovery when this node is started:
The default list of hosts is [“127.0.0.1”, “[::1]”]
#discovery.seed_hosts: [“host1”, “host2”]
discovery.seed_hosts: [“192.168.10.225”,“192.168.10.226”]
Bootstrap the cluster using an initial set of cluster-manager-eligible nodes:
#cluster.initial_cluster_manager_nodes: [“node-1”, “node-2”]
cluster.initial_cluster_manager_nodes: [“192.168.10.225”]
For more information, consult the discovery and cluster formation module documentation.
---------------------------------- Gateway -----------------------------------
Block initial recovery after a full cluster restart until N nodes are started:
#gateway.recover_after_nodes: 3
For more information, consult the gateway module documentation.
---------------------------------- Various -----------------------------------
Require explicit names when deleting indices:
#action.destructive_requires_name: true
---------------------------------- Remote Store -----------------------------------
Controls whether cluster imposes index creation only with remote store enabled
cluster.remote_store.enabled: true
Repository to use for segment upload while enforcing remote store for an index
cluster.remote_store.repository: my-repo-1
Controls whether cluster imposes index creation only with translog remote store enabled
cluster.remote_store.translog.enabled: true
Repository to use for translog upload while enforcing remote store for an index
cluster.remote_store.translog.repository: my-repo-1
---------------------------------- Experimental Features -----------------------------------
Gates the visibility of the experimental segment replication features until they are production ready.
OpenSearch.experimental.feature.segment_replication_experimental.enabled: false
Gates the visibility of the index setting that allows persisting data to remote store along with local disk.
Once the feature is ready for production release, this feature flag can be removed.
OpenSearch.experimental.feature.remote_store.enabled: false
Gates the functionality of a new parameter to the snapshot restore API
that allows for creation of a new index type that searches a snapshot
directly in a remote repository without restoring all index data to disk
ahead of time.
OpenSearch.experimental.feature.searchable_snapshot.enabled: false
Gates the functionality of enabling extensions to work with OpenSearch.
This feature enables applications to extend features of OpenSearch outside of
the core.
OpenSearch.experimental.feature.extensions.enabled: false
Gates the search pipeline feature. This feature enables configurable processors
for search requests and search responses, similar to ingest pipelines.
OpenSearch.experimental.feature.search_pipeline.enabled: false
Added by Yan
configsync.config_path: F:/fess_cluster/opensearch-2.8.0/data/config/
plugins.security.disabled: true
Relevant Logs or Screenshots:
Below is all output info: in my console:
PS F:\fess_cluster\opensearch-2.8.0\bin> .\opensearch.bat
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/F:/fess_cluster/opensearch-2.8.0/lib/opensearch-2.8.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/F:/fess_cluster/opensearch-2.8.0/lib/opensearch-2.8.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2023-10-02T09:39:54,169][INFO ][o.o.n.Node ] [Data-Center] version[2.8.0], pid[7000], build[zip/db90a415ff2fd428b4f7b3f800a51dc229287cb4/2023-06-03T06:29:28.546587700Z], OS[Windows Server 2012 R2/6.3/amd64], JVM[Oracle Corporation/Java HotSpot™ 64-Bit Server VM/19/19+36-2238]
[2023-10-02T09:39:54,173][INFO ][o.o.n.Node ] [Data-Center] JVM home [C:\Program Files\Java\jdk-19], using bundled JDK [false]
[2023-10-02T09:39:54,174][INFO ][o.o.n.Node ] [Data-Center] JVM arguments [-Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms96g, -Xmx96g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\2\opensearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -XX:MaxDirectMemorySize=51539607552, -Dopensearch, -Dopensearch.path.home=F:\fess_cluster\opensearch-2.8.0, -Dopensearch.path.conf=F:\fess_cluster\opensearch-2.8.0\config, -Dopensearch.distribution.type=zip, -Dopensearch.
bundled_jdk=true]
[2023-10-02T09:39:55,390][INFO ][o.o.s.s.t.SSLConfig ] [Data-Center] SSL dual mode is disabled
[2023-10-02T09:39:55,392][WARN ][o.o.s.OpenSearchSecurityPlugin] [Data-Center] OpenSearch Security plugin installed but disabled. This can expose your configuration (including passwords) to the public.
[2023-10-02T09:39:56,237][INFO ][o.o.i.r.ReindexPlugin ] [Data-Center] ReindexPlugin reloadSPI called
[2023-10-02T09:39:56,240][INFO ][o.o.i.r.ReindexPlugin ] [Data-Center] Unable to find any implementation for RemoteReindexExtension
[2023-10-02T09:39:56,284][INFO ][o.o.j.JobSchedulerPlugin ] [Data-Center] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2023-10-02T09:39:56,323][INFO ][o.o.j.JobSchedulerPlugin ] [Data-Center] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2023-10-02T09:39:56,326][INFO ][o.o.j.JobSchedulerPlugin ] [Data-Center] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2023-10-02T09:39:56,411][INFO ][o.o.j.JobSchedulerPlugin ] [Data-Center] Loaded scheduler extension: observability, index: .opensearch-observability-job
[2023-10-02T09:39:56,421][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [aggs-matrix-stats]
[2023-10-02T09:39:56,422][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [analysis-common]
[2023-10-02T09:39:56,424][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [geo]
[2023-10-02T09:39:56,425][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [ingest-common]
[2023-10-02T09:39:56,426][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [ingest-geoip]
[2023-10-02T09:39:56,427][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [ingest-user-agent]
[2023-10-02T09:39:56,428][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [lang-expression]
[2023-10-02T09:39:56,429][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [lang-mustache]
[2023-10-02T09:39:56,430][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [lang-painless]
[2023-10-02T09:39:56,431][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [mapper-extras]
[2023-10-02T09:39:56,432][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [opensearch-dashboards]
[2023-10-02T09:39:56,433][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [parent-join]
[2023-10-02T09:39:56,434][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [percolator]
[2023-10-02T09:39:56,435][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [rank-eval]
[2023-10-02T09:39:56,436][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [reindex]
[2023-10-02T09:39:56,437][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [repository-url]
[2023-10-02T09:39:56,438][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [search-pipeline-common]
[2023-10-02T09:39:56,440][INFO ][o.o.p.PluginsService ] [Data-Center] loaded module [transport-netty4]
[2023-10-02T09:39:56,442][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [analysis-extension]
[2023-10-02T09:39:56,443][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [analysis-fess]
[2023-10-02T09:39:56,444][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [configsync]
[2023-10-02T09:39:56,445][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [minhash]
[2023-10-02T09:39:56,446][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-alerting]
[2023-10-02T09:39:56,447][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-anomaly-detection]
[2023-10-02T09:39:56,449][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-asynchronous-search]
[2023-10-02T09:39:56,450][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-cross-cluster-replication]
[2023-10-02T09:39:56,451][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-geospatial]
[2023-10-02T09:39:56,452][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-index-management]
[2023-10-02T09:39:56,453][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-job-scheduler]
[2023-10-02T09:39:56,454][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-knn]
[2023-10-02T09:39:56,455][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-ml]
[2023-10-02T09:39:56,456][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-neural-search]
[2023-10-02T09:39:56,457][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-notifications]
[2023-10-02T09:39:56,458][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-notifications-core]
[2023-10-02T09:39:56,459][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-observability]
[2023-10-02T09:39:56,460][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-reports-scheduler]
[2023-10-02T09:39:56,462][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-security]
[2023-10-02T09:39:56,463][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-security-analytics]
[2023-10-02T09:39:56,464][INFO ][o.o.p.PluginsService ] [Data-Center] loaded plugin [opensearch-sql]
[2023-10-02T09:39:56,529][INFO ][o.o.e.ExtensionsManager ] [Data-Center] ExtensionsManager initialized
[2023-10-02T09:39:56,544][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] [Data-Center] Using MemorySegmentIndexInput with Java 19; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
[2023-10-02T09:39:56,557][INFO ][o.o.e.NodeEnvironment ] [Data-Center] using [1] data paths, mounts [[繝懊Μ繝･繝ｼ繝 (F:)]], net usable_space [937.4gb], net total_space [1023.9gb], types [NTFS]
[2023-10-02T09:39:56,563][INFO ][o.o.e.NodeEnvironment ] [Data-Center] heap size [96gb], compressed ordinary object pointers [false]
[2023-10-02T09:39:56,628][INFO ][o.o.n.Node ] [Data-Center] node name [Data-Center], node ID [2piMCxDbToSoDWsj2QpncA], cluster name [cluster-tester], roles [cluster_manager]
[2023-10-02T09:40:00,434][INFO ][o.o.a.b.ADCircuitBreakerService] [Data-Center] Registered memory breaker.
[2023-10-02T09:40:00,883][INFO ][o.o.m.b.MLCircuitBreakerService] [Data-Center] Registered ML memory breaker.
[2023-10-02T09:40:00,885][INFO ][o.o.m.b.MLCircuitBreakerService] [Data-Center] Registered ML disk breaker.
[2023-10-02T09:40:00,887][INFO ][o.o.m.b.MLCircuitBreakerService] [Data-Center] Registered ML native memory breaker.
[2023-10-02T09:40:01,009][INFO ][o.r.Reflections ] [Data-Center] Reflections took 49 ms to scan 1 urls, producing 15 keys and 37 values
[2023-10-02T09:40:01,938][INFO ][o.o.t.NettyAllocator ] [Data-Center] creating NettyAllocator with the following configs: [name=opensearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={opensearch.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=32mb}]
[2023-10-02T09:40:02,020][INFO ][o.o.d.DiscoveryModule ] [Data-Center] using discovery type [zen] and seed hosts providers [settings]
[2023-10-02T09:40:02,544][WARN ][o.o.g.DanglingIndicesState] [Data-Center] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-10-02T09:40:03,289][INFO ][o.o.n.Node ] [Data-Center] initialized
[2023-10-02T09:40:03,290][INFO ][o.o.n.Node ] [Data-Center] starting …
[2023-10-02T09:40:03,427][INFO ][o.o.t.TransportService ] [Data-Center] publish_address {192.168.10.225:9300}, bound_addresses {192.168.10.225:9300}
[2023-10-02T09:40:03,596][INFO ][o.o.b.BootstrapChecks ] [Data-Center] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-10-02T09:40:03,624][INFO ][o.o.c.c.Coordinator ] [Data-Center] setting initial configuration to VotingConfiguration{2piMCxDbToSoDWsj2QpncA}
[2023-10-02T09:40:03,745][INFO ][o.o.c.s.MasterService ] [Data-Center] elected-as-cluster-manager ([1] nodes joined)[{Data-Center}{2piMCxDbToSoDWsj2QpncA}{CIq7xakKT_eGfWuc38rbJQ}{192.168.10.225}{192.168.10.225:9300}{m}{shard_indexing_pressure_enabled=true} elect leader, BECOME_CLUSTER_MANAGER_TASK, FINISH_ELECTION], term: 1, version: 1, delta: cluster-manager node changed {previous , current [{Data-Center}{2piMCxDbToSoDWsj2QpncA}{CIq7xakKT_eGfWuc38rbJQ}{192.168.10.225}{192.168.10.225:9300}{m}{shard_indexing_pressure_enabled=true}]}
[2023-10-02T09:40:03,793][INFO ][o.o.c.c.CoordinationState] [Data-Center] cluster UUID set to [p2inlSMCR0ScARpimD5sLg]
[2023-10-02T09:40:03,830][INFO ][o.o.c.s.ClusterApplierService] [Data-Center] cluster-manager node changed {previous , current [{Data-Center}{2piMCxDbToSoDWsj2QpncA}{CIq7xakKT_eGfWuc38rbJQ}{192.168.10.225}{192.168.10.225:9300}{m}{shard_indexing_pressure_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2023-10-02T09:40:03,839][INFO ][o.o.a.c.ADClusterEventListener] [Data-Center] Cluster is not recovered yet.
[2023-10-02T09:40:03,847][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration
[2023-10-02T09:40:03,870][INFO ][o.o.i.i.ManagedIndexCoordinator] [Data-Center] Cache cluster manager node onClusterManager time: 1696207203869
[2023-10-02T09:40:03,891][INFO ][o.o.d.PeerFinder ] [Data-Center] setting findPeersInterval to [1s] as node commission status = [true] for local node [{Data-Center}{2piMCxDbToSoDWsj2QpncA}{CIq7xakKT_eGfWuc38rbJQ}{192.168.10.225}{192.168.10.225:9300}{m}{shard_indexing_pressure_enabled=true}]
[2023-10-02T09:40:03,895][INFO ][o.o.h.AbstractHttpServerTransport] [Data-Center] publish_address {192.168.10.225:9200}, bound_addresses {192.168.10.225:9200}
[2023-10-02T09:40:03,896][INFO ][o.o.n.Node ] [Data-Center] started
[2023-10-02T09:40:03,897][INFO ][o.o.s.OpenSearchSecurityPlugin] [Data-Center] Node started
[2023-10-02T09:40:03,899][INFO ][o.o.s.OpenSearchSecurityPlugin] [Data-Center] 0 OpenSearch Security modules loaded so far:
[2023-10-02T09:40:03,906][INFO ][o.o.a.c.HashRing ] [Data-Center] Node added: [2piMCxDbToSoDWsj2QpncA]
[2023-10-02T09:40:03,911][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration
[2023-10-02T09:40:03,912][INFO ][o.o.a.c.HashRing ] [Data-Center] All nodes with known AD version: {2piMCxDbToSoDWsj2QpncA=ADNodeInfo{version=2.8.0, isEligibleDataNode=false}}
[2023-10-02T09:40:03,914][INFO ][o.o.a.c.HashRing ] [Data-Center] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
[2023-10-02T09:40:03,915][INFO ][o.o.a.c.HashRing ] [Data-Center] Build AD version hash ring successfully
[2023-10-02T09:40:03,916][INFO ][o.o.a.c.ADClusterEventListener] [Data-Center] Init AD version hash ring successfully
[2023-10-02T09:40:03,938][INFO ][o.o.g.GatewayService ] [Data-Center] recovered [0] indices into cluster_state
[2023-10-02T09:40:03,962][INFO ][o.o.p.PluginsService ] [Data-Center] PluginService:onIndexModule index:[.opensearch-observability/wC2BZrlwR5K8Ix3FhDc0Fw]
[2023-10-02T09:40:04,200][INFO ][o.o.c.m.MetadataCreateIndexService] [Data-Center] [.opensearch-observability] creating index, cause [api], templates , shards [1]/[0]
[2023-10-02T09:40:04,213][INFO ][o.o.c.r.a.AllocationService] [Data-Center] Cluster health status changed from [YELLOW] to [RED] (reason: [index [.opensearch-observability] created]).
[2023-10-02T09:40:04,257][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration
[2023-10-02T09:40:04,263][INFO ][o.o.p.PluginsService ] [Data-Center] PluginService:onIndexModule index:[configsync/wh7N0siISpa_97AvFGjAEA]
[2023-10-02T09:40:04,298][INFO ][o.o.c.m.MetadataCreateIndexService] [Data-Center] [configsync] creating index, cause [api], templates , shards [1]/[0]
[2023-10-02T09:40:04,331][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration
[2023-10-02T09:40:34,263][INFO ][o.o.o.i.ObservabilityIndex] [Data-Center] observability:Index .opensearch-observability creation Acknowledged
[2023-10-02T09:40:34,265][INFO ][o.o.o.i.ObservabilityTracesIndex] [Data-Center] observability:createMappingTemplate ss4o_trace_template API called
[2023-10-02T09:40:34,279][INFO ][o.o.p.PluginsService ] [Data-Center] PluginService:onIndexModule index:[validate-template-cvszzmtmtykrtblkdmmldg/4drnWvAbTxqcna5TmNvLcA]
[2023-10-02T09:40:34,325][INFO ][o.o.c.m.MetadataIndexTemplateService] [Data-Center] adding index template [ss4o_trace_template] for index patterns [ss4o_traces--]
[2023-10-02T09:40:34,361][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration
[2023-10-02T09:40:34,363][INFO ][o.o.o.i.ObservabilityTracesIndex] [Data-Center] observability:Mapping Template ss4o_trace_template creation Acknowledged
[2023-10-02T09:40:34,364][INFO ][o.o.o.i.ObservabilityMetricsIndex] [Data-Center] observability:createMappingTemplate ss4o_metric_template API called
[2023-10-02T09:40:34,370][INFO ][o.o.p.PluginsService ] [Data-Center] PluginService:onIndexModule index:[validate-template-hxgme5vorpivv8wf62e4pq/7-XLbscQToe4cTrq26tTaw]
[2023-10-02T09:40:34,408][INFO ][o.o.c.m.MetadataIndexTemplateService] [Data-Center] adding index template [ss4o_metric_template] for index patterns [ss4o_metrics--]
[2023-10-02T09:40:34,445][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration
[2023-10-02T09:40:34,447][INFO ][o.o.o.i.ObservabilityMetricsIndex] [Data-Center] observability:Mapping Template ss4o_metric_template creation Acknowledged
[2023-10-02T09:41:03,875][INFO ][o.o.i.i.ManagedIndexCoordinator] [Data-Center] Performing move cluster state metadata.
[2023-10-02T09:41:03,877][INFO ][o.o.i.i.MetadataService ] [Data-Center] ISM config index not exist, so we cancel the metadata migration job.
[2023-10-02T09:41:03,878][INFO ][o.o.i.i.ManagedIndexCoordinator] [Data-Center] Performing ISM template migration.
[2023-10-02T09:41:03,880][INFO ][o.o.i.i.m.ISMTemplateService] [Data-Center] Doing ISM template migration 1 time.
[2023-10-02T09:41:03,881][INFO ][o.o.i.i.m.ISMTemplateService] [Data-Center] Use 2023-10-01T23:40:03.869Z as migrating ISM template last_updated_time
[2023-10-02T09:41:03,886][INFO ][o.o.p.PluginsService ] [Data-Center] PluginService:onIndexModule index:[simulate_template_index_ljihwhiasrahi4uv1ct_gq/hnnZnH6iQBWZezm-dzbqig]
[2023-10-02T09:41:03,914][INFO ][o.o.p.PluginsService ] [Data-Center] PluginService:onIndexModule index:[simulate_template_index_ljihwhiasrahi4uv1ct_gq/hnnZnH6iQBWZezm-dzbqig]
[2023-10-02T09:41:03,950][INFO ][o.o.p.PluginsService ] [Data-Center] PluginService:onIndexModule index:[simulate_template_index_k-6kn0fhqou8aysqgoyvfq/hvIJQtjZSe-dR0oRiA9C2g]
[2023-10-02T09:41:03,979][INFO ][o.o.p.PluginsService ] [Data-Center] PluginService:onIndexModule index:[simulate_template_index_k-6kn0fhqou8aysqgoyvfq/hvIJQtjZSe-dR0oRiA9C2g]
[2023-10-02T09:41:04,012][INFO ][o.o.i.i.m.ISMTemplateService] [Data-Center] ISM templates: {=[ISMTemplate(indexPatterns=[ss4o_traces--], priority=1, lastUpdatedTime=2023-10-01T23:40:03.869Z), ISMTemplate(indexPatterns=[ss4o_metrics--], priority=1, lastUpdatedTime=2023-10-01T23:40:03.869Z)]}
[2023-10-02T09:41:04,017][INFO ][o.o.i.i.m.ISMTemplateService] [Data-Center] Policies to update:
[2023-10-02T09:41:04,021][INFO ][o.o.i.i.m.ISMTemplateService] [Data-Center] Failure experienced when migrating ISM Template and update ISM policies: {}
[2023-10-02T09:41:04,082][INFO ][o.o.c.s.ClusterSettings ] [Data-Center] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
[2023-10-02T09:41:04,084][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [Data-Center] Detected cluster change event for destination migration
[2023-10-02T09:41:04,088][INFO ][o.o.i.i.m.ISMTemplateService] [Data-Center] Successfully update template migration setting
[2023-10-02T09:41:04,338][INFO ][o.c.o.c.s.ConfigSyncService] [Data-Center] ConfigFileUpdater is started at 1m intervals.
[2023-10-02T09:42:03,876][INFO ][o.o.i.i.ManagedIndexCoordinator] [Data-Center] Cancel background move metadata process.
[2023-10-02T09:42:03,877][INFO ][o.o.i.i.ManagedIndexCoordinator] [Data-Center] Performing move cluster state metadata.
[2023-10-02T09:42:03,878][INFO ][o.o.i.i.MetadataService ] [Data-Center] Move metadata has finished.
[2023-10-02T09:42:04,365][ERROR][o.c.o.c.s.ConfigSyncService] [Data-Center] Failed to process ConfigFileUpdater.
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:664) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:372) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:699) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:472) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:273) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:350) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
at java.lang.Thread.run(Thread.java:1589) [?:?]
[2023-10-02T09:43:04,382][ERROR][o.c.o.c.s.ConfigSyncService] [Data-Center] Failed to process ConfigFileUpdater.
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:664) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:372) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:699) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:472) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:273) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:350) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
at java.lang.Thread.run(Thread.java:1589) [?:?]
[2023-10-02T09:44:04,396][ERROR][o.c.o.c.s.ConfigSyncService] [Data-Center] Failed to process ConfigFileUpdater.
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
=========================================================
From the error:
“org.opensearch.action.search.SearchPhaseExecutionException: all shards failed”
that my opensearch can not create initialized shards,
however, I don’t know how to do next.
Any suggestion is very appreciated.