Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
1.3.9
Describe the issue:
[2023-11-27T10:33:42,521][WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-0] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [opensearch-0, opensearch-1, opensearch-2] to bootstrap a cluster: have discovered [{opensearch-0}{lRptH16WQNyNreat9pxZUg}{2lG26cuISAeUmF5GCmlQaQ}<redacted>{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [<ip>:9300] from hosts providers and [{opensearch-0}{lRptH16WQNyNreat9pxZUg}{2lG26cuISAeUmF5GCmlQaQ}<redacted>{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-11-27T10:33:48,571][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:202) ~[opensearch-1.3.9.jar:1.3.9]
at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:188) ~[opensearch-1.3.9.jar:1.3.9]
at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:76) ~[opensearch-1.3.9.jar:1.3.9]
at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:53) ~[opensearch-1.3.9.jar:1.3.9]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:194) [opensearch-1.3.9.jar:1.3.9]
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:121) [opensearch-index-management-1.3.9.0.jar:1.3.9.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.3.9.jar:1.3.9]
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-1.3.9.0.jar:1.3.9.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.3.9.jar:1.3.9]
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:242) [opensearch-security-1.3.9.0.jar:1.3.9.0]
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:162) [opensearch-security-1.3.9.0.jar:1.3.9.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.3.9.jar:1.3.9]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:169) [opensearch-1.3.9.jar:1.3.9]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:97) [opensearch-1.3.9.jar:1.3.9]
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:108) [opensearch-1.3.9.jar:1.3.9]
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:95) [opensearch-1.3.9.jar:1.3.9]
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433) [opensearch-1.3.9.jar:1.3.9]
at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:554) [opensearch-1.3.9.jar:1.3.9]
at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:212) [opensearch-security-1.3.9.0.jar:1.3.9.0]
at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:103) [opensearch-security-1.3.9.0.jar:1.3.9.0]
at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:375) [opensearch-security-1.3.9.0.jar:1.3.9.0]
at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:321) [opensearch-security-1.3.9.0.jar:1.3.9.0]
at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:306) [opensearch-security-1.3.9.0.jar:1.3.9.0]
at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:166) [opensearch-security-1.3.9.0.jar:1.3.9.0]
at java.lang.Thread.run(Thread.java:829) [?:?]
Configuration:
I have a three node cluster, and trying to bring up three sts pods of opensearch. Based on my requirement I have some configuration in sts yaml file priorityClassName: high-priority dnsPolicy: ClusterFirstWithHostNet hostNetwork: true
Three pods are configured to run on three nodes respectively. Due to hostNetwork:true configuration the host ip and pod ip will be same. when I try to bring the three pods up. Im encountering the above warning and error message.
when I remove the hostNetwork:true and try to run with same configuration, it works, the cluster comes up smoothly.
I tried giving opensearch-0,1,2 in seed_hosts in opensearch.yml but it gave an error saying unable to resolve host.
Below is the part of opensearch.yaml configuration:
# Use a descriptive name for the node:
#
node.name: opensearch-0
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/opensearch
#
# Path to log files:
#
path.logs: /var/log/opensearch
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
http.port: 9200
http.max_initial_line_length: 128kb
#
# --------------------------------- Discovery ----------------------------------
#
discovery.seed_hosts: [os-search-discovery-service]
cluster.initial_master_nodes: ["opensearch-0","opensearch-1","opensearch-2"]
#
# ---------------------------------- Gateway -----------------------------------
#
gateway.recover_after_time: 5m
gateway.expected_nodes: 1
gateway.recover_after_nodes: 1
#
# ---------------------------------- Various -----------------------------------
#
# Disable starting multiple nodes on a single system:
#
node.max_local_storage_nodes: 1
action.destructive_requires_name: true
node.master: true
node.data: true