How to configure opensearch on Azure Kubernetes Service

Open Source Elasticsearch and Kibana
1

Installing opensearch using helm on Azure Kubernetes Cluster gives error.
I am using installation procedure as in “Helm - OpenSearch documentation”
Can someone please help to resolve this.

[2021-09-29T16:07:15,684][WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-cluster-master-0] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{opensearch-cluster-master-0}{T0g89QFvRkqBSrFioudfnw}{2xvdapXBQwiT_V9-f78jcQ}{10.240.0.53}{10.240.0.53:9300}{dimr}, {opensearch-cluster-master-1}{Tj6OdyLUSJCEnIn-r-AmqQ}{kffoRpFPSxKrv8Z1w38F5g}{10.240.0.229}{10.240.0.229:9300}{dimr}, {opensearch-cluster-master-2}{vdq1_y0rSKuI69VKcwFyuQ}{IxCF1pp9TzWEj_xu6myu1Q}{10.240.0.132}{10.240.0.132:9300}{dimr}]; discovery will continue using [10.240.0.132:9300, 10.240.0.229:9300] from hosts providers and [{opensearch-cluster-master-0}{T0g89QFvRkqBSrFioudfnw}{2xvdapXBQwiT_V9-f78jcQ}{10.240.0.53}{10.240.0.53:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 2[2021-09-29T16:07:15,696][ERROR][o.o.s.c.ConfigurationRepository] [opensearch-cluster-master-0] Cannot apply default config (this is maybe not an error!) 3org.opensearch.discovery.MasterNotDiscoveredException: null 4 at org.opensearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:243) ~[opensearch-1.0.0.jar:1.0.0] 5 at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:348) ~[opensearch-1.0.0.jar:1.0.0] 6 at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:265) ~[opensearch-1.0.0.jar:1.0.0] 7 at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:614) ~[opensearch-1.0.0.jar:1.0.0] 8 at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:697) ~[opensearch-1.0.0.jar:1.0.0] 9 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?] 10 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?] 11 at java.lang.Thread.run(Thread.java:832) [?:?]

2

Any thoughts @TheAlgo ?

3

Thanks for your time.

I have tested the same with 1.0.0 version “https://github.com/opensearch-project/helm-charts/tree/1.0.0”. There was error initially but master node was discovered after sometime and cluster status went from yellow to green.
There seems to be some bug with joining nodes to cluster in new helm. Please find logs as below.

[2021-09-30T14:38:49,997][INFO ][o.o.c.c.JoinHelper       ] [opensearch-cluster-master-0] failed to join {opensearch-cluster-master-1}{Tj6OdyLUSJCEnIn-r-AmqQ}{N4IzU1iSQIi1iQFp8YKnBQ}{10.240.1.51}{10.240.1.51:9300}{dimr} with JoinRequest{sourceNode={opensearch-cluster-master-0}{T0g89QFvRkqBSrFioudfnw}{9txLblqLTA25F5gTWTP62Q}{10.240.0.90}{10.240.0.90:9300}{dimr}, minimumTerm=0, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={opensearch-cluster-master-0}{T0g89QFvRkqBSrFioudfnw}{9txLblqLTA25F5gTWTP62Q}{10.240.0.90}{10.240.0.90:9300}{dimr}, targetNode={opensearch-cluster-master-1}{Tj6OdyLUSJCEnIn-r-AmqQ}{N4IzU1iSQIi1iQFp8YKnBQ}{10.240.1.51}{10.240.1.51:9300}{dimr}}]}
org.opensearch.transport.NodeNotConnectedException: [opensearch-cluster-master-1][10.240.1.51:9300] Node not connected
	at org.opensearch.transport.ClusterConnectionManager.getConnection(ClusterConnectionManager.java:202) ~[opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.transport.TransportService.getConnection(TransportService.java:706) ~[opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.transport.TransportService.sendRequest(TransportService.java:626) [opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:317) [opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:242) [opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.cluster.coordination.JoinHelper.lambda$new$3(JoinHelper.java:166) [opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:138) [opensearch-index-management-1.0.0.0.jar:1.0.0.0]
	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:193) [opensearch-security-1.0.0.0.jar:1.0.0.0]
	at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:314) [opensearch-security-1.0.0.0.jar:1.0.0.0]
	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:153) [opensearch-security-1.0.0.0.jar:1.0.0.0]
	at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:644) [opensearch-security-1.0.0.0.jar:1.0.0.0]
	at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:64) [opensearch-performance-analyzer-1.0.0.0.jar:1.0.0.0]
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:85) [opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:318) [opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:756) [opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) [opensearch-1.0.0.jar:1.0.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-30T14:38:50,005][INFO ][o.o.c.c.JoinHelper       ] [opensearch-cluster-master-0] failed to join {opensearch-cluster-master-1}{Tj6OdyLUSJCEnIn-r-AmqQ}{N4IzU1iSQIi1iQFp8YKnBQ}{10.240.1.51}{10.240.1.51:9300}{dimr} with JoinRequest{sourceNode={opensearch-cluster-master-0}{T0g89QFvRkqBSrFioudfnw}{9txLblqLTA25F5gTWTP62Q}{10.240.0.90}{10.240.0.90:9300}{dimr}, minimumTerm=0, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={opensearch-cluster-master-0}{T0g89QFvRkqBSrFioudfnw}{9txLblqLTA25F5gTWTP62Q}{10.240.0.90}{10.240.0.90:9300}{dimr}, targetNode={opensearch-cluster-master-1}{Tj6OdyLUSJCEnIn-r-AmqQ}{N4IzU1iSQIi1iQFp8YKnBQ}{10.240.1.51}{10.240.1.51:9300}{dimr}}]}
org.opensearch.transport.NodeNotConnectedException: [opensearch-cluster-master-1][10.240.1.51:9300] Node not connected
	at org.opensearch.transport.ClusterConnectionManager.getConnection(ClusterConnectionManager.java:202) ~[opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.transport.TransportService.getConnection(TransportService.java:706) ~[opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.transport.TransportService.sendRequest(TransportService.java:626) [opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:317) [opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:242) [opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.cluster.coordination.JoinHelper.lambda$new$3(JoinHelper.java:166) [opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:138) [opensearch-index-management-1.0.0.0.jar:1.0.0.0]
	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:193) [opensearch-security-1.0.0.0.jar:1.0.0.0]
	at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:314) [opensearch-security-1.0.0.0.jar:1.0.0.0]
	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:153) [opensearch-security-1.0.0.0.jar:1.0.0.0]
	at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:644) [opensearch-security-1.0.0.0.jar:1.0.0.0]
	at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:64) [opensearch-performance-analyzer-1.0.0.0.jar:1.0.0.0]
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:85) [opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:318) [opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:756) [opensearch-1.0.0.jar:1.0.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) [opensearch-1.0.0.jar:1.0.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-30T14:38:50,120][INFO ][o.o.c.c.Coordinator      ] [opensearch-cluster-master-0] setting initial configuration to VotingConfiguration{T0g89QFvRkqBSrFioudfnw,Tj6OdyLUSJCEnIn-r-AmqQ,{bootstrap-placeholder}-opensearch-cluster-master-2}
[2021-09-30T14:38:50,263][INFO ][o.o.c.s.MasterService    ] [opensearch-cluster-master-0] elected-as-master ([2] nodes joined)[{opensearch-cluster-master-0}{T0g89QFvRkqBSrFioudfnw}{9txLblqLTA25F5gTWTP62Q}{10.240.0.90}{10.240.0.90:9300}{dimr} elect leader, {opensearch-cluster-master-1}{Tj6OdyLUSJCEnIn-r-AmqQ}{N4IzU1iSQIi1iQFp8YKnBQ}{10.240.1.51}{10.240.1.51:9300}{dimr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 2, version: 1, delta: master node changed {previous [], current [{opensearch-cluster-master-0}{T0g89QFvRkqBSrFioudfnw}{9txLblqLTA25F5gTWTP62Q}{10.240.0.90}{10.240.0.90:9300}{dimr}]}, added {{opensearch-cluster-master-1}{Tj6OdyLUSJCEnIn-r-AmqQ}{N4IzU1iSQIi1iQFp8YKnBQ}{10.240.1.51}{10.240.1.51:9300}{dimr}}
[2021-09-30T14:38:50,336][INFO ][o.o.c.c.CoordinationState] [opensearch-cluster-master-0] cluster UUID set to [3NfB7puKSsSIY8qIMT5hAw]
[2021-09-30T14:38:50,408][INFO ][o.o.c.s.ClusterApplierService] [opensearch-cluster-master-0] master node changed {previous [], current [{opensearch-cluster-master-0}{T0g89QFvRkqBSrFioudfnw}{9txLblqLTA25F5gTWTP62Q}{10.240.0.90}{10.240.0.90:9300}{dimr}]}, added {{opensearch-cluster-master-1}{Tj6OdyLUSJCEnIn-r-AmqQ}{N4IzU1iSQIi1iQFp8YKnBQ}{10.240.1.51}{10.240.1.51:9300}{dimr}}, term: 2, version: 1, reason: Publication{term=2, version=1}
[2021-09-30T14:38:50,421][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [opensearch-cluster-master-0] Config override setting update called with empty string. Ignoring.
[2021-09-30T14:38:50,422][INFO ][o.o.a.c.ADClusterEventListener] [opensearch-cluster-master-0] CLuster is not recovered yet.
[2021-09-30T14:38:50,436][INFO ][o.o.h.AbstractHttpServerTransport] [opensearch-cluster-master-0] publish_address {10.240.0.90:9200}, bound_addresses {[::]:9200}
[2021-09-30T14:38:50,436][INFO ][o.o.n.Node               ] [opensearch-cluster-master-0] started
[2021-09-30T14:38:50,437][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] Node started
[2021-09-30T14:38:50,437][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-master-0] Will attempt to create index .opendistro_security and default configs if they are absent
[2021-09-30T14:38:50,438][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] 0 OpenSearch Security modules loaded so far: []
[2021-09-30T14:38:50,438][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-master-0] Background init thread started. Install default config?: true
[2021-09-30T14:38:50,501][INFO ][o.o.g.GatewayService     ] [opensearch-cluster-master-0] recovered [0] indices into cluster_state
[2021-09-30T14:38:50,610][INFO ][o.o.c.m.MetadataCreateIndexService] [opensearch-cluster-master-0] [.opendistro_security] creating index, cause [api], templates [], shards [1]/[1]
[2021-09-30T14:38:50,779][INFO ][o.o.c.r.a.AllocationService] [opensearch-cluster-master-0] updating number_of_replicas to [2] for indices [.opendistro_security]
[2021-09-30T14:38:50,779][INFO ][o.o.c.s.MasterService    ] [opensearch-cluster-master-0] node-join[{opensearch-cluster-master-2}{vdq1_y0rSKuI69VKcwFyuQ}{l_uiuz-kSrW2-yl2D5UOeg}{10.240.0.132}{10.240.0.132:9300}{dimr} join existing leader], term: 2, version: 4, delta: added {{opensearch-cluster-master-2}{vdq1_y0rSKuI69VKcwFyuQ}{l_uiuz-kSrW2-yl2D5UOeg}{10.240.0.132}{10.240.0.132:9300}{dimr}}
[2021-09-30T14:38:51,005][INFO ][o.o.c.s.ClusterApplierService] [opensearch-cluster-master-0] added {{opensearch-cluster-master-2}{vdq1_y0rSKuI69VKcwFyuQ}{l_uiuz-kSrW2-yl2D5UOeg}{10.240.0.132}{10.240.0.132:9300}{dimr}}, term: 2, version: 4, reason: Publication{term=2, version=4}
[2021-09-30T14:38:51,012][INFO ][o.o.a.c.ADClusterEventListener] [opensearch-cluster-master-0] Data node added  vdq1_y0rSKuI69VKcwFyuQ
[2021-09-30T14:38:51,012][INFO ][o.o.a.c.HashRing         ] [opensearch-cluster-master-0] Rebuild hash ring
[2021-09-30T14:38:51,127][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-master-0] Index .opendistro_security created?: true
[2021-09-30T14:38:51,127][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-master-0] Node started, try to initialize it. Wait for at least yellow cluster state....
[2021-09-30T14:38:51,128][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-cluster-master-0] Will update 'config' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2021-09-30T14:38:51,479][INFO ][o.o.c.m.MetadataMappingService] [opensearch-cluster-master-0] [.opendistro_security/ej0eEMmNSyaYKrHDfzytcQ] create_mapping [_doc]
[2021-09-30T14:38:51,997][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-cluster-master-0] Index .opendistro_security already contains doc with id config, skipping update.
[2021-09-30T14:38:51,997][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-cluster-master-0] Will update 'roles' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2021-09-30T14:38:52,010][INFO ][o.o.c.m.MetadataMappingService] [opensearch-cluster-master-0] [.opendistro_security/ej0eEMmNSyaYKrHDfzytcQ] update_mapping [_doc]
[2021-09-30T14:38:52,185][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-cluster-master-0] Doc with id 'roles' and version 2 is updated in .opendistro_security index.
[2021-09-30T14:38:52,185][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-cluster-master-0] Will update 'rolesmapping' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2021-09-30T14:38:52,201][INFO ][o.o.c.m.MetadataMappingService] [opensearch-cluster-master-0] [.opendistro_security/ej0eEMmNSyaYKrHDfzytcQ] update_mapping [_doc]
[2021-09-30T14:38:52,291][INFO ][o.o.c.r.a.AllocationService] [opensearch-cluster-master-0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opendistro_security][0]]]).
4

@sharath_90 Can you try installing using the main branch and confirm if the same issue happens again? Thanks

5

I have retried installing of latest version on fresh Azure Kubernetes Cluster and endup with Bootstrap error as below.
I think the setting in values.yaml is not getting enforced "“sysctlVmMaxMapCount: 262144”.
I havent changed anything in config. Please help with way out of this.

[2021-10-04T09:15:40,385][INFO ][o.o.b.BootstrapChecks    ] [opensearch-cluster-master-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
6

Looks like the virtual memory is less, we need 262144

7

This got fixed once we changed virtual memory to prescribed value “262144”. My question is why is sytemctld not setting this value when we run the installation?

1 Like