Add new nodes to a cluster and change shard settings

Security
21

opensearch version is 2.7 the latest one .

I want to use both the nodes as master nodes so I have previously put cluster_mater for both of them later I test with no roles so that it picks up default values which includes all the roles

22

It was not forming cluster whether I add roles or not

23

@rhtbansal Did you follow the logs? Look at the OpenSearch logs when nodes are trying to communicate with each other. Search for transport layer messages with the IP addresses and node names defined in discovery.seed_hosts and cluster.initial_cluster_manager_nodes.

Also try this instead.

network.host: 0.0.0.0
http.port: 9200
search.max_buckets: 100000
discovery.seed_hosts: ["<IP_address_of_node1>"]
cluster.name: acoecluster3
node.name: node-1
cluster.initial_cluster_manager_nodes: ["node-1"]

and

network.host: 0.0.0.0
http.port: 9200
search.max_buckets: 100000
discovery.seed_hosts: ["<IP_address_of_node1>"]
cluster.name: acoecluster3
node.name: node-2
24

I have got below logs for my node 2
[2023-06-02T16:01:05,598][INFO ][o.o.n.Node ] [node-2] version[2.7.0], pid[5736], build[tar/b7a6e09e492b1e965d827525f7863b366ef0e304/2023-04-27T21:43:09.523336706Z], OS[Linux/5.15.0-1038-azure/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.6/17.0.6+10]
[2023-06-02T16:01:05,601][INFO ][o.o.n.Node ] [node-2] JVM home [/opensearch-2.7.0/jdk], using bundled JDK [true]
[2023-06-02T16:01:05,601][INFO ][o.o.n.Node ] [node-2] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-15164359606292703526, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/opensearch-2.7.0/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/opensearch-2.7.0, -Dopensearch.path.conf=/opensearch-2.7.0/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2023-06-02T16:01:06,477][INFO ][o.o.s.s.t.SSLConfig ] [node-2] SSL dual mode is disabled
[2023-06-02T16:01:06,477][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-2] OpenSearch Config path is /opensearch-2.7.0/config
[2023-06-02T16:01:06,707][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-2] JVM supports TLSv1.3
[2023-06-02T16:01:06,709][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-2] Config directory is /opensearch-2.7.0/config/, from there the key- and truststore files are resolved relatively
[2023-06-02T16:01:07,068][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-2] TLS Transport Client Provider : JDK
[2023-06-02T16:01:07,068][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-2] TLS Transport Server Provider : JDK
[2023-06-02T16:01:07,068][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-2] TLS HTTP Provider : JDK
[2023-06-02T16:01:07,068][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-2] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2023-06-02T16:01:07,069][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-2] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[2023-06-02T16:01:07,076][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-2] Clustername: acoecluster4
[2023-06-02T16:01:07,082][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] Directory /opensearch-2.7.0/config has insecure file permissions (should be 0700)
[2023-06-02T16:01:07,083][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/esnode-key.pem has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,083][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/kirk-key.pem has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,083][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch.yml has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,084][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] Directory /opensearch-2.7.0/config/opensearch-reports-scheduler has insecure file permissions (should be 0700)
[2023-06-02T16:01:07,084][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-reports-scheduler/reports-scheduler.yml has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,084][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] Directory /opensearch-2.7.0/config/opensearch-performance-analyzer has insecure file permissions (should be 0700)
[2023-06-02T16:01:07,084][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-performance-analyzer/agent-stats-metadata has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,085][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-performance-analyzer/supervisord.conf has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,085][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-performance-analyzer/performance-analyzer.properties has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,085][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-performance-analyzer/plugin-stats-metadata has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,086][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-performance-analyzer/rca_idle_cluster_manager.conf has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,086][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-performance-analyzer/log4j2.xml has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,086][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-performance-analyzer/rca.conf has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,086][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-performance-analyzer/opensearch_security.policy has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,087][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-performance-analyzer/rca_cluster_manager.conf has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,087][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] Directory /opensearch-2.7.0/config/opensearch-notifications-core has insecure file permissions (should be 0700)
[2023-06-02T16:01:07,087][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-notifications-core/notifications-core.yml has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,088][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/log4j2.properties has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,088][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] Directory /opensearch-2.7.0/config/jvm.options.d has insecure file permissions (should be 0700)
[2023-06-02T16:01:07,088][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/esnode.pem has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,088][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] Directory /opensearch-2.7.0/config/opensearch-notifications has insecure file permissions (should be 0700)
[2023-06-02T16:01:07,089][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-notifications/notifications.yml has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,089][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch.keystore has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,089][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/jvm.options has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,089][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/root-ca.pem has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,089][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/kirk.pem has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,090][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] Directory /opensearch-2.7.0/config/opensearch-observability has insecure file permissions (should be 0700)
[2023-06-02T16:01:07,090][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-observability/observability.yml has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,090][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] Directory /opensearch-2.7.0/config/opensearch-security has insecure file permissions (should be 0700)
[2023-06-02T16:01:07,090][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-security/allowlist.yml has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,091][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-security/internal_users.yml has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,091][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-security/audit.yml has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,091][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-security/opensearch.yml.example has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,091][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-security/roles.yml has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,091][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-security/roles_mapping.yml has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,092][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-security/action_groups.yml has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,092][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-security/config.yml has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,092][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-security/tenants.yml has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,092][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-security/nodes_dn.yml has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,093][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-2] File /opensearch-2.7.0/config/opensearch-security/whitelist.yml has insecure file permissions (should be 0600)
[2023-06-02T16:01:07,395][INFO ][o.o.p.c.PluginSettings ] [node-2] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2023-06-02T16:01:07,723][INFO ][o.o.i.r.ReindexPlugin ] [node-2] ReindexPlugin reloadSPI called
[2023-06-02T16:01:07,724][INFO ][o.o.i.r.ReindexPlugin ] [node-2] Unable to find any implementation for RemoteReindexExtension
[2023-06-02T16:01:07,759][INFO ][o.o.j.JobSchedulerPlugin ] [node-2] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2023-06-02T16:01:07,762][INFO ][o.o.j.JobSchedulerPlugin ] [node-2] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2023-06-02T16:01:07,762][INFO ][o.o.j.JobSchedulerPlugin ] [node-2] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2023-06-02T16:01:07,777][INFO ][o.o.j.JobSchedulerPlugin ] [node-2] Loaded scheduler extension: observability, index: .opensearch-observability-job
[2023-06-02T16:01:07,781][INFO ][o.o.p.PluginsService ] [node-2] loaded module [aggs-matrix-stats]
[2023-06-02T16:01:07,781][INFO ][o.o.p.PluginsService ] [node-2] loaded module [analysis-common]
[2023-06-02T16:01:07,781][INFO ][o.o.p.PluginsService ] [node-2] loaded module [geo]
[2023-06-02T16:01:07,781][INFO ][o.o.p.PluginsService ] [node-2] loaded module [ingest-common]
[2023-06-02T16:01:07,782][INFO ][o.o.p.PluginsService ] [node-2] loaded module [ingest-geoip]
[2023-06-02T16:01:07,782][INFO ][o.o.p.PluginsService ] [node-2] loaded module [ingest-user-agent]
[2023-06-02T16:01:07,782][INFO ][o.o.p.PluginsService ] [node-2] loaded module [lang-expression]
[2023-06-02T16:01:07,782][INFO ][o.o.p.PluginsService ] [node-2] loaded module [lang-mustache]
[2023-06-02T16:01:07,782][INFO ][o.o.p.PluginsService ] [node-2] loaded module [lang-painless]
[2023-06-02T16:01:07,782][INFO ][o.o.p.PluginsService ] [node-2] loaded module [mapper-extras]
[2023-06-02T16:01:07,782][INFO ][o.o.p.PluginsService ] [node-2] loaded module [opensearch-dashboards]
[2023-06-02T16:01:07,783][INFO ][o.o.p.PluginsService ] [node-2] loaded module [parent-join]
[2023-06-02T16:01:07,783][INFO ][o.o.p.PluginsService ] [node-2] loaded module [percolator]
[2023-06-02T16:01:07,783][INFO ][o.o.p.PluginsService ] [node-2] loaded module [rank-eval]
[2023-06-02T16:01:07,783][INFO ][o.o.p.PluginsService ] [node-2] loaded module [reindex]
[2023-06-02T16:01:07,783][INFO ][o.o.p.PluginsService ] [node-2] loaded module [repository-url]
[2023-06-02T16:01:07,783][INFO ][o.o.p.PluginsService ] [node-2] loaded module [search-pipeline-common]
[2023-06-02T16:01:07,783][INFO ][o.o.p.PluginsService ] [node-2] loaded module [systemd]
[2023-06-02T16:01:07,784][INFO ][o.o.p.PluginsService ] [node-2] loaded module [transport-netty4]
[2023-06-02T16:01:07,784][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-alerting]
[2023-06-02T16:01:07,784][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-anomaly-detection]
[2023-06-02T16:01:07,784][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-asynchronous-search]
[2023-06-02T16:01:07,784][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-cross-cluster-replication]
[2023-06-02T16:01:07,785][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-geospatial]
[2023-06-02T16:01:07,785][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-index-management]
[2023-06-02T16:01:07,785][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-job-scheduler]
[2023-06-02T16:01:07,785][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-knn]
[2023-06-02T16:01:07,785][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-ml]
[2023-06-02T16:01:07,785][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-neural-search]
[2023-06-02T16:01:07,785][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-notifications]
[2023-06-02T16:01:07,785][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-notifications-core]
[2023-06-02T16:01:07,786][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-observability]
[2023-06-02T16:01:07,786][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-performance-analyzer]
[2023-06-02T16:01:07,786][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-reports-scheduler]
[2023-06-02T16:01:07,786][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-security]
[2023-06-02T16:01:07,786][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-security-analytics]
[2023-06-02T16:01:07,786][INFO ][o.o.p.PluginsService ] [node-2] loaded plugin [opensearch-sql]
[2023-06-02T16:01:07,815][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-2] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting ‘http.compression: true’ in opensearch.yml
[2023-06-02T16:01:07,834][INFO ][o.o.e.NodeEnvironment ] [node-2] using [1] data paths, mounts [[/ (/dev/root)]], net usable_space [24.5gb], net total_space [28.8gb], types [ext4]
[2023-06-02T16:01:07,834][INFO ][o.o.e.NodeEnvironment ] [node-2] heap size [1gb], compressed ordinary object pointers [true]
[2023-06-02T16:01:07,907][INFO ][o.o.n.Node ] [node-2] node name [node-2], node ID [Bs_g_fEhTrGzb3uLxipn_g], cluster name [acoecluster4], roles [ingest, remote_cluster_client, data, cluster_manager]
[2023-06-02T16:01:10,211][WARN ][o.o.s.c.Salt ] [node-2] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2023-06-02T16:01:10,238][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Message routing enabled: true
[2023-06-02T16:01:10,261][INFO ][o.o.s.f.SecurityFilter ] [node-2] indices are made immutable.
[2023-06-02T16:01:10,546][INFO ][o.o.a.b.ADCircuitBreakerService] [node-2] Registered memory breaker.
[2023-06-02T16:01:10,838][INFO ][o.o.m.b.MLCircuitBreakerService] [node-2] Registered ML memory breaker.
[2023-06-02T16:01:10,839][INFO ][o.o.m.b.MLCircuitBreakerService] [node-2] Registered ML disk breaker.
[2023-06-02T16:01:10,840][INFO ][o.o.m.b.MLCircuitBreakerService] [node-2] Registered ML native memory breaker.
[2023-06-02T16:01:10,921][INFO ][o.r.Reflections ] [node-2] Reflections took 36 ms to scan 1 urls, producing 15 keys and 37 values
[2023-06-02T16:01:11,393][INFO ][o.o.t.NettyAllocator ] [node-2] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2023-06-02T16:01:11,460][INFO ][o.o.d.DiscoveryModule ] [node-2] using discovery type [zen] and seed hosts providers [settings]
[2023-06-02T16:01:11,786][WARN ][o.o.g.DanglingIndicesState] [node-2] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-06-02T16:01:11,883][WARN ][o.o.t.TransportService ] [node-2] invalid action name [cluster:feature/tenancy/config/read] must start with one of: [cluster:admin, indices:data/read, indices:monitor, indices:data/write, internal:, indices:internal, cluster:monitor, cluster:internal, indices:admin]
[2023-06-02T16:01:11,888][WARN ][o.o.t.TransportService ] [node-2] invalid action name [cluster:feature/tenancy/config/update] must start with one of: [cluster:admin, indices:data/read, indices:monitor, indices:data/write, internal:, indices:internal, cluster:monitor, cluster:internal, indices:admin]
[2023-06-02T16:01:12,112][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [node-2] PerformanceAnalyzer Enabled: false
[2023-06-02T16:01:12,133][INFO ][o.o.n.Node ] [node-2] initialized
[2023-06-02T16:01:12,134][INFO ][o.o.n.Node ] [node-2] starting …
[2023-06-02T16:01:12,203][INFO ][o.o.t.TransportService ] [node-2] publish_address {10.29.2.6:9300}, bound_addresses {[::]:9300}
[2023-06-02T16:01:12,411][INFO ][o.o.b.BootstrapChecks ] [node-2] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-06-02T16:01:12,415][INFO ][o.o.c.c.Coordinator ] [node-2] cluster UUID [cxKTNr8eTPu4D_wt38SQ2A]
[2023-06-02T16:01:12,571][INFO ][o.o.c.s.MasterService ] [node-2] elected-as-cluster-manager ([1] nodes joined)[{node-2}{Bs_g_fEhTrGzb3uLxipn_g}{dULhJTJiQ-eaYPW7YF5D-w}{10.29.2.6}{10.29.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, BECOME_CLUSTER_MANAGER_TASK, FINISH_ELECTION], term: 43, version: 165, delta: cluster-manager node changed {previous , current [{node-2}{Bs_g_fEhTrGzb3uLxipn_g}{dULhJTJiQ-eaYPW7YF5D-w}{10.29.2.6}{10.29.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true}]}
[2023-06-02T16:01:12,674][INFO ][o.o.c.s.ClusterApplierService] [node-2] cluster-manager node changed {previous , current [{node-2}{Bs_g_fEhTrGzb3uLxipn_g}{dULhJTJiQ-eaYPW7YF5D-w}{10.29.2.6}{10.29.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 43, version: 165, reason: Publication{term=43, version=165}
[2023-06-02T16:01:12,680][INFO ][o.o.a.c.ADClusterEventListener] [node-2] Cluster is not recovered yet.
[2023-06-02T16:01:12,685][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-2] Detected cluster change event for destination migration
[2023-06-02T16:01:12,712][INFO ][o.o.i.i.ManagedIndexCoordinator] [node-2] Cache cluster manager node onClusterManager time: 1685701872712
[2023-06-02T16:01:12,735][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [node-2] Config override setting update called with empty string. Ignoring.
[2023-06-02T16:01:12,748][INFO ][o.o.h.AbstractHttpServerTransport] [node-2] publish_address {10.29.2.6:9200}, bound_addresses {[::]:9200}
[2023-06-02T16:01:12,748][INFO ][o.o.n.Node ] [node-2] started
[2023-06-02T16:01:12,749][INFO ][o.o.d.PeerFinder ] [node-2] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node-2}{Bs_g_fEhTrGzb3uLxipn_g}{dULhJTJiQ-eaYPW7YF5D-w}{10.29.2.6}{10.29.2.6:9300}{dimr}{shard_indexing_pressure_enabled=true}]
[2023-06-02T16:01:12,750][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-2] Node started
[2023-06-02T16:01:12,750][INFO ][o.o.s.c.ConfigurationRepository] [node-2] Will attempt to create index .opendistro_security and default configs if they are absent
[2023-06-02T16:01:12,751][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-2] 0 OpenSearch Security modules loaded so far:
[2023-06-02T16:01:12,755][INFO ][o.o.s.c.ConfigurationRepository] [node-2] Background init thread started. Install default config?: true
[2023-06-02T16:01:12,755][INFO ][o.o.s.c.ConfigurationRepository] [node-2] Wait for cluster to be available …
[2023-06-02T16:01:12,818][INFO ][o.o.c.s.ClusterSettings ] [node-2] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
[2023-06-02T16:01:12,821][INFO ][o.o.a.c.HashRing ] [node-2] Node added: [Bs_g_fEhTrGzb3uLxipn_g]
[2023-06-02T16:01:12,823][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-2] Detected cluster change event for destination migration
[2023-06-02T16:01:12,824][INFO ][o.o.a.c.HashRing ] [node-2] Add data node to AD version hash ring: Bs_g_fEhTrGzb3uLxipn_g
[2023-06-02T16:01:12,825][INFO ][o.o.a.c.HashRing ] [node-2] All nodes with known AD version: {Bs_g_fEhTrGzb3uLxipn_g=ADNodeInfo{version=2.7.0, isEligibleDataNode=true}}
[2023-06-02T16:01:12,826][INFO ][o.o.a.c.HashRing ] [node-2] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
[2023-06-02T16:01:12,826][INFO ][o.o.a.c.HashRing ] [node-2] Build AD version hash ring successfully
[2023-06-02T16:01:12,827][INFO ][o.o.a.c.ADDataMigrator ] [node-2] Start migrating AD data
[2023-06-02T16:01:12,827][INFO ][o.o.a.c.ADDataMigrator ] [node-2] AD job index doesn’t exist, no need to migrate
[2023-06-02T16:01:12,827][INFO ][o.o.a.c.ADClusterEventListener] [node-2] Init AD version hash ring successfully
[2023-06-02T16:01:12,842][INFO ][o.o.g.GatewayService ] [node-2] recovered [3] indices into cluster_state
[2023-06-02T16:01:12,845][WARN ][o.o.o.i.ObservabilityIndex] [node-2] message: index [.opensearch-observability/Q-8RHmSKTTS6eOyqsMqCsA] already exists
[2023-06-02T16:01:12,845][INFO ][o.o.o.i.ObservabilityTracesIndex] [node-2] observability:createMappingTemplate ss4o_trace_template API called
[2023-06-02T16:01:13,029][INFO ][o.o.p.PluginsService ] [node-2] PluginService:onIndexModule index:[.opendistro_security/Wri5aevvQgKFmf3csU_VXg]
[2023-06-02T16:01:13,107][INFO ][o.o.p.PluginsService ] [node-2] PluginService:onIndexModule index:[.opensearch-observability/Q-8RHmSKTTS6eOyqsMqCsA]
[2023-06-02T16:01:13,119][INFO ][o.o.p.PluginsService ] [node-2] PluginService:onIndexModule index:[security-auditlog-2023.05.29/XYQAkl5AQie_NJvSPh3rgw]
[2023-06-02T16:01:13,222][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-2] Detected cluster change event for destination migration
[2023-06-02T16:01:13,233][INFO ][o.o.p.PluginsService ] [node-2] PluginService:onIndexModule index:[validate-template-vfluxwcct32kvfkjlssysq/ePanBUMVQH2ywuUdaJ5vyw]
[2023-06-02T16:01:13,255][INFO ][o.o.c.m.MetadataIndexTemplateService] [node-2] updating index template [ss4o_trace_template] for index patterns [ss4o_traces--]
[2023-06-02T16:01:13,362][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-2] Detected cluster change event for destination migration
[2023-06-02T16:01:13,364][INFO ][o.o.o.i.ObservabilityTracesIndex] [node-2] observability:Mapping Template ss4o_trace_template creation Acknowledged
[2023-06-02T16:01:13,364][INFO ][o.o.o.i.ObservabilityMetricsIndex] [node-2] observability:createMappingTemplate ss4o_metric_template API called
[2023-06-02T16:01:13,368][INFO ][o.o.c.r.a.AllocationService] [node-2] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.opensearch-observability][0], [security-auditlog-2023.05.29][0], [.opendistro_security][0]]]).
[2023-06-02T16:01:13,437][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-2] Detected cluster change event for destination migration
[2023-06-02T16:01:13,442][INFO ][o.o.p.PluginsService ] [node-2] PluginService:onIndexModule index:[validate-template-ywl3qpupsjsqkfgjdstfpw/LsO24LGxTlm247kngzAgAw]
[2023-06-02T16:01:13,459][INFO ][o.o.c.m.MetadataIndexTemplateService] [node-2] updating index template [ss4o_metric_template] for index patterns [ss4o_metrics--]
[2023-06-02T16:01:13,543][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-2] Detected cluster change event for destination migration
[2023-06-02T16:01:13,544][INFO ][o.o.o.i.ObservabilityMetricsIndex] [node-2] observability:Mapping Template ss4o_metric_template creation Acknowledged
[2023-06-02T16:01:13,757][INFO ][o.o.s.c.ConfigurationRepository] [node-2] Index .opendistro_security already exists
[2023-06-02T16:01:13,758][INFO ][o.o.s.c.ConfigurationRepository] [node-2] Node started, try to initialize it. Wait for at least yellow cluster state…
[2023-06-02T16:01:13,762][INFO ][o.o.s.s.ConfigHelper ] [node-2] Will update ‘config’ with /opensearch-2.7.0/config/opensearch-security/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-06-02T16:01:13,808][INFO ][o.o.s.s.ConfigHelper ] [node-2] Index .opendistro_security already contains doc with id config, skipping update.
[2023-06-02T16:01:13,809][INFO ][o.o.s.s.ConfigHelper ] [node-2] Will update ‘roles’ with /opensearch-2.7.0/config/opensearch-security/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-06-02T16:01:13,815][INFO ][o.o.s.s.ConfigHelper ] [node-2] Index .opendistro_security already contains doc with id roles, skipping update.
[2023-06-02T16:01:13,815][INFO ][o.o.s.s.ConfigHelper ] [node-2] Will update ‘rolesmapping’ with /opensearch-2.7.0/config/opensearch-security/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-06-02T16:01:13,820][INFO ][o.o.s.s.ConfigHelper ] [node-2] Index .opendistro_security already contains doc with id rolesmapping, skipping update.
[2023-06-02T16:01:13,821][INFO ][o.o.s.s.ConfigHelper ] [node-2] Will update ‘internalusers’ with /opensearch-2.7.0/config/opensearch-security/internal_users.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-06-02T16:01:13,826][INFO ][o.o.s.s.ConfigHelper ] [node-2] Index .opendistro_security already contains doc with id internalusers, skipping update.
[2023-06-02T16:01:13,826][INFO ][o.o.s.s.ConfigHelper ] [node-2] Will update ‘actiongroups’ with /opensearch-2.7.0/config/opensearch-security/action_groups.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-06-02T16:01:13,828][INFO ][o.o.s.s.ConfigHelper ] [node-2] Index .opendistro_security already contains doc with id actiongroups, skipping update.
[2023-06-02T16:01:13,828][INFO ][o.o.s.s.ConfigHelper ] [node-2] Will update ‘tenants’ with /opensearch-2.7.0/config/opensearch-security/tenants.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-06-02T16:01:13,830][INFO ][o.o.s.s.ConfigHelper ] [node-2] Index .opendistro_security already contains doc with id tenants, skipping update.
[2023-06-02T16:01:13,830][INFO ][o.o.s.s.ConfigHelper ] [node-2] Will update ‘nodesdn’ with /opensearch-2.7.0/config/opensearch-security/nodes_dn.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2023-06-02T16:01:13,831][INFO ][o.o.s.s.ConfigHelper ] [node-2] Index .opendistro_security already contains doc with id nodesdn, skipping update.
[2023-06-02T16:01:13,831][INFO ][o.o.s.s.ConfigHelper ] [node-2] Will update ‘whitelist’ with /opensearch-2.7.0/config/opensearch-security/whitelist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2023-06-02T16:01:13,833][INFO ][o.o.s.s.ConfigHelper ] [node-2] Index .opendistro_security already contains doc with id whitelist, skipping update.
[2023-06-02T16:01:13,833][INFO ][o.o.s.s.ConfigHelper ] [node-2] Will update ‘allowlist’ with /opensearch-2.7.0/config/opensearch-security/allowlist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2023-06-02T16:01:13,835][INFO ][o.o.s.s.ConfigHelper ] [node-2] Index .opendistro_security already contains doc with id allowlist, skipping update.
[2023-06-02T16:01:13,835][INFO ][o.o.s.s.ConfigHelper ] [node-2] Will update ‘audit’ with /opensearch-2.7.0/config/opensearch-security/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2023-06-02T16:01:13,848][INFO ][o.o.s.s.ConfigHelper ] [node-2] Index .opendistro_security already contains doc with id audit, skipping update.
[2023-06-02T16:01:13,955][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Auditing on REST API is enabled.
[2023-06-02T16:01:13,956][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
[2023-06-02T16:01:13,956][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Auditing on Transport API is enabled.
[2023-06-02T16:01:13,956][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
[2023-06-02T16:01:13,956][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Auditing of request body is enabled.
[2023-06-02T16:01:13,956][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Bulk requests resolution is disabled during request auditing.
[2023-06-02T16:01:13,956][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Index resolution is enabled during request auditing.
[2023-06-02T16:01:13,956][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Sensitive headers auditing is enabled.
[2023-06-02T16:01:13,957][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Auditing requests from kibanaserver users is disabled.
[2023-06-02T16:01:13,959][WARN ][o.o.s.a.r.AuditMessageRouter] [node-2] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[2023-06-02T16:01:13,959][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Auditing of external configuration is disabled.
[2023-06-02T16:01:13,959][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Auditing of internal configuration is enabled.
[2023-06-02T16:01:13,959][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Auditing only metadata information for read request is enabled.
[2023-06-02T16:01:13,960][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Auditing will watch {} for read requests.
[2023-06-02T16:01:13,960][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Auditing read operation requests from kibanaserver users is disabled.
[2023-06-02T16:01:13,960][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Auditing only metadata information for write request is enabled.
[2023-06-02T16:01:13,960][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Auditing diffs for write requests is disabled.
[2023-06-02T16:01:13,960][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Auditing write operation requests from kibanaserver users is disabled.
[2023-06-02T16:01:13,960][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Auditing will watch for write requests.
[2023-06-02T16:01:13,960][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] .opendistro_security is used as internal security index.
[2023-06-02T16:01:13,960][INFO ][o.o.s.a.i.AuditLogImpl ] [node-2] Internal index used for posting audit logs is null
[2023-06-02T16:01:13,961][INFO ][o.o.s.c.ConfigurationRepository] [node-2] Hot-reloading of audit configuration is enabled
[2023-06-02T16:01:13,961][INFO ][o.o.s.c.ConfigurationRepository] [node-2] Node ‘node-2’ initialized
[2023-06-02T16:02:12,714][INFO ][o.o.i.i.ManagedIndexCoordinator] [node-2] Performing move cluster state metadata.
[2023-06-02T16:02:12,714][INFO ][o.o.i.i.MetadataService ] [node-2] ISM config index not exist, so we cancel the metadata migration job.
[2023-06-02T16:03:12,714][INFO ][o.o.i.i.ManagedIndexCoordinator] [node-2] Cancel background move metadata process.
[2023-06-02T16:03:12,715][INFO ][o.o.i.i.ManagedIndexCoordinator] [node-2] Performing move cluster state metadata.
[2023-06-02T16:03:12,715][INFO ][o.o.i.i.MetadataService ] [node-2] Move metadata has finished.

25

@pablo didn’t find anywhere any mention of node-1.
I am not sure I am making any silly mistake or what but I ma stuck at this