Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
this is my opensearch.yml configuration for 3 nodes.
i am using external IP address
Node 1 (general-vm-0):
cluster.name: opensearch-cluster
node.name: node-1
node.roles: [master, data, ingest]
network.host: 0.0.0.0
discovery.seed_hosts: ["node-1 ip", "node-2 ip", "node-3 ip"]
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
Node 2 (general-vm-1):
cluster.name: opensearch-cluster
node.name: node-2
node.roles: [master, data, ingest]
network.host: 0.0.0.0
discovery.seed_hosts: ["node-1 ip", "node-2 ip", "node-3 ip"]
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
Node 3 (general-vm-2):
cluster.name: opensearch-cluster
node.name: node-3
node.roles: [master, data, ingest]
network.host: 0.0.0.0
discovery.seed_hosts: ["node-1 ip", "node-2 ip", "node-3 ip"]
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
after this i am running this command to check cluster status, but i am not getting 3 nodes.
Describe the issue:
Configuration:
Relevant Logs or Screenshots:
also i am not able to ping each other.
logs from node 1
sbhawsar@BA-000254518@general-vm-0:/var/log/opensearch$ sudo tail -100f opensearch-cluster.log
[2024-05-29T06:12:45,900][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Wait for cluster to be available ...
[2024-05-29T06:12:45,916][INFO ][o.o.c.s.ClusterSettings ] [node-1] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
[2024-05-29T06:12:45,932][INFO ][o.o.a.c.HashRing ] [node-1] Node added: [gUTqYPwWTYuAjPc7INR1AQ]
[2024-05-29T06:12:45,941][INFO ][o.o.a.c.HashRing ] [node-1] Add data node to AD version hash ring: gUTqYPwWTYuAjPc7INR1AQ
[2024-05-29T06:12:45,944][INFO ][o.o.a.c.HashRing ] [node-1] All nodes with known AD version: {gUTqYPwWTYuAjPc7INR1AQ=ADNodeInfo{version=2.14.0, isEligibleDataNode=true}}
[2024-05-29T06:12:45,944][INFO ][o.o.a.c.HashRing ] [node-1] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
[2024-05-29T06:12:45,945][INFO ][o.o.a.c.HashRing ] [node-1] Build AD version hash ring successfully
[2024-05-29T06:12:45,955][INFO ][o.o.a.c.ADDataMigrator ] [node-1] Start migrating AD data
[2024-05-29T06:12:45,956][INFO ][o.o.a.c.ADDataMigrator ] [node-1] AD job index doesn't exist, no need to migrate
[2024-05-29T06:12:45,957][INFO ][o.o.a.c.ADClusterEventListener] [node-1] Init AD version hash ring successfully
[2024-05-29T06:12:46,138][INFO ][o.o.g.GatewayService ] [node-1] recovered [9] indices into cluster_state
[2024-05-29T06:12:46,142][WARN ][o.o.o.i.ObservabilityIndex] [node-1] message: index [.opensearch-observability/epbDFU8bRjKoeklCfcOQjw] already exists
[2024-05-29T06:12:46,145][INFO ][o.o.s.l.LogTypeService ] [node-1] Loading builtin types!
[2024-05-29T06:12:46,158][INFO ][o.o.s.l.LogTypeService ] [node-1] Indexing [429] fieldMappingDocs from logTypes: 24
[2024-05-29T06:12:46,216][WARN ][o.o.c.r.a.AllocationService] [node-1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-05-29T06:12:46,243][WARN ][o.o.s.SecurityAnalyticsPlugin] [node-1] Failed to initialize LogType config index and builtin log types
[2024-05-29T06:12:46,486][INFO ][o.o.p.PluginsService ] [node-1] PluginService:onIndexModule index:[security-auditlog-2024.05.27/a41dnTV_Q1WbjT5m5kLkFQ]
[2024-05-29T06:12:46,711][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-05-29T06:12:46,758][INFO ][o.o.s.l.LogTypeService ] [node-1] Loading builtin types!
[2024-05-29T06:12:46,759][INFO ][o.o.s.l.LogTypeService ] [node-1] Indexing [429] fieldMappingDocs from logTypes: 24
[2024-05-29T06:12:46,774][INFO ][o.o.s.l.LogTypeService ] [node-1] Loading builtin types!
[2024-05-29T06:12:46,775][INFO ][o.o.s.l.LogTypeService ] [node-1] Indexing [429] fieldMappingDocs from logTypes: 24
[2024-05-29T06:12:46,784][INFO ][o.o.s.i.DetectorIndexManagementService] [node-1] info deleteOldIndices
[2024-05-29T06:12:46,831][INFO ][o.o.s.i.DetectorIndexManagementService] [node-1] No Old Correlation Indices to delete
[2024-05-29T06:12:46,861][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-05-29T06:12:46,879][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-05-29T06:12:46,883][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-05-29T06:12:47,068][INFO ][o.o.p.PluginsService ] [node-1] PluginService:onIndexModule index:[.ql-datasources/oHAcZajsSSeHxs4uTZLwPA]
[2024-05-29T06:12:47,099][INFO ][o.o.p.PluginsService ] [node-1] PluginService:onIndexModule index:[security-auditlog-2024.05.23/VnvAk2epSLmZseuILJKmJQ]
[2024-05-29T06:12:47,142][INFO ][o.o.p.PluginsService ] [node-1] PluginService:onIndexModule index:[.opendistro_security/TinVHLoeQ86cRvuFJFhldw]
[2024-05-29T06:12:47,289][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Index .opendistro_security already exists
[2024-05-29T06:12:47,290][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Node started, try to initialize it. Wait for at least yellow cluster state....
[2024-05-29T06:12:47,997][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'config' with /etc/opensearch/opensearch-security/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2024-05-29T06:12:48,002][WARN ][o.o.c.r.a.AllocationService] [node-1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-05-29T06:12:48,134][INFO ][o.o.p.PluginsService ] [node-1] PluginService:onIndexModule index:[.kibana_92668751_admin_1/JB3rSwxFQ1KDOJamqoQClw]
[2024-05-29T06:12:48,189][INFO ][o.o.p.PluginsService ] [node-1] PluginService:onIndexModule index:[.plugins-ml-config/wajy09egT4i0cnsklOOljA]
[2024-05-29T06:12:48,209][INFO ][o.o.p.PluginsService ] [node-1] PluginService:onIndexModule index:[.opensearch-observability/epbDFU8bRjKoeklCfcOQjw]
[2024-05-29T06:12:48,231][INFO ][o.o.p.PluginsService ] [node-1] PluginService:onIndexModule index:[.kibana_1/RPWj0jLvRImPKbkxV1vC-w]
[2024-05-29T06:12:48,388][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-05-29T06:12:48,393][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-05-29T06:12:48,397][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-05-29T06:12:48,420][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-05-29T06:12:48,452][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id config, skipping update.
[2024-05-29T06:12:48,452][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'roles' with /etc/opensearch/opensearch-security/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2024-05-29T06:12:48,509][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id roles, skipping update.
[2024-05-29T06:12:48,509][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'rolesmapping' with /etc/opensearch/opensearch-security/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2024-05-29T06:12:48,543][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id rolesmapping, skipping update.
[2024-05-29T06:12:48,545][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'internalusers' with /etc/opensearch/opensearch-security/internal_users.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2024-05-29T06:12:48,605][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id internalusers, skipping update.
[2024-05-29T06:12:48,606][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'actiongroups' with /etc/opensearch/opensearch-security/action_groups.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2024-05-29T06:12:48,625][WARN ][o.o.c.r.a.AllocationService] [node-1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-05-29T06:12:48,634][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id actiongroups, skipping update.
[2024-05-29T06:12:48,634][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'tenants' with /etc/opensearch/opensearch-security/tenants.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2024-05-29T06:12:48,656][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id tenants, skipping update.
[2024-05-29T06:12:48,656][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'nodesdn' with /etc/opensearch/opensearch-security/nodes_dn.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2024-05-29T06:12:48,662][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id nodesdn, skipping update.
[2024-05-29T06:12:48,663][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'whitelist' with /etc/opensearch/opensearch-security/whitelist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2024-05-29T06:12:48,678][INFO ][o.o.p.PluginsService ] [node-1] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/xqOhNr2hRbeK-9ouF63kMw]
[2024-05-29T06:12:48,690][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id whitelist, skipping update.
[2024-05-29T06:12:48,691][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'allowlist' with /etc/opensearch/opensearch-security/allowlist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2024-05-29T06:12:48,696][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id allowlist, skipping update.
[2024-05-29T06:12:48,696][INFO ][o.o.s.s.ConfigHelper ] [node-1] Will update 'audit' with /etc/opensearch/opensearch-security/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2024-05-29T06:12:48,756][INFO ][o.o.s.s.ConfigHelper ] [node-1] Index .opendistro_security already contains doc with id audit, skipping update.
[2024-05-29T06:12:48,955][INFO ][o.o.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.opensearch-sap-log-types-config][0]]]).
[2024-05-29T06:12:49,005][WARN ][o.o.c.r.a.AllocationService] [node-1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[2024-05-29T06:12:49,256][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing on REST API is enabled.
[2024-05-29T06:12:49,257][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
[2024-05-29T06:12:49,257][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing on Transport API is enabled.
[2024-05-29T06:12:49,258][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
[2024-05-29T06:12:49,258][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing of request body is enabled.
[2024-05-29T06:12:49,258][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Bulk requests resolution is disabled during request auditing.
[2024-05-29T06:12:49,259][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Index resolution is enabled during request auditing.
[2024-05-29T06:12:49,259][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Sensitive headers auditing is enabled.
[2024-05-29T06:12:49,259][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing requests from kibanaserver users is disabled.
[2024-05-29T06:12:49,261][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing request headers <NONE> is disabled.
[2024-05-29T06:12:49,263][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing request url params <NONE> is disabled.
[2024-05-29T06:12:49,269][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[2024-05-29T06:12:49,271][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing of external configuration is disabled.
[2024-05-29T06:12:49,271][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing of internal configuration is enabled.
[2024-05-29T06:12:49,276][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing only metadata information for read request is enabled.
[2024-05-29T06:12:49,277][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing will watch {} for read requests.
[2024-05-29T06:12:49,278][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing read operation requests from kibanaserver users is disabled.
[2024-05-29T06:12:49,278][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing only metadata information for write request is enabled.
[2024-05-29T06:12:49,278][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing diffs for write requests is disabled.
[2024-05-29T06:12:49,278][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing write operation requests from kibanaserver users is disabled.
[2024-05-29T06:12:49,279][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing will watch <NONE> for write requests.
[2024-05-29T06:12:49,279][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] .opendistro_security is used as internal security index.
[2024-05-29T06:12:49,279][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Internal index used for posting audit logs is null
[2024-05-29T06:12:49,280][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Hot-reloading of audit configuration is enabled
[2024-05-29T06:12:49,280][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Node 'node-1' initialized
[2024-05-29T06:12:50,588][WARN ][o.o.p.c.u.JsonConverter ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-05-29T06:12:55,590][WARN ][o.o.p.c.u.JsonConverter ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-05-29T06:12:55,870][INFO ][o.o.m.a.MLModelAutoReDeployer] [node-1] Index not found, not performing auto reloading!
[2024-05-29T06:12:55,871][INFO ][o.o.m.c.MLCommonsClusterManagerEventListener] [node-1] Starting ML sync up job...
[2024-05-29T06:13:00,586][WARN ][o.o.p.c.u.JsonConverter ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-05-29T06:13:05,590][WARN ][o.o.p.c.u.JsonConverter ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-05-29T06:13:05,939][INFO ][o.o.m.c.MLSyncUpCron ] [node-1] ML configuration already initialized, no action needed
[2024-05-29T06:13:10,590][WARN ][o.o.p.c.u.JsonConverter ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-05-29T06:13:15,588][WARN ][o.o.p.c.u.JsonConverter ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-05-29T06:13:20,590][WARN ][o.o.p.c.u.JsonConverter ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-05-29T06:13:25,598][WARN ][o.o.p.c.u.JsonConverter ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
