Nomad-Deployed OpenSearch v3.x Cluster: Nodes Missing after Startup

OpenSearch
1

Versions: opensearch v3.1.0,v3.2.0

We are running a Nomad job with the following setup:

  • 3 master nodes

  • 8 data nodes

  • 2 query nodes

  • 2 ingest nodes

When the cluster starts, the master nodes come up correctly, but 2–4 data nodes are missing, and the query and ingest nodes are often not visible initially. After 10–30 minutes, all nodes eventually join the cluster, and the status turns green.

This issue did not occur with version 2.x. In that version, the cluster would restart and become green within a few minutes, with all nodes present.

Configuration: I use the discovery.seed_hosts parameter on all nodes

Relevant Logs or Screenshots:
On the nodes that are “waiting “

docker logs 6815f13a2d82
Enabling OpenSearch Security Plugin
Disabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
Disabling execution of /usr/share/opensearch/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by net.bytebuddy.dynamic.loading.ClassInjector$UsingUnsafe$Dispatcher$CreationAction
WARNING: Please consider reporting this to the maintainers of class net.bytebuddy.dynamic.loading.ClassInjector$UsingUnsafe$Dispatcher$CreationAction
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release

2

@gvdb Are you seeing any errors in nomad logs? Can you share the nomad configuration file you are using, redact any sensitive details and surround it in code block tags please

4

@gvdb what are you seeing in nomad logs?

6

The strace output of the nodes that were ‘waiting’ contained futex_wait and references to /dev/random.

Installing rng-tools fixed the issue.

2 Likes