Error cluster-manager not discovered or elected yet

OpenSearch
1

Versions Opensearch 2.0.6

Describe the issue:
Hi all, after a system crash (VM that hosts Opensearch) of 4 nodes of 8 nodes opensearch cluster, system cannot be restored and continue to propose me a message that cannot find eligible master node, but after crash nodes are already up. This message is present in all nodes. Where is a solution to reset and re-initialize the cluster without index data loss?

Configuration:
8 nodes on docker swarm hosted in 8 Redhat Virtual Machine

Relevant Logs or Screenshots:
[2023-12-20T17:06:12,526][WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node138] cluster-manager not discovered or elected yet, an election requires at least 3 nodes with ids from [DLX4Sb2NQVuUJsk0mki_Cw, MJOjIGghSY2tDvgSKzsXow, C-RQbYb_SEWelFWkfIZ4TA, pRAaw5rVRtyyI12PdpkxBQ, lKWeKuRST8q_PdvHZkbmAQ], have discovered [{opensearch-node138}{MJOjIGghSY2tDvgSKzsXow}{Srqu7VSgSOG5Ica1mw1xYQ}{10.0.4.87}{10.0.4.87:9300}{dimr}{zone=zoneA, shard_indexing_pressure_enabled=true}, {opensearch-node192}{C-RQbYb_SEWelFWkfIZ4TA}{msr8_cebSt2NLzIN0MNhFA}{10.0.4.92}{10.0.4.92:9300}{dimr}{zone=zoneB, shard_indexing_pressure_enabled=true}, {opensearch-node193}{pRAaw5rVRtyyI12PdpkxBQ}{wvDIA94qSPqEhO6vOv-v9A}{10.0.4.96}{10.0.4.96:9300}{dimr}{zone=zoneB, shard_indexing_pressure_enabled=true}, {opensearch-node194}{DLX4Sb2NQVuUJsk0mki_Cw}{JhxEioE6QbmIUY7uRjibOg}{10.0.4.81}{10.0.4.81:9300}{dimr}{zone=zoneB, shard_indexing_pressure_enabled=true}, {opensearch-node195}{lKWeKuRST8q_PdvHZkbmAQ}{7ct5u7F1Sd-ZoQL_7TWpfg}{10.0.4.76}{10.0.4.76:9300}{dimr}{zone=zoneB, shard_indexing_pressure_enabled=true}, {opensearch-node137}{0DEpK09tR4ayvq2IehcSgg}{SyHyfl1bQaCzEjmCoQxx9A}{10.0.4.94}{10.0.4.94:9300}{dimr}{zone=zoneA, shard_indexing_pressure_enabled=true}, {opensearch-node135}{qAxlwiG5Qd6hgObvsE436g}{oycqYEhsTMmyOUy7VteXbg}{10.0.4.85}{10.0.4.85:9300}{dimr}{zone=zoneA, shard_indexing_pressure_enabled=true}, {opensearch-node136}{dnp53JsUQ0qfaRi6y82IkQ}{32uYFHwKTkGO4I026sMqLw}{10.0.4.83}{10.0.4.83:9300}{dimr}{zone=zoneA, shard_indexing_pressure_enabled=true}] which is a quorum; discovery will continue using [10.0.4.84:9300, 10.0.4.82:9300, 10.0.4.93:9300, 10.0.4.86:9300, 10.0.4.91:9300, 10.0.4.95:9300, 10.0.4.80:9300, 10.0.4.75:9300] from hosts providers and [{opensearch-node138}{MJOjIGghSY2tDvgSKzsXow}{Srqu7VSgSOG5Ica1mw1xYQ}{10.0.4.87}{10.0.4.87:9300}{dimr}{zone=zoneA, shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 77, last-accepted version 1708 in term 77

2

Could you please share your opensearch.yml file(s).

thanks,
mj

3

HI, I do not modify openasearch.yml file, I use default docker image with compose, below a subsection that will be repeated for all nodes:

  opensearch-node135:
    image: 192.168.8.136:8083/opensearchproject/opensearch:2.6.0
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node135 # Name the node that will run in this container
      - node.attr.zone=zoneA
      - discovery.seed_hosts=opensearch-node135,opensearch-node136,opensearch-node137,opensearch-node138,opensearch-node192,opensearch-node193,opensearch-node194,opensearch-node195 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=opensearch-node135,opensearch-node136,opensearch-node137,opensearch-node138,opensearch-node192,opensearch-node193,opensearch-node194,opensearch-node195 # Nodes eligibile to serve as cluster manager
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - node.max_local_storage_nodes=8
      - "OPENSEARCH_JAVA_OPTS=-Xms1024m -Xmx1024m" # Set min and max JVM heap sizes to at least 50% of system RAM
      - "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
      - "DISABLE_SECURITY_PLUGIN=true" # Disables Security plugin
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    volumes:
      - /mnt/data/opensearch/data:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
    networks:
      - opensearch_network # All of the containers will join the same Docker bridge network
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.hostname==goa-ers-135]
      restart_policy:
        condition: any
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.opensearchA.entrypoints=opensearch"
        - "traefik.http.routers.opensearchA.rule=Host(`192.168.23.133`)"
        - "traefik.http.services.opensearchA.loadbalancer.server.port=9200"

Thanks for all suggestion.

Corrado.