How to do Multinode opensearch setup using docker compose by multiple machines :

OpenSearch
, ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue: i want to make Opensearch cluster using 3 machines in same network , what are the customizations should i do ,

Configuration:

version: '3'
services:
  opensearch-master1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-master1
    environment:
      # - OPENSEARCH_SECURITY_ADMIN_PASSWORD=@Mine1623
      - cluster.name=opensearch-cluster
      - node.name=opensearch-master1
      - discovery.seed_hosts=opensearch-master1,opensearch-master2,opensearch-master3
      - cluster.initial_cluster_manager_nodes=opensearch-master1,opensearch-master2,opensearch-master3
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
      - "DISABLE_SECURITY_PLUGIN=true"  # Disable security plugin
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}

    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch_opensearch-master-data1:/usr/share/opensearch/data
        #- ./certs:/usr/share/opensearch/config/certs
    ports:
      - 9201:9200
      - 9601:9600
    networks:
      - opensearch-net

  opensearch-master2:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-master2
    environment:
      # - OPENSEARCH_SECURITY_ADMIN_PASSWORD=@Mine1623
      - cluster.name=opensearch-cluster
      - node.name=opensearch-master2
      - discovery.seed_hosts=opensearch-master1,opensearch-master2,opensearch-master3
      - cluster.initial_cluster_manager_nodes=opensearch-master1,opensearch-master2,opensearch-master3
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
      - "DISABLE_SECURITY_PLUGIN=true"  # Disable security plugin
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch_opensearch-master-data2:/usr/share/opensearch/data
        # - ./certs:/usr/share/opensearch/config/certs
    ports:
      - 9202:9200
      - 9602:9600
    networks:
      - opensearch-net

  opensearch-master3:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-master3
    environment:
      # - OPENSEARCH_SECURITY_ADMIN_PASSWORD=@Mine1623
      - cluster.name=opensearch-cluster
      - node.name=opensearch-master3
      - discovery.seed_hosts=opensearch-master1,opensearch-master2,opensearch-master3
      - cluster.initial_cluster_manager_nodes=opensearch-master1,opensearch-master2,opensearch-master3
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
      - "DISABLE_SECURITY_PLUGIN=true"  # Disable security plugin
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch_opensearch-master-data3:/usr/share/opensearch/data
        # - ./certs:/usr/share/opensearch/config/certs
    ports:
      - 9203:9200
      - 9603:9600
    networks:
      - opensearch-net
  # Data Nodes
  opensearch-data1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-data1
    environment:
      #- OPENSEARCH_SECURITY_ADMIN_PASSWORD=@Mine1623
      - cluster.name=opensearch-cluster
      - node.name=opensearch-data1
      - node.roles=data
      - discovery.seed_hosts=opensearch-master1,opensearch-master2,opensearch-master3
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
      - "DISABLE_SECURITY_PLUGIN=true"
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch_opensearch-data1:/usr/share/opensearch/data
        # - ./certs:/usr/share/opensearch/config/certs
    ports:
      - 9204:9200
      - 9604:9600
    networks:
      - opensearch-net

  opensearch-data2:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-data2
    environment:
      # - OPENSEARCH_SECURITY_ADMIN_PASSWORD=@Mine1623
      - cluster.name=opensearch-cluster
      - node.name=opensearch-data2
      - node.roles=data
      - discovery.seed_hosts=opensearch-master1,opensearch-master2,opensearch-master3
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
      - "DISABLE_SECURITY_PLUGIN=true"
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch_opensearch-data2:/usr/share/opensearch/data
        # - ./certs:/usr/share/opensearch/config/certs
    ports:
      - 9205:9200
      - 9605:9600
    networks:
      - opensearch-net

  opensearch-data3:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-data3
    environment:
      # - OPENSEARCH_SECURITY_ADMIN_PASSWORD=@Mine1623
      - cluster.name=opensearch-cluster
      - node.name=opensearch-data3
      - node.roles=data
      - discovery.seed_hosts=opensearch-master1,opensearch-master2,opensearch-master3
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
      - "DISABLE_SECURITY_PLUGIN=true"
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch_opensearch-data3:/usr/share/opensearch/data
        # - ./certs:/usr/share/opensearch/config/certs
    ports:
      - 9206:9200
      - 9606:9600
    networks:
      - opensearch-net



  opensearch-client1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-client1
    environment:
      # - OPENSEARCH_SECURITY_ADMIN_PASSWORD=@Mine1623
      - cluster.name=opensearch-cluster
      - node.name=opensearch-client1
      - node.roles=ingest,remote_cluster_client
      - discovery.seed_hosts=opensearch-master1,opensearch-master2,opensearch-master3
      - cluster.initial_cluster_manager_nodes=opensearch-master1,opensearch-master2,opensearch-master3
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
      - "DISABLE_SECURITY_PLUGIN=true"  # Disable security plugin
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
          # volumes:
     # - opensearch-client-data1:/usr/share/opensearch/data
     #  - ./certs:/usr/share/opensearch/config/certs
    #  - ./opensearch-config/opensearch-client.yml:/usr/share/opensearch/config/opensearch.yml
    ports:
      - 9207:9200
      - 9607:9600
    networks:
      - opensearch-net

  opensearch-client2:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-client2
    environment:
      # - OPENSEARCH_SECURITY_ADMIN_PASSWORD=@Mine1623
      - cluster.name=opensearch-cluster 
      - node.name=opensearch-client2
      - node.roles=ingest,remote_cluster_client
      - discovery.seed_hosts=opensearch-master1,opensearch-master2,opensearch-master3
      - cluster.initial_cluster_manager_nodes=opensearch-master1,opensearch-master2,opensearch-master3
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
      - "DISABLE_SECURITY_PLUGIN=true"  # Disable security plugin
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
          #   volumes:
     # - opensearch-client-data2:/usr/share/opensearch/data
     #  - ./certs:/usr/share/opensearch/config/certs
    ports:
      - 9208:9200
      - 9608:9600
    networks:
      - opensearch-net

  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    environment:
     # - OPENSEARCH_PASSWORD=@Mine1623
      - 'OPENSEARCH_HOSTS=["http://opensearch-master1:9200","http://opensearch-master2:9200","http://opensearch-master3:9200"]'
      - "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true"  # Disable security plugin
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
    ports:
      - 5601:5601
    networks:
      - opensearch-net
#environment:
  #- OPENSEARCH_SECURITY_ADMIN_PASSWORD=@VMware12

volumes:
  opensearch_opensearch-master-data1:
  opensearch_opensearch-master-data2:
  opensearch_opensearch-master-data3:
  opensearch_opensearch-data1:
  opensearch_opensearch-data2:
  opensearch_opensearch-data3:
 # opensearch-client-data1:
  #opensearch-client-data2:


networks:
  opensearch-net:

Relevant Logs or Screenshots:

image
image1366×768 243 KB

nodes are not aware of each other because 3 are in 1 machine , 3 are in 2nd machine and 3 are in other .

2

@Govind12 I’ve used your exact docker-compose and the cluster came up with no issues.

image
image1545×410 79.8 KB

image
image2002×204 115 KB

3

Yes its working , but i want to make a cluster by using 3 machines which are in same network , how these container will communicate to the containers , which are in different host machine??

i want that , 1 data node 1 master node and 1 client node in each machine and then they should make a cluster , also all the containers should visible in dashboard , i want to make highly available cluster .

4

@Govind12 In that case you’re talking about docker swarm which is not supported as far as I remember.
I suggest Kubernetes and Helm for a distributed environment.

1 Like
5

How to do data syncing in distributed environment ?

6

@Govind12 As you may know, data in OpenSearch are kept in shards that are placed on different Opensearch data nodes. All OpenSearch nodes have access to all data within the same cluster. OpenSearch is not aware of the underlying container infrastructure i.e. Docker, Kubernetes.
Distributed design regards OpenSearch pods which are managed by Kubernetes.

2 Likes
7

would you recommend any reference for Setup of Opensearch cluster in K8s , a part from the official documentation ?

8

@Govind12 I’ve tested official OpenSearch charts with k3 Rancher and k8s (containerd, calico) on Ubuntu 22.04 LTS.
I had no issues with deployment.

9

@pablo Can you please gave me any source from where I can check officially that is it possible to create multi-node OpenSearch cluster using separate machines via docker swarm, we have decided to implement that on our production environment using docker swarm but I’m trying and facing several issues. as the node’s container can only see it self and not other container which are running on another node. Thanks in advance.

10

@hamza00 According to this GitHub bug, Docker Swarm was never supported and all docker-compose.yml examples were tested and build against a single Docker node.

11

@pablo please check if you can contribute any useful tip in this:

[BUG][Docker Swarm Multi Node Cluster] · Issue #224 · opensearch-project/opensearch-devops · GitHub

12

@pablo @Govind12 I did it not with docker swarm but having separate docker-compose.yml files on each machine and setup works fine for me. Thanks

1 Like