How can I run custom OpenSearch in Docker? Or, can I install a plugin?

OpenSearch
1

I cloned the repository at opensearch
and customized the code.

Q:

• How can I run this in Docker?

• Alternatively, how can I install plugins such as the CCR plugin?

Either solution would be helpful. I would greatly appreciate your help.

Within this repository, there are multiple DockerFiles and Docker-Compose files, and ./gradlew run also works.

Problems:

• However, when running the project, the plugins are missing, so I need to install them. The official documentation provides a method for installing plugins when running with Docker, but I’m unsure how to do it when using ./gradlew run.

• When I launch the provided Docker-Compose, no logs appear, so I cannot confirm if my changes have been applied properly. Even when I check the log folder, no logs appear, unlike with a deployed OpenSearch instance.

# Only used for testing the docker images
version: '3'
services:
  opensearch-1:
    image: opensearch:test
    environment:
       - node.name=opensearch-1
       - cluster.initial_cluster_manager_nodes=opensearch-1,opensearch-2
       - discovery.seed_hosts=opensearch-2:9300
       - cluster.name=opensearch
       - bootstrap.memory_lock=true
       - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
       - path.repo=/tmp/opensearch-repo
       - node.attr.testattr=test
       - cluster.routing.allocation.disk.watermark.low=1b
       - cluster.routing.allocation.disk.watermark.high=1b
       - cluster.routing.allocation.disk.watermark.flood_stage=1b
       - node.store.allow_mmap=false
    volumes:
       - ./build/repo:/tmp/opensearch-repo
       - ./build/logs/1:/usr/share/opensearch/logs
    ports:
      - "9200"
    ulimits:
      memlock:
        soft: -1
        hard: -1
  opensearch-2:
    image: opensearch:test
    environment:
       - node.name=opensearch-2
       - cluster.initial_cluster_manager_nodes=opensearch-1,opensearch-2
       - discovery.seed_hosts=opensearch-1:9300
       - cluster.name=opensearch
       - bootstrap.memory_lock=true
       - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
       - path.repo=/tmp/opensearch-repo
       - node.attr.testattr=test
       - cluster.routing.allocation.disk.watermark.low=1b
       - cluster.routing.allocation.disk.watermark.high=1b
       - cluster.routing.allocation.disk.watermark.flood_stage=1b
       - node.store.allow_mmap=false
    volumes:
       - ./build/repo:/tmp/opensearch-repo
       - ./build/logs/2:/usr/share/opensearch/logs
    ports:
      - "9200"
    ulimits:
      memlock:
        soft: -1
        hard: -1
2

@sbilac Could you share the docker compose command that you’ve used?

Does it mean that you use a custom docker repository?

1 Like
3

No, that’s the opensearch distribution/docker/docker-compose.yml file.

Should I build the image using a Dockerfile for the tar.gz I assembled?

I am looking for the source code related to this.

Thank you for your comment.

4

@sbilac I think you could use a docker-compose.yml and Dockerfile to build an image with required plugin. However, this will pull the plugin every time when you run docker-compose.

Alternatively you could build an image with a plugin and push it to your remote or local repository and then point docker-compose to that custom image.

1 Like
5

thank you so much!
It was really helpful :blush: :blush: :blush: :blush: :blush:

6

Hello, @pablo!

Thanks to your help, I successfully installed a custom OpenSearch and plugins using the Dockerfile and docker-compose. I followed the first method you suggested, and I want to thank you again for your assistance.

During this process, I came up with another question and wanted to ask if it’s okay to inquire.

“Even though I build the Dockerfile with plugins included, I still need to install the plugins in a new space when I run docker-compose.” I’m curious why this happens.

Could you explain why containers in docker-compose can’t access the host’s volume?

Additionally, when I tried running two clusters with docker-compose, it seemed like the clusters couldn’t share the same plugin volume. I’m curious why this is the case as well.

Thank you!

7

@sbilac What do you mean by “new space”?

Your volume mounting in docker-compose.yml looks correct and should map the host folder to your container one.

Did you run two separate docker-compose.yml files or had all the clusters in one file?
What errors did you see?

8

I’m sorry. It seems my environment was misconfigured. After deleting the Docker-related files and trying again, I was able to confirm that mounting the volume as you suggested was the correct approach. I successfully ran it using the docker-compose.yml. I included two clusters within a single docker-compose.yml file. Thank you for your help.

1 Like
9

@sbilac I am trying to do the same thing. If you don’t mind then could you please share what steps you performed in the Dockerfile?

I am pretty new with Dockerfile.

10

@pbrar Hello, I referred to the following file: opensearch-build/docker/release/dockerfiles/opensearch.al2023.dockerfile at 89ccd0936fc4d2f813aa876ffda3e7f04a7793b8 · opensearch-project/opensearch-build · GitHub. I hope this helps as a reference for you.

11

@sbilac thank you for sharing the link. I hope you have a great day / night!

1 Like
12

@pbrar Additionally, please refer to the description of this PR for the files required to execute the Dockerfile: [BUGFIX] Fix missing fields to resolve Strict Dynamic Mapping issue when saving task result by inpink · Pull Request #16201 · opensearch-project/OpenSearch · GitHub.

Have a great day!