OpenSearch docker-compose error

Hi, I am new to opensearch and appreciate any help to point me in the right direction.

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch/Dashboard: latest version
docker-compose-plugin 2.21.0-1
docker-ce 5:24.0.7-1
Debian 11 bullseye
Safari Version 17.2.1

Describe the issue:
It seems, that the opensearch dashboard can’t connect to the node1 or 2.

Configuration:

My compose file:
version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node1 # Name the node that will run in this container
      - discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligibile to serve as cluster manager
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
      - "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
      - "DISABLE_SECURITY_PLUGIN=true" # Disables Security plugin
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    volumes:
      - '${DATA}/opensearch-data1:/usr/share/opensearch/data' # Creates volume called opensearch-data1 and mounts it to the container
    ports:
      - 9200:9200 # REST API
      - 9600:9600 # Performance Analyzer
    networks:
      - opensearch-net # All of the containers will join the same Docker bridge network
  opensearch-node2:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node2 # Name the node that will run in this container
      - discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligibile to serve as cluster manager
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
      - "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
      - "DISABLE_SECURITY_PLUGIN=true" # Disables Security plugin
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    volumes:
      - '${DATA}/opensearch-data2:/usr/share/opensearch/data' # Creates volume called opensearch-data2 and mounts it to the container
    networks:
      - opensearch-net # All of the containers will join the same Docker bridge network
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601 # Map host port 5601 to container port 5601
    expose:
      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
    environment:
      - 'OPENSEARCH_HOSTS=["http://opensearch-node1:9200","http://opensearch-node2:9200"]'
      - "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

Relevant Logs or Screenshots:

opensearch-node2  | Disabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
opensearch-node2  | Disabling OpenSearch Security Plugin
opensearch-node2  | Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
opensearch-node2  | WARNING: A terminally deprecated method in java.lang.System has been called
opensearch-node2  | WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.11.1.jar)
opensearch-node2  | WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch

I assume this is a copying error.

The warnings are expected and don’t affect the OpenSearch node’s startup process. What other errors do you see?

I’ve tested your docker-compose and all worked.

Are the OpenSearch containers still running? Can you connect with curl?

Sorry, it was a copy and paste issue only to the forum, not in the installation. I will edit the compose file in the post. OpenSearch container still showing the error message. And the OpenSearch-dashboard shows:

opensearch-dashboards  | {"type":"log","@timestamp":"2024-01-18T06:13:02Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.27.0.3:9200"}

My host network is: 192.168.1.0/24

# curl https://172.27.0.3:9200 -ku 'admin:admin'
curl: (7) Failed to connect to 172.27.0.3 port 9200: Connection refused

The Containers are up and running.

# docker container ls | grep opensearch
c3903bddf482   opensearchproject/opensearch-dashboards:latest   "./opensearch-dashbo…"   16 hours ago   Up 42 minutes           0.0.0.0:5601->5601/tcp                                               opensearch-dashboards
da611c7583ef   opensearchproject/opensearch:latest              "./opensearch-docker…"   16 hours ago   Up 42 minutes           0.0.0.0:9200->9200/tcp, 9300/tcp, 0.0.0.0:9600->9600/tcp, 9650/tcp   opensearch-node1
d18d4ce3acf2   opensearchproject/opensearch:latest              "./opensearch-docker…"   16 hours ago   Up 42 minutes           9200/tcp, 9300/tcp, 9600/tcp, 9650/tcp                               opensearch-node2

# docker network inspect opensearch_opensearch-net
[
    {
        "Name": "opensearch_opensearch-net",
        "Id": "1eae8748c38bbc611b85fb785d49b966dcbce2313efd9e73dee2bcdc21496b4a",
        "Created": "2024-01-17T15:57:09.412505824+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.27.0.0/16",
                    "Gateway": "172.27.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "c3903bddf48235b89f3c5e8c016238d125926759fb740217e62462b4da9ec037": {
                "Name": "opensearch-dashboards",
                "EndpointID": "6b08de37056b576fb6ba055bb18ff86285a97aec6c6dc97f31022c02039b81ef",
                "MacAddress": "02:42:ac:1b:00:04",
                "IPv4Address": "172.27.0.4/16",
                "IPv6Address": ""
            },
            "d18d4ce3acf293d257c186c1f5079d3dd4a61ccb46551b177e93749ed6eb732d": {
                "Name": "opensearch-node2",
                "EndpointID": "ed704573c38531e81032aeb6a9a52096f04a635c1a7e8a88d58ef979fcbe3729",
                "MacAddress": "02:42:ac:1b:00:02",
                "IPv4Address": "172.27.0.2/16",
                "IPv6Address": ""
            },
            "da611c7583ef5cdbc42b949ab71592e2810428bd36c0f75b98c8281b9b5548f4": {
                "Name": "opensearch-node1",
                "EndpointID": "7bfea1d8d9db7c2159a63e30229b77ffcd460533338e5bb9667cf72fcf3e603a",
                "MacAddress": "02:42:ac:1b:00:03",
                "IPv4Address": "172.27.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "opensearch-net",
            "com.docker.compose.project": "opensearch",
            "com.docker.compose.version": "2.21.0"
        }
    }
]

more log details of node 1:

[2024-01-18T07:12:52,535][INFO ][o.o.e.ExtensionsManager  ] [opensearch-node1] ExtensionsManager initialized
[2024-01-18T07:12:52,644][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-node1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: OpenSearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/opensearch/data/nodes];
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.11.1.jar:2.11.1]
	at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.11.1.jar:2.11.1]
Caused by: org.opensearch.OpenSearchException: failed to bind service
	at org.opensearch.node.Node.<init>(Node.java:1260) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.node.Node.<init>(Node.java:407) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.11.1.jar:2.11.1]
	... 6 more
Caused by: java.nio.file.AccessDeniedException: /usr/share/opensearch/data/nodes
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397) ~[?:?]
	at java.nio.file.Files.createDirectory(Files.java:700) ~[?:?]
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) ~[?:?]
	at java.nio.file.Files.createDirectories(Files.java:793) ~[?:?]
	at org.opensearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:325) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:262) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.env.NodeEnvironment.<init>(NodeEnvironment.java:323) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.node.Node.<init>(Node.java:525) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.node.Node.<init>(Node.java:407) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.11.1.jar:2.11.1]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.11.1.jar:2.11.1]
	... 6 more
uncaught exception in thread [main]
OpenSearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/opensearch/data/nodes];
Likely root cause: java.nio.file.AccessDeniedException: /usr/share/opensearch/data/nodes
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
	at java.base/java.nio.file.Files.createDirectory(Files.java:700)
	at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
	at java.base/java.nio.file.Files.createDirectories(Files.java:793)
	at org.opensearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:325)
	at org.opensearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:262)
	at org.opensearch.env.NodeEnvironment.<init>(NodeEnvironment.java:323)
	at org.opensearch.node.Node.<init>(Node.java:525)
	at org.opensearch.node.Node.<init>(Node.java:407)
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch-cluster.log

END OF LINE

Running the container with UID=1000 and GID=1000 enviroment variables worked for me.

@Meadows Thanks for sharing the solution.
The other option would be changing the owner of all files in the docker folder to user:group with ID 1000. This would translate to the opensearch user and group in the docker container.