Docker Image 3.1.1 doesn't seem to work

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

opensearchproject/opensearch:3.1.0

I’ve been scratching my head all day as to why my extremely basic docker based config wouldn’t work. By basic I mean just using the example docker compose file in the documentation. It was working yesterday, but today it didn’t.

Then I realised I was pulling opensearchproject/opensearch:latest, and when I check the Docker Hub, the images tagged 3.1.0 was published last night.

I reverted to tag 3.0.0 and everything works fine again, I’m presuming there is something not quite right in the 3.1.0 tagged image. My guess is certificates as I couldn’t get the dashboard nodes to connect to the data nodes, and even basic curl requests to the data nodes were being rejected.

@Crickes I’ve just used docker-compose.yml from the documentation and successfully deployed clusters using latest and 3.1.0 tags.

What exactly didn’t work for you? Please share your errors and startup logs.

Hi Pablo,

You’re correct in that the unmodified docker-compose file does work, but that has the DISABLE_INSTALL_DEMO_CONFIG parameter set to true. If you comment this out and the line that disables the security plugin, the nodes seem to start up but are in complete isolation from each other.

Here is the docker compose file I am testing with. Currently with the 3.0.0 image, the cluster starts.

services:
  opensearch-node1:
    image: opensearchproject/opensearch:3.0.0
    # image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node1 # Name the node that will run in this container
      - discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligibile to serve as cluster manager
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}    # Sets the demo admin user password when using demo configuration, required for OpenSearch 2.12 and later
      # - "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
      # - "DISABLE_SECURITY_PLUGIN=true" # Disables Security plugin
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
    ports:
      - 9200:9200 # REST API
      - 9600:9600 # Performance Analyzer
    networks:
      - opensearch-net # All of the containers will join the same Docker bridge network
  opensearch-node2:
    image: opensearchproject/opensearch:3.0.0
    # image: opensearchproject/opensearch:latest
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node2 # Name the node that will run in this container
      - discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligibile to serve as cluster manager
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}    # Sets the demo admin user password when using demo configuration, required for OpenSearch 2.12 and later

      # - "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
      # - "DISABLE_SECURITY_PLUGIN=true" # Disables Security plugin
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data # Creates volume called opensearch-data2 and mounts it to the container
    networks:
      - opensearch-net # All of the containers will join the same Docker bridge network
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:3.0.0
    # image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601 # Map host port 5601 to container port 5601
    expose:
      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
    environment:
      - 'OPENSEARCH_HOSTS=["http://opensearch-node1:9200","http://opensearch-node2:9200"]'
      - "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

If I change the image to use the latest tag, then the 3 nodes seem to come up, but that can’t talk to each other which I suspect is an issue with generating the demo certificates.

On deploying using the latest tag, the output stops here:

Enabling OpenSearch Security Plugin

Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin 

OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user. 

Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string. 

If a password is not provided, the setup will quit. 

 For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/

### OpenSearch Security Demo Installer

### ** Warning: Do not use on production or public reachable systems **

OpenSearch install type: rpm/deb on Linux 4.18.0-553.33.1.el8_10.x86_64 amd64

OpenSearch config dir: /usr/share/opensearch/config/

OpenSearch config file: /usr/share/opensearch/config/opensearch.yml

OpenSearch bin dir: /usr/share/opensearch/bin/

OpenSearch plugins dir: /usr/share/opensearch/plugins/

OpenSearch lib dir: /usr/share/opensearch/lib/

Detected OpenSearch Version: 3.1.0

Detected OpenSearch Security Version: 3.1.0.0

Admin password set successfully.

### Success

### Execute this script now on all your nodes and then start all nodes

### OpenSearch Security will be automatically initialized.

### If you like to change the runtime configuration 

### change the files in ../../../config/opensearch-security and execute: 

sudo "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/config/opensearch-security" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv

### or run ./securityadmin_demo.sh

### To use the Security Plugin ConfigurationGUI

### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/<your-custom-admin-password>.

### (Ignore the SSL certificate warning because we installed self-signed demo certificates)

Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin

WARNING: Using incubator modules: jdk.incubator.vector

WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens


Full appreciate this may be a picnic problem (problem in chair, not in code/computer). I’m a long time user of Elasticsearch and needing to migrate a huge project into OpenSearch. I’m very grateful for any pointers or advice.

@Crickes I’ve tested this Docker Compose setup. With the current configuration, you will encounter SSL errors.

The root cause lies in your OpenSearch Dashboards configuration.
When you enabled the security plugin in OpenSearch nodes, you didn’t enable it in OpenSearch Dashboards. As a result, OpenSearch Dashboards is attempting to communicate with both OpenSearch nodes over HTTP instead of HTTPS.

Since both nodes are listed in the OPENSEARCH_HOSTS environment variable, OpenSearch Dashboards uses round-robin requests. This causes errors to appear on both OpenSearch nodes almost simultaneously.

It may appear that the cluster has not formed, but if you run the following command against the cluster, you’ll see that both nodes are actually part of the same cluster

curl --insecure -u admin:<password> -XGET https://<OpenSearch_node_IP_or_FQDN>:9200/_cat/nodes

This is a fixed OpenSearch Dashboards configuration.

  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:3.0.0
    # image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601 # Map host port 5601 to container port 5601
    expose:
      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
    environment:
      - 'OPENSEARCH_HOSTS=["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
    networks:
      - opensearch-net

Please be aware that the same OpenSearch documentation page contains two docker compose exmples. One is with security plugin enabled and the other one with disabled.

@Crickes Regarding the container freeze with latest tag, do you have enough disk space on your docker host?

I would expect that to work at it’s using the 3.0.0. image.

I’ve started from scratch and I’m using the Installation Quickstart guide: https://docs.opensearch.org/docs/latest/getting-started/quickstart/

I’ve copied the docker-compose.yml file from the linked GIT page:

If I leave the images tags set to latest, the cluster never forms and a curl request to the nodes replies with:

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:9200 

However, if all I do is change the image tags to version 3.0.0, the nodes start, the cluster forms and I get a good reply from the cluster.

This has to be an issue with the script that generates the demo certificates for a security enabled cluster.

@Crickes what versions are you using that are tagged as latest

You can verify this using docker inspect opensearchproject/opensearch:latest | grep version

3.1.0

@Crickes The demo certificates are not generated during the docker deployment.
They are hardcoded.

Just to be clear. You’re running docker compose with 3.0.0 and all is working.
However, when you swtich to 3.1.0, nodes are going up but you’re getting the error and cluster is not formed. So you no longer get freeze during the start-up process.

If that’s the case, could you share the full startup logs from one of the OpenSearch nodes?

How did you deploy this cluster? Is it running on a virtual machine, Mac, Windows?

Have you compare digest between local image and the one in Docker Hub?

docker images --digests

I tried running the same docker compose on a different server (actually my laptop), and its start up flawlessly. So this is obviously an issues with my local environment on the server I am trying to use. It’s weird as I though using Docker was meant to prevent such issues.

Anyway, this is clearly not an OpenSearch fault but something weird happening or not happening on my server.

Thanks for your help so far.

Same is happening with me. Looking at your resolution, changing to tag 3.0.0 worked for me as well. My server is AlmaLinux 8.10.

@Crickes @SpecCobra Have you checked disk space on your failing server?
Have you tried deleting OpenSearch image with the latest tag and pulling again?

What is your docker version?

I am also on AlmaLinux v8.10

Client: Docker Engine - Community
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:15:16 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:12 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

To answer your questions, when attempting to deploy 3.1.0, I get no error messages, the start up of Opensearch just hangs. The output log I see is pasted in a post above, but it always stops and hangs after these two lines are output:

WARNING: Using incubator modules: jdk.incubator.vector

WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens

Looking at the 3.0.0 startup logs, the same lines appear but after about a second, we see the normal node start up logs as the node starts to come up and discover things.

My host server is a virtual machine on VMWare running Alma 8.10.

@Crickes Does your VM has enough space in the VMware datastore? Did you set your hard drive to thin or thick provisioning?
Maybe your OS has enough free space, but the backing datastore is full.

Have you tested with another clean Alma VM?

Same here with a multi nodes cluster.

Hang with:

(...)
Enabling OpenSearch Security Plugin
Disabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens

I don’t know where and how but one time , I’ve seen:

Error: Could not find or load main class Cannot
Caused by: java.lang.ClassNotFoundException: Cannot

Hi Pablo, thanks for actively discussing this issue. I have more than 70G of disk space available on the server and also tried deleting and pulling the latest tag. My docker version is “podman version 4.9.4-rhel”.

My logs are getting stuck at the same part as @Crickes:

WARNING: Using incubator modules: jdk.incubator.vector
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens

Same here, multi node cluster.
On 3.0.0, every node started shortly
On 3.1.0, I restarted the docker swarm at 6h40pm, every nodes stay blocked here:

WARNING: Using incubator modules: jdk.incubator.vector
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
Enabling OpenSearch Security Plugin
Disabling execution of install_demo_configuration.sh for OpenSearch Security Plugin

The next first lines of boot appears long time ago:
data01 8h24pm
data02 8h11pm
data03 7h27pm
man01 8h24pm
man02 8h08pm
man03 6h40pm
coor01 9h22pm

ok servers may have a lake of storage…

so I tried on docker playground

$ date && docker run --rm \
-e "DISABLE_SECURITY_PLUGIN=true" \
-e "DISABLE_INSTALL_DEMO_CONFIG=true" \
-e "discovery.type=single-node" \
opensearchproject/opensearch:3.1.0
Fri Jul  4 07:23:50 UTC 2025
Unable to find image 'opensearchproject/opensearch:3.1.0' locally
3.1.0: Pulling from opensearchproject/opensearch
b32084d69388: Pull complete 
ddc38bf49e93: Pull complete 
3af532ae6979: Pull complete 
fd38a77f9c77: Pull complete 
4f4fb700ef54: Pull complete 
01cedce40074: Pull complete 
a4f0e2153528: Pull complete 
Digest: sha256:44cf7770dd9ba55ab7ca4d08319a74952e92dbd1ffd872f78ebbbe799f15ca2d
Status: Downloaded newer image for opensearchproject/opensearch:3.1.0
Disabling OpenSearch Security Plugin
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
[2025-07-04T07:27:43,420][WARN ][stderr                   ] [a803f9f6cc04] Jul 04, 2025 7:27:43 AM org.opensearch.javaagent.bootstrap.AgentPolicy setPolicy

3.1.0 take 5min to pull and boot while 3.0.0 take less than 1min to pull image en boot:

[node1] (local) root@192.168.0.68 ~
$ date && docker run --rm -e "DISABLE_SECURITY_PLUGIN=true" -e "DISABLE_INSTALL_DEMO_CONFIG=true" -e "discovery.type=single-node" opensearchproject/opensearch:3.0.0
Fri Jul  4 07:36:13 UTC 2025
Unable to find image 'opensearchproject/opensearch:3.0.0' locally
3.0.0: Pulling from opensearchproject/opensearch
1c3112c87ab2: Pull complete 
ff4865a50634: Pull complete 
7d604cfa63b2: Pull complete 
2315e093daf2: Pull complete 
4f4fb700ef54: Pull complete 
77ca5659ae40: Pull complete 
c60d2de8f758: Pull complete 
Digest: sha256:00565f63fa1aa1643d5a2cf61cb5f09b4aeddee2f12703f01c534f051bff1a99
Status: Downloaded newer image for opensearchproject/opensearch:3.0.0
Disabling OpenSearch Security Plugin
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
[2025-07-04T07:37:02,013][WARN ][stderr                   ] [1dbd5a7a3dcf] Jul 04, 2025 7:37:01 AM org.opensearch.javaagent.bootstrap.AgentPolicy setPolicy

Does 3.1.0 have a different hardware requirement compare to 3.0.0?
Thank you!

@Crickes, I’ve tested your scenario with AlmaLinux 8.10. I’ve got the same behaviour. The startup froze after the reported warning message.

WARNING: Using incubator modules: jdk.incubator.vector

WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens

As per your tests, OpenSearch 3.0.0 worked just fine.

I’ve also found that the issue is not related to the Docker engine version, as I used your version and the latest.
Next, I’ve installed AlmaLinux 10 and OpenSearch nodes started successfully and formed the cluster.

[pablo@localhost ~]$ curl --insecure -u admin:Eliatra123 https://localhost:9200/_cat/nodes?v
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles                                        cluster_manager name
172.18.0.2           73          83   6    0.13    0.69     0.42 dimr      cluster_manager,data,ingest,remote_cluster_client *               opensearch-node2
172.18.0.3           52          83   6    0.13    0.69     0.42 dimr      cluster_manager,data,ingest,remote_cluster_client -               opensearch-node1
[pablo@localhost ~]$ cat /etc/*release
AlmaLinux release 10.0 (Purple Lion)
NAME="AlmaLinux"
VERSION="10.0 (Purple Lion)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="10.0"
PLATFORM_ID="platform:el10"
PRETTY_NAME="AlmaLinux 10.0 (Purple Lion)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:10::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
VENDOR_NAME="AlmaLinux"
VENDOR_URL="https://almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-10"
ALMALINUX_MANTISBT_PROJECT_VERSION="10.0"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="10.0"
SUPPORT_END=2035-06-01
AlmaLinux release 10.0 (Purple Lion)
AlmaLinux release 10.0 (Purple Lion)
[pablo@localhost ~]$ 

It’s really a bad news.

So all updates are freezed on our cluster.

Any workaround ? Kernel version ?

java is waiting for something:

[<0>] futex_wait_queue_me+0xa3/0x100
[<0>] futex_wait+0x11f/0x210
[<0>] do_futex+0x2f9/0x4d0
[<0>] __x64_sys_futex+0x145/0x1f0
[<0>] do_syscall_64+0x5b/0x1b0
[<0>] entry_SYSCALL_64_after_hwframe+0x61/0xc6