Docker Image 3.1.1 doesn't seem to work

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

opensearchproject/opensearch:3.1.0

I’ve been scratching my head all day as to why my extremely basic docker based config wouldn’t work. By basic I mean just using the example docker compose file in the documentation. It was working yesterday, but today it didn’t.

Then I realised I was pulling opensearchproject/opensearch:latest, and when I check the Docker Hub, the images tagged 3.1.0 was published last night.

I reverted to tag 3.0.0 and everything works fine again, I’m presuming there is something not quite right in the 3.1.0 tagged image. My guess is certificates as I couldn’t get the dashboard nodes to connect to the data nodes, and even basic curl requests to the data nodes were being rejected.

@Crickes I’ve just used docker-compose.yml from the documentation and successfully deployed clusters using latest and 3.1.0 tags.

What exactly didn’t work for you? Please share your errors and startup logs.

Hi Pablo,

You’re correct in that the unmodified docker-compose file does work, but that has the DISABLE_INSTALL_DEMO_CONFIG parameter set to true. If you comment this out and the line that disables the security plugin, the nodes seem to start up but are in complete isolation from each other.

Here is the docker compose file I am testing with. Currently with the 3.0.0 image, the cluster starts.

services:
  opensearch-node1:
    image: opensearchproject/opensearch:3.0.0
    # image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node1 # Name the node that will run in this container
      - discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligibile to serve as cluster manager
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}    # Sets the demo admin user password when using demo configuration, required for OpenSearch 2.12 and later
      # - "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
      # - "DISABLE_SECURITY_PLUGIN=true" # Disables Security plugin
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
    ports:
      - 9200:9200 # REST API
      - 9600:9600 # Performance Analyzer
    networks:
      - opensearch-net # All of the containers will join the same Docker bridge network
  opensearch-node2:
    image: opensearchproject/opensearch:3.0.0
    # image: opensearchproject/opensearch:latest
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node2 # Name the node that will run in this container
      - discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligibile to serve as cluster manager
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}    # Sets the demo admin user password when using demo configuration, required for OpenSearch 2.12 and later

      # - "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
      # - "DISABLE_SECURITY_PLUGIN=true" # Disables Security plugin
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data # Creates volume called opensearch-data2 and mounts it to the container
    networks:
      - opensearch-net # All of the containers will join the same Docker bridge network
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:3.0.0
    # image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601 # Map host port 5601 to container port 5601
    expose:
      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
    environment:
      - 'OPENSEARCH_HOSTS=["http://opensearch-node1:9200","http://opensearch-node2:9200"]'
      - "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

If I change the image to use the latest tag, then the 3 nodes seem to come up, but that can’t talk to each other which I suspect is an issue with generating the demo certificates.

On deploying using the latest tag, the output stops here:

Enabling OpenSearch Security Plugin

Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin 

OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user. 

Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string. 

If a password is not provided, the setup will quit. 

 For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/

### OpenSearch Security Demo Installer

### ** Warning: Do not use on production or public reachable systems **

OpenSearch install type: rpm/deb on Linux 4.18.0-553.33.1.el8_10.x86_64 amd64

OpenSearch config dir: /usr/share/opensearch/config/

OpenSearch config file: /usr/share/opensearch/config/opensearch.yml

OpenSearch bin dir: /usr/share/opensearch/bin/

OpenSearch plugins dir: /usr/share/opensearch/plugins/

OpenSearch lib dir: /usr/share/opensearch/lib/

Detected OpenSearch Version: 3.1.0

Detected OpenSearch Security Version: 3.1.0.0

Admin password set successfully.

### Success

### Execute this script now on all your nodes and then start all nodes

### OpenSearch Security will be automatically initialized.

### If you like to change the runtime configuration 

### change the files in ../../../config/opensearch-security and execute: 

sudo "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/config/opensearch-security" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv

### or run ./securityadmin_demo.sh

### To use the Security Plugin ConfigurationGUI

### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/<your-custom-admin-password>.

### (Ignore the SSL certificate warning because we installed self-signed demo certificates)

Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin

WARNING: Using incubator modules: jdk.incubator.vector

WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens


Full appreciate this may be a picnic problem (problem in chair, not in code/computer). I’m a long time user of Elasticsearch and needing to migrate a huge project into OpenSearch. I’m very grateful for any pointers or advice.

@Crickes I’ve tested this Docker Compose setup. With the current configuration, you will encounter SSL errors.

The root cause lies in your OpenSearch Dashboards configuration.
When you enabled the security plugin in OpenSearch nodes, you didn’t enable it in OpenSearch Dashboards. As a result, OpenSearch Dashboards is attempting to communicate with both OpenSearch nodes over HTTP instead of HTTPS.

Since both nodes are listed in the OPENSEARCH_HOSTS environment variable, OpenSearch Dashboards uses round-robin requests. This causes errors to appear on both OpenSearch nodes almost simultaneously.

It may appear that the cluster has not formed, but if you run the following command against the cluster, you’ll see that both nodes are actually part of the same cluster

curl --insecure -u admin:<password> -XGET https://<OpenSearch_node_IP_or_FQDN>:9200/_cat/nodes

This is a fixed OpenSearch Dashboards configuration.

  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:3.0.0
    # image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601 # Map host port 5601 to container port 5601
    expose:
      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
    environment:
      - 'OPENSEARCH_HOSTS=["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
    networks:
      - opensearch-net

Please be aware that the same OpenSearch documentation page contains two docker compose exmples. One is with security plugin enabled and the other one with disabled.

@Crickes Regarding the container freeze with latest tag, do you have enough disk space on your docker host?

I would expect that to work at it’s using the 3.0.0. image.

I’ve started from scratch and I’m using the Installation Quickstart guide: https://docs.opensearch.org/docs/latest/getting-started/quickstart/

I’ve copied the docker-compose.yml file from the linked GIT page:

If I leave the images tags set to latest, the cluster never forms and a curl request to the nodes replies with:

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:9200 

However, if all I do is change the image tags to version 3.0.0, the nodes start, the cluster forms and I get a good reply from the cluster.

This has to be an issue with the script that generates the demo certificates for a security enabled cluster.

@Crickes what versions are you using that are tagged as latest

You can verify this using docker inspect opensearchproject/opensearch:latest | grep version