I’ve been scratching my head all day as to why my extremely basic docker based config wouldn’t work. By basic I mean just using the example docker compose file in the documentation. It was working yesterday, but today it didn’t.
Then I realised I was pulling opensearchproject/opensearch:latest, and when I check the Docker Hub, the images tagged 3.1.0 was published last night.
I reverted to tag 3.0.0 and everything works fine again, I’m presuming there is something not quite right in the 3.1.0 tagged image. My guess is certificates as I couldn’t get the dashboard nodes to connect to the data nodes, and even basic curl requests to the data nodes were being rejected.
You’re correct in that the unmodified docker-compose file does work, but that has the DISABLE_INSTALL_DEMO_CONFIG parameter set to true. If you comment this out and the line that disables the security plugin, the nodes seem to start up but are in complete isolation from each other.
Here is the docker compose file I am testing with. Currently with the 3.0.0 image, the cluster starts.
services:
opensearch-node1:
image: opensearchproject/opensearch:3.0.0
# image: opensearchproject/opensearch:latest
container_name: opensearch-node1
environment:
- cluster.name=opensearch-cluster # Name the cluster
- node.name=opensearch-node1 # Name the node that will run in this container
- discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
- cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligibile to serve as cluster manager
- bootstrap.memory_lock=true # Disable JVM heap memory swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
- OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD} # Sets the demo admin user password when using demo configuration, required for OpenSearch 2.12 and later
# - "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
# - "DISABLE_SECURITY_PLUGIN=true" # Disables Security plugin
ulimits:
memlock:
soft: -1 # Set memlock to unlimited (no soft or hard limit)
hard: -1
nofile:
soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
hard: 65536
volumes:
- opensearch-data1:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
ports:
- 9200:9200 # REST API
- 9600:9600 # Performance Analyzer
networks:
- opensearch-net # All of the containers will join the same Docker bridge network
opensearch-node2:
image: opensearchproject/opensearch:3.0.0
# image: opensearchproject/opensearch:latest
container_name: opensearch-node2
environment:
- cluster.name=opensearch-cluster # Name the cluster
- node.name=opensearch-node2 # Name the node that will run in this container
- discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
- cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligibile to serve as cluster manager
- bootstrap.memory_lock=true # Disable JVM heap memory swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
- OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD} # Sets the demo admin user password when using demo configuration, required for OpenSearch 2.12 and later
# - "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
# - "DISABLE_SECURITY_PLUGIN=true" # Disables Security plugin
ulimits:
memlock:
soft: -1 # Set memlock to unlimited (no soft or hard limit)
hard: -1
nofile:
soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
hard: 65536
volumes:
- opensearch-data2:/usr/share/opensearch/data # Creates volume called opensearch-data2 and mounts it to the container
networks:
- opensearch-net # All of the containers will join the same Docker bridge network
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:3.0.0
# image: opensearchproject/opensearch-dashboards:latest
container_name: opensearch-dashboards
ports:
- 5601:5601 # Map host port 5601 to container port 5601
expose:
- "5601" # Expose port 5601 for web access to OpenSearch Dashboards
environment:
- 'OPENSEARCH_HOSTS=["http://opensearch-node1:9200","http://opensearch-node2:9200"]'
- "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
networks:
- opensearch-net
volumes:
opensearch-data1:
opensearch-data2:
networks:
opensearch-net:
If I change the image to use the latest tag, then the 3 nodes seem to come up, but that can’t talk to each other which I suspect is an issue with generating the demo certificates.
On deploying using the latest tag, the output stops here:
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 4.18.0-553.33.1.el8_10.x86_64 amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 3.1.0
Detected OpenSearch Security Version: 3.1.0.0
Admin password set successfully.
### Success
### Execute this script now on all your nodes and then start all nodes
### OpenSearch Security will be automatically initialized.
### If you like to change the runtime configuration
### change the files in ../../../config/opensearch-security and execute:
sudo "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/config/opensearch-security" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv
### or run ./securityadmin_demo.sh
### To use the Security Plugin ConfigurationGUI
### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/<your-custom-admin-password>.
### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
Full appreciate this may be a picnic problem (problem in chair, not in code/computer). I’m a long time user of Elasticsearch and needing to migrate a huge project into OpenSearch. I’m very grateful for any pointers or advice.
@Crickes I’ve tested this Docker Compose setup. With the current configuration, you will encounter SSL errors.
The root cause lies in your OpenSearch Dashboards configuration.
When you enabled the security plugin in OpenSearch nodes, you didn’t enable it in OpenSearch Dashboards. As a result, OpenSearch Dashboards is attempting to communicate with both OpenSearch nodes over HTTP instead of HTTPS.
Since both nodes are listed in the OPENSEARCH_HOSTS environment variable, OpenSearch Dashboards uses round-robin requests. This causes errors to appear on both OpenSearch nodes almost simultaneously.
It may appear that the cluster has not formed, but if you run the following command against the cluster, you’ll see that both nodes are actually part of the same cluster
This is a fixed OpenSearch Dashboards configuration.
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:3.0.0
# image: opensearchproject/opensearch-dashboards:latest
container_name: opensearch-dashboards
ports:
- 5601:5601 # Map host port 5601 to container port 5601
expose:
- "5601" # Expose port 5601 for web access to OpenSearch Dashboards
environment:
- 'OPENSEARCH_HOSTS=["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
networks:
- opensearch-net
Please be aware that the same OpenSearch documentation page contains two docker compose exmples. One is with security plugin enabled and the other one with disabled.
@Crickes The demo certificates are not generated during the docker deployment.
They are hardcoded.
Just to be clear. You’re running docker compose with 3.0.0 and all is working.
However, when you swtich to 3.1.0, nodes are going up but you’re getting the error and cluster is not formed. So you no longer get freeze during the start-up process.
If that’s the case, could you share the full startup logs from one of the OpenSearch nodes?
How did you deploy this cluster? Is it running on a virtual machine, Mac, Windows?
Have you compare digest between local image and the one in Docker Hub?
I tried running the same docker compose on a different server (actually my laptop), and its start up flawlessly. So this is obviously an issues with my local environment on the server I am trying to use. It’s weird as I though using Docker was meant to prevent such issues.
Anyway, this is clearly not an OpenSearch fault but something weird happening or not happening on my server.
To answer your questions, when attempting to deploy 3.1.0, I get no error messages, the start up of Opensearch just hangs. The output log I see is pasted in a post above, but it always stops and hangs after these two lines are output:
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
Looking at the 3.0.0 startup logs, the same lines appear but after about a second, we see the normal node start up logs as the node starts to come up and discover things.
My host server is a virtual machine on VMWare running Alma 8.10.
@Crickes Does your VM has enough space in the VMware datastore? Did you set your hard drive to thin or thick provisioning?
Maybe your OS has enough free space, but the backing datastore is full.
Hi Pablo, thanks for actively discussing this issue. I have more than 70G of disk space available on the server and also tried deleting and pulling the latest tag. My docker version is “podman version 4.9.4-rhel”.
My logs are getting stuck at the same part as @Crickes:
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
@Crickes, I’ve tested your scenario with AlmaLinux 8.10. I’ve got the same behaviour. The startup froze after the reported warning message.
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
As per your tests, OpenSearch 3.0.0 worked just fine.
I’ve also found that the issue is not related to the Docker engine version, as I used your version and the latest.
Next, I’ve installed AlmaLinux 10 and OpenSearch nodes started successfully and formed the cluster.