I’ve been scratching my head all day as to why my extremely basic docker based config wouldn’t work. By basic I mean just using the example docker compose file in the documentation. It was working yesterday, but today it didn’t.
Then I realised I was pulling opensearchproject/opensearch:latest, and when I check the Docker Hub, the images tagged 3.1.0 was published last night.
I reverted to tag 3.0.0 and everything works fine again, I’m presuming there is something not quite right in the 3.1.0 tagged image. My guess is certificates as I couldn’t get the dashboard nodes to connect to the data nodes, and even basic curl requests to the data nodes were being rejected.
You’re correct in that the unmodified docker-compose file does work, but that has the DISABLE_INSTALL_DEMO_CONFIG parameter set to true. If you comment this out and the line that disables the security plugin, the nodes seem to start up but are in complete isolation from each other.
Here is the docker compose file I am testing with. Currently with the 3.0.0 image, the cluster starts.
services:
opensearch-node1:
image: opensearchproject/opensearch:3.0.0
# image: opensearchproject/opensearch:latest
container_name: opensearch-node1
environment:
- cluster.name=opensearch-cluster # Name the cluster
- node.name=opensearch-node1 # Name the node that will run in this container
- discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
- cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligibile to serve as cluster manager
- bootstrap.memory_lock=true # Disable JVM heap memory swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
- OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD} # Sets the demo admin user password when using demo configuration, required for OpenSearch 2.12 and later
# - "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
# - "DISABLE_SECURITY_PLUGIN=true" # Disables Security plugin
ulimits:
memlock:
soft: -1 # Set memlock to unlimited (no soft or hard limit)
hard: -1
nofile:
soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
hard: 65536
volumes:
- opensearch-data1:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
ports:
- 9200:9200 # REST API
- 9600:9600 # Performance Analyzer
networks:
- opensearch-net # All of the containers will join the same Docker bridge network
opensearch-node2:
image: opensearchproject/opensearch:3.0.0
# image: opensearchproject/opensearch:latest
container_name: opensearch-node2
environment:
- cluster.name=opensearch-cluster # Name the cluster
- node.name=opensearch-node2 # Name the node that will run in this container
- discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
- cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligibile to serve as cluster manager
- bootstrap.memory_lock=true # Disable JVM heap memory swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
- OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD} # Sets the demo admin user password when using demo configuration, required for OpenSearch 2.12 and later
# - "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
# - "DISABLE_SECURITY_PLUGIN=true" # Disables Security plugin
ulimits:
memlock:
soft: -1 # Set memlock to unlimited (no soft or hard limit)
hard: -1
nofile:
soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
hard: 65536
volumes:
- opensearch-data2:/usr/share/opensearch/data # Creates volume called opensearch-data2 and mounts it to the container
networks:
- opensearch-net # All of the containers will join the same Docker bridge network
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:3.0.0
# image: opensearchproject/opensearch-dashboards:latest
container_name: opensearch-dashboards
ports:
- 5601:5601 # Map host port 5601 to container port 5601
expose:
- "5601" # Expose port 5601 for web access to OpenSearch Dashboards
environment:
- 'OPENSEARCH_HOSTS=["http://opensearch-node1:9200","http://opensearch-node2:9200"]'
- "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
networks:
- opensearch-net
volumes:
opensearch-data1:
opensearch-data2:
networks:
opensearch-net:
If I change the image to use the latest tag, then the 3 nodes seem to come up, but that can’t talk to each other which I suspect is an issue with generating the demo certificates.
On deploying using the latest tag, the output stops here:
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 4.18.0-553.33.1.el8_10.x86_64 amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 3.1.0
Detected OpenSearch Security Version: 3.1.0.0
Admin password set successfully.
### Success
### Execute this script now on all your nodes and then start all nodes
### OpenSearch Security will be automatically initialized.
### If you like to change the runtime configuration
### change the files in ../../../config/opensearch-security and execute:
sudo "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/config/opensearch-security" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv
### or run ./securityadmin_demo.sh
### To use the Security Plugin ConfigurationGUI
### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/<your-custom-admin-password>.
### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
Full appreciate this may be a picnic problem (problem in chair, not in code/computer). I’m a long time user of Elasticsearch and needing to migrate a huge project into OpenSearch. I’m very grateful for any pointers or advice.
@Crickes I’ve tested this Docker Compose setup. With the current configuration, you will encounter SSL errors.
The root cause lies in your OpenSearch Dashboards configuration.
When you enabled the security plugin in OpenSearch nodes, you didn’t enable it in OpenSearch Dashboards. As a result, OpenSearch Dashboards is attempting to communicate with both OpenSearch nodes over HTTP instead of HTTPS.
Since both nodes are listed in the OPENSEARCH_HOSTS environment variable, OpenSearch Dashboards uses round-robin requests. This causes errors to appear on both OpenSearch nodes almost simultaneously.
It may appear that the cluster has not formed, but if you run the following command against the cluster, you’ll see that both nodes are actually part of the same cluster
This is a fixed OpenSearch Dashboards configuration.
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:3.0.0
# image: opensearchproject/opensearch-dashboards:latest
container_name: opensearch-dashboards
ports:
- 5601:5601 # Map host port 5601 to container port 5601
expose:
- "5601" # Expose port 5601 for web access to OpenSearch Dashboards
environment:
- 'OPENSEARCH_HOSTS=["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
networks:
- opensearch-net
Please be aware that the same OpenSearch documentation page contains two docker compose exmples. One is with security plugin enabled and the other one with disabled.