Hi,
I’m trying to deploy with docker compose a 2 nodes cluster with the security plugin activated and my own certificates.
I started from the docker-compose.yml provided in OpenSeach install documentation.
And updated it:
version: '3'
services:
opensearch-node1:
restart: always
image: opensearchproject/opensearch:1.2.4
container_name: opensearch-node1
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node1
- discovery.seed_hosts=opensearch-node1,opensearch-node2
- cluster.initial_master_nodes=opensearch-node1,opensearch-node2
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
- "DISABLE_INSTALL_DEMO_CONFIG=true" # disables execution of install_demo_configuration.sh bundled with security plugin, which installs demo certificates and security configurations to OpenSearch
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
hard: 65536
volumes:
- opensearch-data1:/usr/share/opensearch/data
- ./config/certificats/root-ca.pem:/usr/share/opensearch/config/root-ca.pem
- ./config/certificats/node1.pem:/usr/share/opensearch/config/node1.pem
- ./config/certificats/node1-key.pem:/usr/share/opensearch/config/node1-key.pem
- ./config/certificats/admin.pem:/usr/share/opensearch/config/admin.pem
- ./config/certificats/admin-key.pem:/usr/share/opensearch/config/admin-key.pem
- ./config/custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml
- ./config/internal_users.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml
- ./config/roles_mapping.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml
- ./config/tenants.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml
- ./config/roles.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml
- ./config/action_groups.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.ymlm
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
networks:
- opensearch-net
opensearch-node2:
restart: always
image: opensearchproject/opensearch:1.2.4
container_name: opensearch-node2
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node2
- discovery.seed_hosts=opensearch-node1,opensearch-node2
- cluster.initial_master_nodes=opensearch-node1,opensearch-node2
- bootstrap.memory_lock=true
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
- "DISABLE_INSTALL_DEMO_CONFIG=true" # disables execution of install_demo_configuration.sh bundled with security plugin, which installs demo certificates and security configurations to OpenSearch
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- opensearch-data2:/usr/share/opensearch/data
- ./config/certificats/root-ca.pem:/usr/share/opensearch/config/root-ca.pem
- ./config/certificats/node1.pem:/usr/share/opensearch/config/node1.pem
- ./config/certificats/node1-key.pem:/usr/share/opensearch/config/node1-key.pem
- ./config/certificats/admin.pem:/usr/share/opensearch/config/admin.pem
- ./config/certificats/admin-key.pem:/usr/share/opensearch/config/admin-key.pem
- ./config/custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml
- ./config/internal_users.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml
- ./config/roles_mapping.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml
- ./config/tenants.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml
- ./config/roles.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml
- ./config/action_groups.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml
networks:
- opensearch-net
opensearch-dashboards:
restart: always
image: opensearchproject/opensearch-dashboards:1.2.0
container_name: opensearch-dashboards
ports:
- 5601:5601
expose:
- "5601"
environment:
- 'OPENSEARCH_HOSTS=["https://opensearch-node1:9200","https://opensearch-node2:9200"]' # must be a string with no spaces when specified as an environment variable
- "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
networks:
- opensearch-net
volumes:
opensearch-data1:
opensearch-data2:
networks:
opensearch-net:
I followed the documentation to generate my own certificates (root, admin, node1), and used the default YAML files for internal_users, roles, roles_mapping ect…
My custom opensearch.yml look like this:
plugins.security.ssl.transport.pemcert_filepath: node1.pem
plugins.security.ssl.transport.pemkey_filepath: node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: node1.pem
plugins.security.ssl.http.pemkey_filepath: node1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.authcz.admin_dn:
- 'CN=admin,OU=,O=,L=,ST=,C='
plugins.security.nodes_dn:
- 'CN=node1,OU=,O=,L=,ST=e,C='
plugins.security.allow_default_init_securityindex: true
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
opendistro_security.audit.config.disabled_rest_categories: NONE
opendistro_security.audit.config.disabled_transport_categories: NONE
In the config.yml file I only allowed as authc ‘clientcert_auth_domain’ and ‘basic_internal_auth_domain’ on http and transport.
Everything was working perfectly with the demo configuration but with my own the cluster fails to deploy.
The main differences between the two that I see in the logs is the published and bound address ips of the nodes:
Demo cluster:
opensearch-node1 | [2022-02-16T16:09:41,541][INFO ][o.o.t.TransportService ] [opensearch-node1] publish_address {172.23.0.3:9300}, bound_addresses {0.0.0.0:9300}
opensearch-node2 | [2022-02-16T16:09:39,772][INFO ][o.o.t.TransportService ] [opensearch-node2] publish_address {172.23.0.2:9300}, bound_addresses {0.0.0.0:9300}
My cluster:
opensearch-node1 | [2022-02-16T16:16:24,825][INFO ][o.o.t.TransportService ] [opensearch-node1] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
opensearch-node2 | [2022-02-16T16:16:24,396][INFO ][o.o.t.TransportService ] [opensearch-node2] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
I guess that’s why it is not working since the discovery is looking for nodes at [172.23.0.4:9300, 172.23.0.3:9300].
So nodes can only see themselves:
opensearch-node2 | [2022-02-16T16:16:24,671][INFO ][o.o.c.s.MasterService ] [opensearch-node2] elected-as-master ([1] nodes joined)[{opensearch-node2}{aLvp4UYhSLOTsrACM87a4A}{zdl58d-yR5-6L7WRNCi8IA}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 17, version: 362, delta: master node changed {previous [], current [{opensearch-node2}{aLvp4UYhSLOTsrACM87a4A}{zdl58d-yR5-6L7WRNCi8IA}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]}
opensearch-node1 | [2022-02-16T16:17:04,974][WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node1] master not discovered or elected yet, an election requires a node with id [aLvp4UYhSLOTsrACM87a4A], have discovered [{opensearch-node1}{ln7oMODJT5ylShzE-LpYSQ}{z5UXH919Sn2UaN-HRf6PJw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum;discovery will continue using [172.23.0.4:9300, 172.23.0.3:9300] from hosts providers and [{opensearch-node1}{ln7oMODJT5ylShzE-LpYSQ}{z5UXH919Sn2UaN-HRf6PJw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 16, last-accepted version 360 in term 16
Yet I don’t know what property is missing to correct it.
So if anyone could help me with this it would be much appreciated.
Sincerely