Nodes unable to communicate while creating cluster with custom CA files

Security
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
version: opensearch 2.17.0 (apt installed)
os: Debian bookworm.

Describe the issue:
Unable to start OpenSearch after applying custom CA-files before initial setup.

Steps:

  1. Installing OpenSearch using apt.
  2. Applying custom opensearch.yml file to the configuration folder.
  3. Removing demo certificates in configuration folder.
  4. Writing custom CA-files from CA-server to configuration folder.
  5. Starting the service

Arch:
3 index nodes within the cluster. Each will be assigned as master. negotiation shall be voted internally.
Each node is a dedicated VM inside of a Proxmox node.
Deployment done through Ansible.

Configuration:
Each node will have identical configuration file except proper values like filenames on certs/keys and node name.

cluster.name: ES
node.name: "log1"
cluster.initial_master_nodes: ["log1","log2","log3"]
discovery.seed_hosts: ["log1","log2","log3"]
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 0.0.0.0

plugins.security.disabled: false

plugins.security.ssl_cert_reload_enabled: true

plugins.security.ssl.transport.enforce_hostname_verification: false

plugins.security.ssl.transport.pemcert_filepath: log1.crt
plugins.security.ssl.transport.pemkey_filepath: log1.key
plugins.security.ssl.transport.pemtrustedcas_filepath: root.crt
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: log1.crt
plugins.security.ssl.http.pemkey_filepath: log1.key
plugins.security.ssl.http.pemtrustedcas_filepath: root.crt

plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true

# RFC2253 format
plugins.security.nodes_dn:
  - "CN=log1,O=ES,C=DK"
  - "CN=log2,O=ES,C=DK"
  - "CN=log3,O=ES,C=DK"

plugins.security.authcz.admin_dn:
  - 'CN=ADMIN,O=ES,C=DK'

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [all_access, security_rest_api_access]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: 
  - .plugins-ml-agent
  - .plugins-ml-config
  - .plugins-ml-connector
  - .plugins-ml-controller
  - .plugins-ml-model-group
  - .plugins-ml-model
  - .plugins-ml-task
  - .plugins-ml-conversation-meta
  - .plugins-ml-conversation-interactions
  - .plugins-ml-memory-meta
  - .plugins-ml-memory-message
  - .plugins-ml-stop-words
  - .opendistro-alerting-config
  - .opendistro-alerting-alert*
  - .opendistro-anomaly-results*
  - .opendistro-anomaly-detector*
  - .opendistro-anomaly-checkpoints
  - .opendistro-anomaly-detection-state
  - .opendistro-reports-*
  - .opensearch-notifications-*
  - .opensearch-notebooks
  - .opensearch-observability
  - .ql-datasources
  - .opendistro-asynchronous-search-response*
  - .replication-metadata-store
  - .opensearch-knn-models
  - .geospatial-ip2geo-data*
  - .plugins-flow-framework-config
  - .plugins-flow-framework-templates
  - .plugins-flow-framework-state
node.max_local_storage_nodes: 3

Relevant Logs or Screenshots:
Log sample 1 (same output from all nodes):

[WARN ][o.o.c.c.ClusterFormationFailureHelper] [log1] cluster-manager not discovered or elected yet, an election requires at least 2 nodes with ids from...
Wait for cluster to be available....

Questions:

  1. Do we need to execute securityadmin.sh while installing opensearch from apt?
  2. Are we required to preform any other actions except for starting the services after proper configuration files has been applied?
  3. Making the log1 as master only will make the node log1 start with health of GREEN. But the other nodes (log2, log3) wont be able to join as “slaves”. This should verify that the certificate files works as intended?
  4. tcpdump provides information of communication between nodes at port 9300 (transport). a manual verification to port 9200 (http) with node certificate using openssl or curl also outputs ok. Does there exist any other tools provided by opensearch for troubleshooting?
2

Hi @echo,

  1. securityadmin.sh allows you to update/change the security plugin’s configuration (i.e.: users, roles, permissions, and backend settings) without restarting the cluster and eliminates the need to edit configuration files on every individual node - you would use it once you nodes form a cluster.
  2. To start a cluster, if configuration is “proper” - No.
  3. Not sure if I got your question correctly, but if the nodes form a cluster and can successfully communicate, that would be a good indicator that your certs are all good.
  4. There are some Troubleshooting tips here: Troubleshoot TLS - OpenSearch Documentation

Best,
mj

3

How did you generate your certificates?

Have you followed: Generating self-signed certificates - OpenSearch Documentation

Would you mid sharing your opensearch.yml from all three nodes?

Thanks,
mj