How to add nodes to an existing cluster

Security
1

Hi,
I’m attempting to add nodes to a running ODFE cluster.
So far I’ve:

  • Installed ODFE on the new node
  • Created certificates and sent them to this machine
  • Configured the previously running node’s elasticsearch.yml to have discovery.seed_hosts and cluster.initial_master_nodes to include the new node.
  • I’ve configured the same on the new node (with the information for the previous node).

Once I restart the new ODFE node’s API I get: Open Distro Security not initialized.
When I try to run securityadmin.sh on the older node nothing seems out of the ordinary and in the new one it will state:

Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
  Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

Is there any documentation on how to add nodes to existing clusters that have security enabled?

2

@poshpotoo What version of odfe are you running?
There is no need to re-run securityadmin.sh, if the configuration is correct the node will be added to cluster and security index updated with relevant details.
Please also note that initial_master_nodes don’t need to be updated if you already have a cluster, only needed when new cluster is being formed.
Can you confirm that the certificate and key are signed by the same CA as the rest of the certs in the cluster?
If so, can you try to disable below and see if you get any error
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.transport.resolve_hostname: false

3

Any update on the above issue ? I am too having same issues when intializing the cluster.

4

@aayush Are you adding a node to already existing cluster or creating new cluster?
Which certificates are you using? Also, can you share the elasticsearch.yml files for existing nodes and new node? Redact any sensitive information

5

Hello @Anthony the issues has been resolved. Sorry for my late response. Meanwhile the issues was related to the certificate generation and firewall issues. Now every thing is fine.

Also I have another question, um i have 3 opendistro elasticsearch nodes in cluster. I have generated certificate for them using the securityadmin.sh. So my question is if i want to add another elasticsearch node on the existing cluster, do i need to rerun the securityadmin.sh ??
If not how will i generate the certificate for the new elasticsearch node ??

6

@aayush If you used securityadmin.sh are you using same certificates for 3 nodes? If that is the case, the easiest way would be to copy the certificates from one of the running clusters and use these for new node. It should work as expected

7

@Anthony I have the following configuration in instances.yml

Elasticsearch nodes

elasticsearch-nodes:

  • name: es1
    ip:
    • x.x.x.1
  • name: es2
    ip:
    • x.x.x.2
  • name: es3
    ip:
    • x.x.x.3

Wazuh server nodes

wazuh-servers:

  • name: filebeat
    ip:
    • x.x.x.4

Kibana node

kibana:

  • name: kibana
    ip:
    • x.x.x.5

So as you can see i have different certificates for each elasticsearch nodes (es1, es2, es3)

Now if i had to add es4 how can i generate the certificate ??? Should i add the es4 in the instances.yml and run the sec. But if did that then i will end up generating new certificates for each nodes and of course the admin and root ca also will be generated new. Am i right or am i missing something ??

Also i am using wazuh-cert-tool.sh from wazuh official documentaion for creating certificates.

8

Reading through Wazuh docs it seems there is no option to create a new cert from already existing ca.pem. In which case you have 2 options:

  1. Manually create a cert using openssl signed by existing ca. Need to ensure that the SAN entries are correct, can use previous certs for reference. (no downtime, but prone to errors)
  2. Rerun the Wazuh script with updated instances.yml file, replacing old files with new ca and certs, (will require a restart, but is quicker and more reliable, if this is not a dev cluster - I would try this on a test cluster first to make sure it behaves as expected)
9

yes may be solution 2 will do the trick. It will require restart of all the nodes, kibana etc.

Thank you i will further look into it more.