Cluster certificate update

OpenSearch
1

Version 2.9

Current certs are about to expire referenced in the OpenSearch yaml config. After I stage new certs in the same location, and doing a rolling restart, do I need anything else? Will the nodes talk to each other while others have new certs after a restart vs those with old certs? Do I need to run the securityadmin script? Nothing changed as far roles go.

Going to do coordinator, data and master nodes in this order.

Thanks in advance

2

@stecino

  1. Which method did you use to host your OpenSearch Clusters? (ex. helm, k8s operator, docker, etc)

  2. Yes, you need .pem(s) or .crt(s) files to call securityadmin.sh when making a change in opensearch/config/opensearch-security directory. See the below code.

/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
  -cacert /etc/opensearch/root-ca.pem \
  -cert /etc/opensearch/kirk.pem \
  -key /etc/opensearch/kirk-key.pem \
  -cd /usr/share/opensearch/config/opensearch-security/
3

Thanks for the reply. Do I run the security admin script after rolling restart? Distribution I use is tar distro, I have physicals and VMs.

4

@stecino securityadmin.sh is required only for the security plugin configuration.
Certificates must be managed separately.
You’ll need to just replace certificates.

Which certificates expired? Node, client, RootCA or all of them?
How big is the cluster?

5

@pablo thanks for the reply. It’s the node certificate and one admin cert. I have total of 19 nodes, 6 coordinator, 3 master and 10 data nodes. So just replace those and do rolling restart or not even that?

6

@stecino As long as RootCA will remain the same and DNs in the new node certs match the old ones you are ok.

7

Awesome!! Thanks a lot. This will buy me another 2 years :joy::joy:

8

@stecino You should also consider the hot reload. Since you’ll be restarting all of the nodes, you can enable hot reloading in opensearch.yml and avoid rolling restarts in the future.

9

Does that apply to 2.9? I will be upgrading to latest version in a few months

10

@pablo thanks for your time

For what I’ve understood, if plugins.security.ssl_cert_reload_enabled setting is true, superadmin can replace the expired certificates to new ones using the Reload Certificates API.

curl --cacert <ca.pem> --cert <admin.pem> --key <admin.key> \
  -XPUT https://localhost:9200/_plugins/_security/api/ssl/transport/reloadcerts

But what if @stecino 's cluster has been set neither plugins.security.ssl.certificates_hot_reload.enabled nor plugins.security.ssl_cert_reload_enabled? They seem like to be false by default.

  1. In this case, just replacing certificates in the same file path is enough?
  2. If we want to use the Reload Certificates API, do we have to set true to both certificates_hot_reload and ssl_cert_reload_enabled and restart all nodes?