Cluster certificate update

OpenSearch
1

Version 2.9

Current certs are about to expire referenced in the OpenSearch yaml config. After I stage new certs in the same location, and doing a rolling restart, do I need anything else? Will the nodes talk to each other while others have new certs after a restart vs those with old certs? Do I need to run the securityadmin script? Nothing changed as far roles go.

Going to do coordinator, data and master nodes in this order.

Thanks in advance

2

@stecino

  1. Which method did you use to host your OpenSearch Clusters? (ex. helm, k8s operator, docker, etc)

  2. Yes, you need .pem(s) or .crt(s) files to call securityadmin.sh when making a change in opensearch/config/opensearch-security directory. See the below code.

/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
  -cacert /etc/opensearch/root-ca.pem \
  -cert /etc/opensearch/kirk.pem \
  -key /etc/opensearch/kirk-key.pem \
  -cd /usr/share/opensearch/config/opensearch-security/
3

Thanks for the reply. Do I run the security admin script after rolling restart? Distribution I use is tar distro, I have physicals and VMs.

4

@stecino securityadmin.sh is required only for the security plugin configuration.
Certificates must be managed separately.
You’ll need to just replace certificates.

Which certificates expired? Node, client, RootCA or all of them?
How big is the cluster?

5

@pablo thanks for the reply. It’s the node certificate and one admin cert. I have total of 19 nodes, 6 coordinator, 3 master and 10 data nodes. So just replace those and do rolling restart or not even that?

6

@stecino As long as RootCA will remain the same and DNs in the new node certs match the old ones you are ok.

7

Awesome!! Thanks a lot. This will buy me another 2 years :joy::joy:

8

@stecino You should also consider the hot reload. Since you’ll be restarting all of the nodes, you can enable hot reloading in opensearch.yml and avoid rolling restarts in the future.

9

Does that apply to 2.9? I will be upgrading to latest version in a few months

10

@pablo thanks for your time

For what I’ve understood, if plugins.security.ssl_cert_reload_enabled setting is true, superadmin can replace the expired certificates to new ones using the Reload Certificates API.

curl --cacert <ca.pem> --cert <admin.pem> --key <admin.key> \
  -XPUT https://localhost:9200/_plugins/_security/api/ssl/transport/reloadcerts

But what if @stecino 's cluster has been set neither plugins.security.ssl.certificates_hot_reload.enabled nor plugins.security.ssl_cert_reload_enabled? They seem like to be false by default.

  1. In this case, just replacing certificates in the same file path is enough?
  2. If we want to use the Reload Certificates API, do we have to set true to both certificates_hot_reload and ssl_cert_reload_enabled and restart all nodes?
11

@stecino According to the documentation the hot reloading wasn’t available until version 2.13 .
Are you planning to upgrade?

1 Like
12

@yeonghyeonKo

No, it’s not. When hot reloading is not activate then rolling restart is required.

As per OpenSearch documentation. The below option enables active monitoring of the certificates keystore and replaces certificates once the keystore is updated.

plugins.security.ssl.certificates_hot_reload.enabled: true

The below option, gives you more control as hot reload must be initiated manually with an API call.

plugins.security.ssl_cert_reload_enabled: true

Any change of the opensearch.yml file requires restart of the OpenSearch service.

1 Like
13

I will be updating in a few months, yes.

14

Thanks for your time, then in my opinion it’s recommended to set those two options True, when users try to initially configure the clusters.
(I mean, there’s no need to leave those options to be default : False)