Node certificate set up

Security
1

Since the same node certificate on all nodes is not recommended. what is recommended way to set up node certificate in kubernetes?

And is this right that setting up transport layer certificate to node certificate ?
ssl:
## TLS is mandatory for the transport layer and can not be disabled
transport:
existingCertSecret: node-cert
existingCertSecretCertSubPath: node.pem
existingCertSecretKeySubPath: node-key.pem
existingCertSecretRootCASubPath: root-ca.pem

2

@nueah
The above config is correct, you can use the same certificate for transport and rest, of course better to have separate ones.

Regarding kubernetes, in order to generate individual certificates you would need to look into using initContainers to generate certificate and store it in a volume that the container running opensearch can then consume.

There are plenty of examples out there, but below is the gist of it:

spec:
  initContainers:
  - name: ssl-cert
    image: ...
    command: ["/bin/sh"]
    args: ["-c","mkdir -p /etc/nginx/ssl;openssl req -newkey rsa:2048 -nodes -keyout /etc/nginx/ssl/nginx.key -x509 -days 365 -out /etc/nginx/ssl/nginx.crt -subj '/C=../ST=../L=../O=../OU=../CN=..'"]
    volumeMounts: 
      - name: ssl-cert
        mountPath: /etc/nginx/ssl # so whatever files are in this path in init container, will be copied to the volume

Hope this helps

2 Likes
3

if i want to create seperate certificate, which way should i follow in generate certificates?

4

@nueah I don’t follow your question.

You can generate certificates as described here, if you want to run separate certs for transport and rest, you just need to create additional cert to the ones listed there. The rest of the process is the same.

5

I had no idea an init-container could be used like this. This is brilliant, thank you for sharing.

6

what i asked is how to generate separate certificates as you explained. thank you

And why is it better to use the separate certificates than use the same certificate for transport and rest?

7

@nueah in general it’s always recommended to use separate certificates in case one of the certificates get compromised, you limit the exposure by using separate ones.

1 Like
8

@Anthony: but how can the different nodes then accept the certificates of the others?
if each node creates its own certificate

  • they either all need to have access to the private key of the (intermediate) CA (which would obviously be a bad design security-wise)
  • or they’d have purely self-signed certificates where they couldn’t validate the certificates of the others

the only way i could see this working is if the nodes would then create a CSR (Certificate Signing Request) and send that to a CA which gives them back a signed certificate. but then you have the trust issue again as the CA can’t really know whether the CSR came from the real system or an impostor.

we’re currently looking at using cert-manager for the certificates.

however, i haven’t found out yet how we could get it to issue certificates for each node individually since you have to create a Certificate resource in k8s to get a Secret containing the certificate. if you’d use an initContainer for that you’d need API access to k8s (incl. authentication) in your pod which would also open up security issues again (IMHO pods shouldn’t have management-access to their PaaS).

of course, an operator for opensearch would solve most of these issues because it could then take care of creating the certificate requests for the nodes it spins up and also manage certificate renewals :innocent: (e.g. the elastic ECK manages these things)

1 Like
9

@ralph installing opendistro with helm now has no option for not mounting root CA.
so if it is better to use the individual certificates then, i think using init container to generate certificates is great.

10

i’m not familiar with the helm charts for opensearch/opendistro, but would strongly presume that they mean that you must mount the public key of the CA, which of course is absolutely Ok? opensearch security / opendistro security itself makes no use of the private key of the CA and any such usage would be weird, to say the least

11

I agree.

I might wrong, But as you can see, we can disable ssl or enable it and mount root CA.
if i don’t mount it, then it won’t start up.
ssl:
## TLS is mandatory for the transport layer and can not be disabled
transport:
existingCertSecret:
existingCertSecretCertSubPath: elk-transport-crt.pem
existingCertSecretKeySubPath: elk-transport-key.pem
existingCertSecretRootCASubPath: elk-transport-root-ca.pem
rest:
enabled: false
existingCertSecret:
existingCertSecretCertSubPath: elk-rest-crt.pem
existingCertSecretKeySubPath: elk-rest-key.pem
existingCertSecretRootCASubPath: elk-rest-root-ca.pem
admin:
enabled: false
existingCertSecret:
existingCertSecretCertSubPath: admin-crt.pem
existingCertSecretKeySubPath: admin-key.pem
existingCertSecretRootCASubPath: admin-root-ca.pem

Also Depending on the setting kibana needs to mount the root CA also. kibana.