K8s: Define SSL certificates via secrets

Security
21

Where is opensearch-dashboard-cert set up? I can see opensearch-tls in extraVolumes, but not opensearch-dashboard-cert

22

It is also generated via cert-manager and only the secret is needed.
So there is no need to mount this cert at the current time.

More important is the connection from extern to the OpenSearch-Cluster.

23

@skunz
to be honest I haven’t had much experience with ingress and not sure why you are using nginx-external.

But I did notice that in my config ingress tls section also has hosts:

tls:
  - hosts:
      - https-example.foo.com
    secretName: testsecret-tls

Overall the error you are seeing relates to certificates, it would seem the cert/config from ingress is causing this

24

just to chime in here with something important to keep in mind when working with cert-manager and/or let’s encrypt: these certificates are only valid for a limited period of time, after which they get renewed. opensearch won’t notice this automatically (unless there’s a new feature i’ve missed) and you’ll instead have to do an API call (or use the admin shell script) to tell it to load the new certificates. depending on whether the root CA also changed (shouldn’t be an issue in your case when you’re using let’s encrypt, but this is an issue when using a self-signed CA with e.g. cert-manager) this will be extremely tricky to handle.

if you figure out how to handle this properly please let us know :smiley:

25

@skunz
After some testing locally using minikube, I got it working using below config for ingress:

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
  hosts:
    - host: chart-example.local
      paths:
        - path: /
          backend:
            serviceName: opensearch-dashboards
            servicePort: 5601
  tls: 
    - secretName: chart-example-tls
      hosts:
        - chart-example.local

After helm install, I did get 502 Bad Gateway for a couple of minutes while OSDashboards was getting started. After this everything worked as expected.

Hope this helps

26

@ralph

Since the certificate details are stored in a secret which is then mounted, won’t it be up to date automatically?
The secret will be updated by the cert-manager if the certificate is renewed. Will the mounted files (tls.crt, tls.key) not always refer the current secret content?

@Anthony

It was an missing configuration in the ingress of OpenSearch:

nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"

This is by default set to “HTTP”. After I set OpenSearch to communicate over https the Nginx server receives the wrong protocol and therefore throws an error.

After I added this line the “Bad Gateway”-Error is gone.

27

Now I wanted to activate the SSl-Verificiation in OpenSearch Dashboards:

config:
    # Default OpenSearch Dashboards configuration from docker image of Dashboards
    opensearch_dashboards.yml:
      opensearch.ssl.verificationMode: full
      opensearch.ssl.certificateAuthorities: [ "/usr/share/opensearch-dashboards/config/certs/tls.crt" ]

If I upgrade my deployment, Dashboards is not working anymore:

{"type":"log","@timestamp":"2021-11-16T10:40:34Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: unable to get issuer certificate"}

Do I need to split my tls.crt-file into the intermediate and root cert?
https://github.com/opensearch-project/documentation-website/issues/144

28

@skunz
In this case kibana/openDashboards is looking for a root-ca, which is not provided as part of the 3 certs received from lets encrypt. Therefore to get this working locally I had to download root-ca from lets encrypt here

The name of the downloaded cert in my case is isrgrootx1.pem (it’s the self-signed .pem option).

No modification was needed to the 3in1.pem received originally.

29

I had a look into the 3 certs which are part of tls.crt:

  • <domain>
  • R3
  • ISRG Root X1

So I think it is part of the file (or the cert chain) but it is not recognized. Maybe it has to be in that single file. I will try it your way and report back.
Thanks in advance!

30

the mounted files will be updated by k8s, but opensearch-security - to my knowledge - does not monitor the filesystem for changes and will thus not load the new certificates. instead, you’ll either have to restart the node or have to trigger a hot-reload through an API call (though i couldn’t find anything in the opensearch-security docs about this, but Search Guard supports this and i’d thus presume that opensearch-security will have the same feature).

31

@skunz Did you get this working without downloading the self signed cert?

32

@Anthony
We did the certification step on another level and therefore had no need to investigate further.
Therefore I have not tried anything more.

33

Hello Guys,

I have been trying to set up certs using Lets Encrypt in K8s via secrets too. I am have been able to mount the secret (containing the tls.crt and tls.key). The pod fails with

caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: OpenSearchException[Unable to read /usr/share/opensearch/config/certs/tls.crt (/usr/share/opensearch/config/certs/tls.crt). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath]
34

HI @Anthony @skunz : Did you try to set up openid connect as backend for auth?

I have these issues while authenticating:
Edge: ERR_TOO_MANY_REDIRECTS
FireFox: The page isn’t redirecting properly

Any suggestions ?
35

@Malone Could you open a new thread and report your issue there.
Please also include config.yml content.

36

@pablo Thanks , got it working.

38

@powerful_clouds Could you open a new thread and share your current opensearch.yml

39

Hi @skunz followed your progress on using the certificates and auth for Opensearch, I was wondering if you were able to document the whole process in detail and would be able to provide any link to it.

Thank You

1 Like