Where is opensearch-dashboard-cert set up? I can see opensearch-tls in extraVolumes, but not opensearch-dashboard-cert
It is also generated via cert-manager and only the secret is needed.
So there is no need to mount this cert at the current time.
More important is the connection from extern to the OpenSearch-Cluster.
@skunz
to be honest I havenât had much experience with ingress and not sure why you are using nginx-external.
But I did notice that in my config ingress tls section also has hosts:
tls:
- hosts:
- https-example.foo.com
secretName: testsecret-tls
Overall the error you are seeing relates to certificates, it would seem the cert/config from ingress is causing this
just to chime in here with something important to keep in mind when working with cert-manager and/or letâs encrypt: these certificates are only valid for a limited period of time, after which they get renewed. opensearch wonât notice this automatically (unless thereâs a new feature iâve missed) and youâll instead have to do an API call (or use the admin shell script) to tell it to load the new certificates. depending on whether the root CA also changed (shouldnât be an issue in your case when youâre using letâs encrypt, but this is an issue when using a self-signed CA with e.g. cert-manager) this will be extremely tricky to handle.
if you figure out how to handle this properly please let us know 
@skunz
After some testing locally using minikube, I got it working using below config for ingress:
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
hosts:
- host: chart-example.local
paths:
- path: /
backend:
serviceName: opensearch-dashboards
servicePort: 5601
tls:
- secretName: chart-example-tls
hosts:
- chart-example.local
After helm install, I did get 502 Bad Gateway for a couple of minutes while OSDashboards was getting started. After this everything worked as expected.
Hope this helps
Since the certificate details are stored in a secret which is then mounted, wonât it be up to date automatically?
The secret will be updated by the cert-manager if the certificate is renewed. Will the mounted files (tls.crt, tls.key) not always refer the current secret content?
It was an missing configuration in the ingress of OpenSearch:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
This is by default set to âHTTPâ. After I set OpenSearch to communicate over https the Nginx server receives the wrong protocol and therefore throws an error.
After I added this line the âBad Gatewayâ-Error is gone.
Now I wanted to activate the SSl-Verificiation in OpenSearch Dashboards:
config:
# Default OpenSearch Dashboards configuration from docker image of Dashboards
opensearch_dashboards.yml:
opensearch.ssl.verificationMode: full
opensearch.ssl.certificateAuthorities: [ "/usr/share/opensearch-dashboards/config/certs/tls.crt" ]
If I upgrade my deployment, Dashboards is not working anymore:
{"type":"log","@timestamp":"2021-11-16T10:40:34Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: unable to get issuer certificate"}
Do I need to split my tls.crt-file into the intermediate and root cert?
https://github.com/opensearch-project/documentation-website/issues/144
@skunz
In this case kibana/openDashboards is looking for a root-ca, which is not provided as part of the 3 certs received from lets encrypt. Therefore to get this working locally I had to download root-ca from lets encrypt here
The name of the downloaded cert in my case is isrgrootx1.pem (itâs the self-signed .pem option).
No modification was needed to the 3in1.pem received originally.
I had a look into the 3 certs which are part of tls.crt:
- <domain>
- R3
- ISRG Root X1
So I think it is part of the file (or the cert chain) but it is not recognized. Maybe it has to be in that single file. I will try it your way and report back.
Thanks in advance!
the mounted files will be updated by k8s, but opensearch-security - to my knowledge - does not monitor the filesystem for changes and will thus not load the new certificates. instead, youâll either have to restart the node or have to trigger a hot-reload through an API call (though i couldnât find anything in the opensearch-security docs about this, but Search Guard supports this and iâd thus presume that opensearch-security will have the same feature).
@skunz Did you get this working without downloading the self signed cert?
@Anthony
We did the certification step on another level and therefore had no need to investigate further.
Therefore I have not tried anything more.
Hello Guys,
I have been trying to set up certs using Lets Encrypt in K8s via secrets too. I am have been able to mount the secret (containing the tls.crt and tls.key). The pod fails with
caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: OpenSearchException[Unable to read /usr/share/opensearch/config/certs/tls.crt (/usr/share/opensearch/config/certs/tls.crt). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath]
HI @Anthony @skunz : Did you try to set up openid connect as backend for auth?
I have these issues while authenticating:
Edge: ERR_TOO_MANY_REDIRECTS
FireFox: The page isnât redirecting properly
Any suggestions ?@Malone Could you open a new thread and report your issue there.
Please also include config.yml content.
@pablo Thanks , got it working.
@powerful_clouds Could you open a new thread and share your current opensearch.yml
Hi @skunz followed your progress on using the certificates and auth for Opensearch, I was wondering if you were able to document the whole process in detail and would be able to provide any link to it.
Thank You
1 Like
Hi @rnutakki,
sorry for the delay.
I have no further documentation.
Did you get everything working?