Kibana startup failure after configuring SSL/TLS

Security
1

I have configured self signed certificates and replaced the demo certificates. Kibana simply fails to start. I have also ran securityadmin.sh to get the other things working.
Note that I am fairly confident of ElasticSearch settings because I tested other services that are successfully able to use TLS to talk to ElasticSearch, such as Fluentd. I also tested ElasticSearch and its certificates using Curl - that works.
Its just something about Kibana apparently. If I don’t use kibana with the SSL settings, it starts ok and works fine, but just with http, obviously.

I am using docker-compose and ES version 7.7.

Kibana fails with a message like -

kibana    | {"type":"log","@timestamp":"2020-07-02T07:09:36Z","tags":["info","savedobjects-service"],"pid":1,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
kibana    | {"type":"log","@timestamp":"2020-07-02T07:09:36Z","tags":["error","savedobjects-service"],"pid":1,"message":"Unable to retrieve version information from Elasticsearch nodes."}

In the Kibana environment in docker-compose.yml,
ELASTICSEARCH_URL: https://es:9200
ELASTICSEARCH_HOSTS: https://es:9200
SERVER_SSL_ENABLED: “true”
SERVER_SSL_KEY: /usr/share/kibana/config/kibana-key.pem
SERVER_SSL_CERTIFICATE: /usr/share/kibana/config/kibana.pem

Also have a kibana.yml specified,
elasticsearch.ssl.certificateAuthorities: [“/usr/share/kibana/config/root-ca.pem”]
elasticsearch.ssl.verificationMode: certificate
server.port: 5601
server.ssl.enabled: true
server.ssl.certificate: /usr/share/kibana/config/kibana.pem
server.ssl.key: /usr/share/kibana/config/kibana-key.pem

The same is picked up through volumes in docker-compose,
volumes:

  • ./kibana/custom-kibana.yml:/usr/share/kibana/config/kibana.yml
  • ./kibana/root-ca.pem:/usr/share/kibana/config/root-ca.pem
  • ./kibana/kibana.pem:/usr/share/kibana/config/kibana.pem
  • ./kibana/kibana-key.pem:/usr/share/kibana/config/kibana-key.pem

Thoughts … ?

2

Gurus, I have my back to the wall on this…

Even if you provide a laundry list of things I can verify and nothing specific, I will proceed with those. Best, if you provide an example config with Kibana-ES and TLS enabled.

3

An update : If I tested further with,

  1. Disabling the certificate verification between Kibana and ElasticSearch,
  2. Have TLS enabled for Kibana Server-Browser
  3. Have TLS verification between ElasticSearch and other services, like Fluentd

Things work correctly.
So my only missing piece is how to make TLS between Kibana and ElasticSearch work.
Note that when I try that, I also added the keys in kibana.yml,
elasticsearch.ssl.certificateAuthorities: “/usr/share/kibana/config/root-ca.pem”
elasticsearch.ssl.certificate: /usr/share/kibana/config/kibana.pem
elasticsearch.ssl.key: /usr/share/kibana/config/kibana-key.pem
elasticsearch.ssl.verificationMode: certificate