Cannot get TLS to work

Using sample docker-compose.yml

I tried the following approaches for generating the certificates:

I used the output of openssl x509 -subject -nameopt RFC2253 -noout -in node.pem to fill out the config for nodes_dn and admin_dn.

I am always stuck on this error. Getting cluster health only shows one node.

odfe-node1 | [2019-05-14T13:26:29,436][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [vMkcCJQ] ElasticsearchException[Illegal parameter in http or transport request found.
odfe-node1 | This means that one node is trying to connect to another with
odfe-node1 | a non-node certificate (no OID or opendistro_security.nodes_dn incorrect configured) or that someone
odfe-node1 | is spoofing requests. Check your TLS certificate setup as described in documentation]

I̶f̶ ̶I̶ ̶e̶x̶e̶c̶u̶t̶e̶ ̶s̶e̶c̶u̶r̶i̶t̶y̶a̶d̶m̶i̶n̶.̶s̶h̶ ̶I̶ ̶a̶m̶ ̶b̶e̶i̶n̶g̶ ̶t̶o̶l̶d̶ ̶t̶h̶a̶t̶ ̶I̶ ̶a̶m̶ ̶n̶o̶t̶ ̶a̶n̶ ̶a̶d̶m̶i̶n̶ ̶u̶s̶e̶r̶.̶ ̶E̶v̶e̶n̶ ̶t̶h̶o̶u̶g̶h̶ ̶I̶ ̶c̶o̶p̶y̶ ̶p̶a̶s̶t̶e̶d̶ ̶f̶r̶o̶m̶ ̶t̶h̶e̶ ̶d̶o̶c̶s̶.̶

Problem still persists after running securityadmin successfully

elasticsearch-node1.yml (node2 is the same but with node2 certs/CN) "docker-cluster"

# minimum_master_nodes need to be explicitly set when bound on a public IP

# set to 1 to allow single node clusters

# Details:

discovery.zen.minimum_master_nodes: 1

######## Start OpenDistro for Elasticsearch Security Demo Configuration ########

# WARNING: revise all the lines below before you go into production

opendistro_security.ssl.transport.pemcert_filepath: odfe-node1.pem

opendistro_security.ssl.transport.pemkey_filepath: odfe-node1.key

opendistro_security.ssl.transport.pemtrustedcas_filepath: root-ca.pem

opendistro_security.ssl.transport.enforce_hostname_verification: false

opendistro_security.ssl.http.enabled: true

opendistro_security.ssl.http.pemcert_filepath: odfe-node1.pem

opendistro_security.ssl.http.pemkey_filepath: odfe-node1.key

opendistro_security.ssl.http.pemtrustedcas_filepath: root-ca.pem

opendistro_security.allow_default_init_securityindex: true





opendistro_security.audit.type: internal_elasticsearch

opendistro_security.enable_snapshot_restore_privilege: true

opendistro_security.check_snapshot_restore_write_privileges: true

opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]

cluster.routing.allocation.disk.threshold_enabled: false

node.max_local_storage_nodes: 3

######## End OpenDistro for Elasticsearch Security Demo Configuration ########


version: '3'
    image: amazon/opendistro-for-elasticsearch:0.9.0
    container_name: odfe-node1
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
        soft: -1
        hard: -1
      - odfe-data1:/usr/share/elasticsearch/data
      - ./root-ca.pem:/usr/share/elasticsearch/config/root-ca.pem
      - ./admin.pem:/usr/share/elasticsearch/config/admin.pem
      - ./admin.key:/usr/share/elasticsearch/config/admin.key
      - ./odfe-node1.pem:/usr/share/elasticsearch/config/odfe-node1.pem
      - ./odfe-node1.key:/usr/share/elasticsearch/config/odfe-node1.key
      - ./elasticsearch-node1.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
      - odfe-net
    image: amazon/opendistro-for-elasticsearch:0.9.0
    container_name: odfe-node2
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
        soft: -1
        hard: -1
      - odfe-data2:/usr/share/elasticsearch/data
      - ./root-ca.pem:/usr/share/elasticsearch/config/root-ca.pem
      - ./admin.pem:/usr/share/elasticsearch/config/admin.pem
      - ./admin.key:/usr/share/elasticsearch/config/admin.key
      - ./odfe-node2.pem:/usr/share/elasticsearch/config/odfe-node2.pem
      - ./odfe-node2.key:/usr/share/elasticsearch/config/odfe-node2.key
      - ./elasticsearch-node2.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - odfe-net
    image: amazon/opendistro-for-elasticsearch-kibana:0.9.0
    container_name: odfe-kibana
      - 5601:5601
      - "5601"
      ELASTICSEARCH_URL: https://odfe-node1:9200
      SERVER_SSL_ENABLED: "true"
      SERVER_SSL_KEY: /usr/share/kibana/config/kibana.key
      SERVER_SSL_CERTIFICATE: /usr/share/kibana/config/kibana.pem
      - ./root-ca.pem:/usr/share/kibana/config/root-ca.pem
      - ./kibana.pem:/usr/share/kibana/config/kibana.pem
      - ./kibana.key:/usr/share/kibana/config/kibana.key
      - odfe-net


How to fix this?

I guess node_dn follow format as below with quotes.

  - "CN=*, OU=SSL, O=Test, L=Test, C=DE"
  - ", OU=SSL, O=Test, L=Test, C=DE"

Can you please try similar format?

I suspect single or double quotes are both fine (I’ve always used single), but it’s important not to include whitespace between elements. So "CN=*,OU=SSL,O=Test,L=Test,C=DE". I’ll tweak the docs to be more clear about the quotes.

As @aetter mentioned you should not include any whitespace in that string. I got it resolved by adding both nodes on each elasticsearch-nodeX.yml file (sadly not mentioned in the docs).

Hi ecklf, I’d argue that it was included on a different page, but I changed a couple sentences to further clarify. I appreciate the heads-up. :+1:

1 Like

Awesome, thank you! In case you are able to moderate the blog post it might be a great addition aswell :wink:

hi guys, could you please explain me, for witch purposes we need to use admin certificate?

Admin certificates are client certificates that have elevated rights to perform administrative tasks. […] Admin certificates are configured in elasticsearch.yml by simply stating their DN(s). You can use any valid client certificate as an admin certificate.”

Hello Guys !
I used to work with self-signed certificates and recently switched to PKI ones.
My problem is now I can’t find a way to make the admin certificates work.
Do these certificates need special parameter? like search guard documentation showed up (Search Guard Doc) :

admin If set to true, this certificate will be marked as admin certificate in the generated configuration snippet.
Note that you need to mark at least one client certificate as admin certificate.

My admin certs generated have Extended Key Usage serverAuth,clientAuth like the nodes of the cluster.
If I use the certs, I get unauthorized.

Can you help me on this?
Thank you !

I succeed only with self-signed admin certificates with specifying all parameters in: opendistro_security.authcz.admin_dn:, moreover, I’m failed with tls between elastic search nodes in my cluster after starting using PKI cert, and now reconfigure to a single node.

Hello @ogulman !
Thank you for the answer, if you want you can create a new topic with your problems in details, I can try to help you.

This week I’m gonna try something, I think I found the source of my errors :slight_smile: :

elasticsearch.yml :

  - 'emailAddress=tt@tt.cie,,O=CIE,C=FR'

  - 'emailAddress=tt@tt.cie,CN=adm*,O=CIE,C=FR"

I have to change opendistro_security_nodes to not include the admin certificates. adm* includes adm-srv00.

I will try and give some feedbacks.

UPDATE 12/17/2019 :
I changed the opendistro_security.nodes_dn configuration to not include the admin certificates. This fixed my problem !

Anyone knows the best practice to restart a cluster to reload configurations?
Turn off Logstash, turn off every nodes, change elasticsearch.yml then restart everything?
I don’t want to corrupt data …