Cannot get TLS to work

ecklf · May 14, 2019, 1:46pm

I tried the following approaches for generating the certificates:

I used the output of openssl x509 -subject -nameopt RFC2253 -noout -in node.pem to fill out the config for nodes_dn and admin_dn.

I am always stuck on this error. Getting cluster health only shows one node.

odfe-node1 | [2019-05-14T13:26:29,436][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [vMkcCJQ] ElasticsearchException[Illegal parameter in http or transport request found.
odfe-node1 | This means that one node is trying to connect to another with
odfe-node1 | a non-node certificate (no OID or opendistro_security.nodes_dn incorrect configured) or that someone
odfe-node1 | is spoofing requests. Check your TLS certificate setup as described in documentation]

I̶f̶ ̶I̶ ̶e̶x̶e̶c̶u̶t̶e̶ ̶s̶e̶c̶u̶r̶i̶t̶y̶a̶d̶m̶i̶n̶.̶s̶h̶ ̶I̶ ̶a̶m̶ ̶b̶e̶i̶n̶g̶ ̶t̶o̶l̶d̶ ̶t̶h̶a̶t̶ ̶I̶ ̶a̶m̶ ̶n̶o̶t̶ ̶a̶n̶ ̶a̶d̶m̶i̶n̶ ̶u̶s̶e̶r̶.̶ ̶E̶v̶e̶n̶ ̶t̶h̶o̶u̶g̶h̶ ̶I̶ ̶c̶o̶p̶y̶ ̶p̶a̶s̶t̶e̶d̶ ̶f̶r̶o̶m̶ ̶t̶h̶e̶ ̶d̶o̶c̶s̶.̶

Problem still persists after running securityadmin successfully

elasticsearch-node1.yml (node2 is the same but with node2 certs/CN)

cluster.name: "docker-cluster"

network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP

# set to 1 to allow single node clusters

# Details: https://github.com/elastic/elasticsearch/pull/17288

discovery.zen.minimum_master_nodes: 1

######## Start OpenDistro for Elasticsearch Security Demo Configuration ########

# WARNING: revise all the lines below before you go into production

opendistro_security.ssl.transport.pemcert_filepath: odfe-node1.pem

opendistro_security.ssl.transport.pemkey_filepath: odfe-node1.key

opendistro_security.ssl.transport.pemtrustedcas_filepath: root-ca.pem

opendistro_security.ssl.transport.enforce_hostname_verification: false

opendistro_security.ssl.http.enabled: true

opendistro_security.ssl.http.pemcert_filepath: odfe-node1.pem

opendistro_security.ssl.http.pemkey_filepath: odfe-node1.key

opendistro_security.ssl.http.pemtrustedcas_filepath: root-ca.pem

opendistro_security.allow_default_init_securityindex: true

opendistro_security.authcz.admin_dn:

- emailAddress=support@example.com,CN=admin,OU=Inc.,O=EXAMPLE,L=Munich,ST=Bavaria,C=DE

opendistro_security.nodes_dn:

- emailAddress=support@example.com,CN=node1.example.com,OU=Inc.,O=EXAMPLE,L=Munich,ST=Bavaria,C=DE

opendistro_security.audit.type: internal_elasticsearch

opendistro_security.enable_snapshot_restore_privilege: true

opendistro_security.check_snapshot_restore_write_privileges: true

opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]

cluster.routing.allocation.disk.threshold_enabled: false

node.max_local_storage_nodes: 3

######## End OpenDistro for Elasticsearch Security Demo Configuration ########

docker-compose.yml

version: '3'
services:
  odfe-node1:
    image: amazon/opendistro-for-elasticsearch:0.9.0
    container_name: odfe-node1
    environment:
      - cluster.name=odfe-cluster
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - odfe-data1:/usr/share/elasticsearch/data
      - ./root-ca.pem:/usr/share/elasticsearch/config/root-ca.pem
      - ./admin.pem:/usr/share/elasticsearch/config/admin.pem
      - ./admin.key:/usr/share/elasticsearch/config/admin.key
      - ./odfe-node1.pem:/usr/share/elasticsearch/config/odfe-node1.pem
      - ./odfe-node1.key:/usr/share/elasticsearch/config/odfe-node1.key
      - ./elasticsearch-node1.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - odfe-net
  odfe-node2:
    image: amazon/opendistro-for-elasticsearch:0.9.0
    container_name: odfe-node2
    environment:
      - cluster.name=odfe-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.zen.ping.unicast.hosts=odfe-node1
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - odfe-data2:/usr/share/elasticsearch/data
      - ./root-ca.pem:/usr/share/elasticsearch/config/root-ca.pem
      - ./admin.pem:/usr/share/elasticsearch/config/admin.pem
      - ./admin.key:/usr/share/elasticsearch/config/admin.key
      - ./odfe-node2.pem:/usr/share/elasticsearch/config/odfe-node2.pem
      - ./odfe-node2.key:/usr/share/elasticsearch/config/odfe-node2.key
      - ./elasticsearch-node2.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    networks:
      - odfe-net
  kibana:
    image: amazon/opendistro-for-elasticsearch-kibana:0.9.0
    container_name: odfe-kibana
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      ELASTICSEARCH_URL: https://odfe-node1:9200
      SERVER_SSL_ENABLED: "true"
      SERVER_SSL_KEY: /usr/share/kibana/config/kibana.key
      SERVER_SSL_CERTIFICATE: /usr/share/kibana/config/kibana.pem
    volumes:
      - ./root-ca.pem:/usr/share/kibana/config/root-ca.pem
      - ./kibana.pem:/usr/share/kibana/config/kibana.pem
      - ./kibana.key:/usr/share/kibana/config/kibana.key
    networks:
      - odfe-net
volumes:
  odfe-data1:
  odfe-data2:

networks:
  odfe-net:

How to fix this?

hardik-k-shah · May 16, 2019, 12:15am

I guess node_dn follow format as below with quotes.

opendistro_security.nodes_dn:
  - "CN=*.example.com, OU=SSL, O=Test, L=Test, C=DE"
  - "CN=node.other.com, OU=SSL, O=Test, L=Test, C=DE"

Can you please try similar format?

aetter · May 16, 2019, 4:47pm

I suspect single or double quotes are both fine (I’ve always used single), but it’s important not to include whitespace between elements. So "CN=*.example.com,OU=SSL,O=Test,L=Test,C=DE". I’ll tweak the docs to be more clear about the quotes.

ecklf · May 16, 2019, 5:27pm

As @aetter mentioned you should not include any whitespace in that string. I got it resolved by adding both nodes on each elasticsearch-nodeX.yml file (sadly not mentioned in the docs).

aetter · May 16, 2019, 5:58pm

Hi ecklf, I’d argue that it was included on a different page, but I changed a couple sentences to further clarify. I appreciate the heads-up.

ecklf · May 16, 2019, 8:12pm

Awesome, thank you! In case you are able to moderate the blog post it might be a great addition aswell

ogulman · May 29, 2019, 9:34am

hi guys, could you please explain me, for witch purposes we need to use admin certificate?

ecklf · May 31, 2019, 6:00am

“Admin certificates are client certificates that have elevated rights to perform administrative tasks. […] Admin certificates are configured in elasticsearch.yml by simply stating their DN(s). You can use any valid client certificate as an admin certificate.”

ThibaudF · November 28, 2019, 12:52pm

Hello Guys !
I used to work with self-signed certificates and recently switched to PKI ones.
My problem is now I can’t find a way to make the admin certificates work.
Do these certificates need special parameter? like search guard documentation showed up (Search Guard Doc) :

admin If set to true, this certificate will be marked as admin certificate in the generated configuration snippet.
Note that you need to mark at least one client certificate as admin certificate.

My admin certs generated have Extended Key Usage serverAuth,clientAuth like the nodes of the cluster.
If I use the certs, I get unauthorized.

Can you help me on this?
Thank you !
Thi

ogulman · November 29, 2019, 11:00am

I succeed only with self-signed admin certificates with specifying all parameters in: opendistro_security.authcz.admin_dn:, moreover, I’m failed with tls between elastic search nodes in my cluster after starting using PKI cert, and now reconfigure to a single node.

ThibaudF · December 3, 2019, 11:09am

Hello @ogulman !
Thank you for the answer, if you want you can create a new topic with your problems in details, I can try to help you.

This week I’m gonna try something, I think I found the source of my errors :

elasticsearch.yml :

opendistro_security.authcz.admin_dn:
  - 'emailAddress=tt@tt.cie,CN=adm-srv00.cie.dev,O=CIE,C=FR'

opendistro_security.nodes_dn:
  - 'emailAddress=tt@tt.cie,CN=adm*,O=CIE,C=FR"

I have to change opendistro_security_nodes to not include the admin certificates. adm* includes adm-srv00.

I will try and give some feedbacks.

UPDATE 12/17/2019 :
I changed the opendistro_security.nodes_dn configuration to not include the admin certificates. This fixed my problem !

Anyone knows the best practice to restart a cluster to reload configurations?
Turn off Logstash, turn off every nodes, change elasticsearch.yml then restart everything?
I don’t want to corrupt data …

Thi

Topic		Replies	Views
Cerificate error Open Source Elasticsearch and Kibana	3	5272	August 11, 2019
SSL Woes with OpenDistro on Docker Security	6	1066	May 31, 2021
Configuring TLS Certificates on Open Distro for Elasticsearch Security	1	1742	May 10, 2021
Multiple Node Deployment on AWS ECS Security	1	2098	June 8, 2020
Security.nodes_dn incorrect configured Security	1	1021	February 7, 2023

Cannot get TLS to work

Related topics