HTTPS/TLS with own PKI

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): Graylog 7.0 + mongodb 8.x + opensearch/opensearch dashboards 2.19.4
unbuntu 24.04 virtual server
install made by apt package

Describe the issue:
one opensearch node but i will make a cluster later
Try to follow this link : Configuring TLS certificates - OpenSearch Documentation

Use my own corporate pki to configure opensearch in secure mode and i don’t understant all the process
i have a pfx, crt et key file

generate certificate by these commands :

openssl pkcs12 -in certificate.pfx -nokeys -out fullchain.pem
awk ‘BEGIN {c=0} /BEGIN CERTIFICATE/ {c++} c==1 {print}’ fullchain.pem > node.crt
awk ‘BEGIN {c=0} /BEGIN CERTIFICATE/ {c++} c>1 {print}’ fullchain.pem > ca.crt
openssl pkcs12 -in certificate.pfx -nocerts -out node_key_encrypted.pem
openssl rsa -in node_key_encrypted.pem -out node.key

chmod 644 \*.crt
chmod 600 node.key

chown -R opensearch:opensearch /etc/opensearch/*

Control the file* *.crt and key, everything seems to be allright (CA root+intermediate present)

Configuration:

cluster.name: graylog-cluster
node.name: ${HOSTNAME}
path.data: /opensearch_data/opensearch
path.logs: /opensearch_data/log
path.repo: /opensearch_data/snapshots
bootstrap.memory_lock: false
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: \[“xxxx”\]
#cluster.initial_cluster_manager_nodes: \[“xxxx”\]
reindex.remote.allowlist: \[“xxxx:9200”\]
plugins.security.disabled: true
plugins.security.allow_unsafe_democertificates: false

\---------------------------------- TLS -----------------------------------

#plugins.security.ssl.transport.enabled: false
#plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/config/certs/node.crt
#plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/config/certs/node.key

#plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/config/certs/ca.crt

#plugins.security.ssl.transport.enforce_hostname_verification: false

#plugins.security.ssl.transport.enable_openssl_if_available: true

\---------------------------------- HTTPS -----------------------------------

#plugins.security.ssl.http.enabled: false
#plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/config/certs/node.crt
#plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/config/certs/node.key
#plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/config/certs/ca.crt
#plugins.security.ssl.http.clientauth_mode: NONE

Relevant Logs or Screenshots:
When i activate ssl/tls and put false on plugins.security.disabled + plugin.security.ssl.transport.enabled= true & plugin.security.ssl.http.enabled = true. I have this issue on log :

mars 19 17:00:34 xxxxx systemd-entrypoint\[603609\]: Likely root cause: java.lang.RuntimeException: Demo certificates found \[25e34a9a5d4f1dceed1666eb624397bf3fe5787a7133cd32838ace0381bce1f7, ba9>

i’m stuck here now, i tried yesterday to move all demo certificates in /etc/opensearch/ inside a separate folder but no way, still this error shows.

Have already done this on elasticsearch 8.17.3 → ssl my own certificate and tls self signed certificate. Maybe i can do this with opensearch ?
Can someone help me to move forward ?

Nearly solved, i found an old post relative to demo certificates:
Demo certificates found? - Security - OpenSearch

remove totally from /etc/opensearch folder demo certificates + add this line in opensearch.yml :
plugins.security.allow_default_init_securityindex: true

then add an override.conf file for blocking creation or usage of demo certs:
/etc/systemd/system/opensearch.service.d/override.conf

[Service]
Environment=“DISABLE_INSTALL_DEMO_CONFIG=true”

service now starts perfectly, http is disable and https access require a user/password.
I tried admin/admin but no access is granted.

Searching in progress :wink:

@cguillaume When you set the below option to true

plugins.security.allow_unsafe_democertificates: false

You must disable the installation of the demo configuration that also includes demo certificates.

DISABLE_INSTALL_DEMO_CONFIG=true"

Regarding the admin password, did you set the below in variables??

OPENSEARCH_INITIAL_ADMIN_PASSWORD

Also, did you provide all security plugin configuration files? Since DEMO configuration is disabled, you must provide your custom configuration files.

Hi Pablo, thanks for your reply.
DISABLE_INSTALL_DEMO_CONFIG=true" must be set inside opensearch.yml ? Or inside override.conf file it will works too ?
”Regarding the admin password, did you set the below in variables?? OPENSEARCH_INITIAL_ADMIN_PASSWORD” –> not at all.

i try some commands this morning:

Export key:

openssl pkcs12 -in votre_certificat.pfx -nocerts -out temp_admin_key.pem -nodes

convert key format to unencrypted PKCS#8.

openssl pkcs8 -topk8 -inform PEM -outform PEM -in temp_admin_key.pem -out admin-key.pem -nocrypt

Extract certificate:

openssl pkcs12 -in votre_certificat.pfx -clcerts -nokeys -out admin.pem

root CA

openssl pkcs12 -in votre_certificat.pfx -cacerts -nokeys -chain -out root-ca.pem

Retrieve DN :

openssl x509 -in admin.pem -inform PEM -subject -nameopt RFC2253 -noout
result : subject=CN=NOM,OU=OPS,O=ORG,L=Paris,C=FR

add it inside opensearch.yml :

plugins.security.authcz.admin_dn:
 -‘CN=NOM,OU=OPS,O=ORG,L=Paris,C=FR’

change admin password
/etc/opensearch/opensearch-security/internal_users.yml

create new hash

/usr/share/opensearch/plugins/opensearch-security/tools/hash.sh -p Password

replace hash inside and save internal_users.yml

adjust plugin security config :

/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh
-cd /etc/opensearch/opensearch-security/
-icl -nhnv
-cacert /etc/opensearch/root-ca.pem
-cert /etc/opensearch/admin.pem
-key /etc/opensearch/admin-key.pem

now i’m stuck here because my certificate is an ServerAuth not a clientAuth.

Command result :

javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:1241)
at org.opensearch.client.RestClient.performRequest(RestClient.java:358)
at org.opensearch.client.RestClient.performRequest(RestClient.java:346)
at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:575)
at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:165)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

log :

\[2026-03-23T10:40:03,484\]\[WARN \]\[i.n.c.AbstractChannelHandlerContext\] \[svprd1195\] An exception ‘OpenSearchSecurityException\[The provided TCP channel is invalid.\]; nested: DecoderException\[javax.net.ssl.SSLHandshakeException: Extended key usage does not permit use for TLS client authentication\]; nested: SSLHandshakeException\[Extended key usage does not permit use for TLS client authentication\]; nested: ValidatorException\[Extended key usage does not permit use for TLS client authentication\];’ \[enable DEBUG level for full stacktrace\] was thrown by a user handler’s exceptionCaught() method while handling the following exception:
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Extended key usage does not permit use for TLS client authentication

From what I understand, modifying security config requires MTLS.

I’ve set the directive plugins.security.ssl.http.clientauth_mode: REQUIRE and OPTIONAL.

This doesn’t change anything as long as my certificate isn’t client-side.

i’m thinking of using self signed certificates for tls and ssl with my own pki. Is this a good idea ?

New chapter of my journey
yesterday i try to follow this page : Generating self-signed certificates - OpenSearch Documentation

i kept ssl for http with my own pki and create all selfsigned certificates for tls

openssl genrsa -out root-ca-key.pem 2048
openssl req -new -x509 -sha256 -key root-ca-key.pem -out root-ca.pem -days 730
openssl genrsa -out admin-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in admin-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out admin-key.pem
openssl req -new -key admin-key.pem -out admin.csr
openssl x509 -req -in admin.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out admin.pem -days 730
openssl genrsa -out node1-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in node1-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out node1-key.pem
openssl req -new -key node1-key.pem -out node1.csr
openssl pkcs8 -inform PEM -outform PEM -in node1-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out node1-key.pem
openssl req -new -key node1-key.pem -out node1.csr
openssl x509 -req -in node1.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out node1.pem -days 730 -extfile opensearch-gen.cnf -extensions v3_node

opensearch-gen.cnf content:

[req]
distinguished_name = req_distinguished_name
attributes = req_attributes
extensions = v3_ca

[req_distinguished_name]

[v3_ca]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[v3_node]
basicConstraints = CA:FALSE
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_namesalt_names

[alt_names]

Modifie ici avec tes vrais noms DNS et IPs

DNS.1 = localhost
DNS.2 = xx
DNS.3 = xx
DNS.4 = xx
DNS.5 = xx
DNS.6 = xx
DNS.7 = xx
IP.1 = 127.0.0.1
IP.2 = xx
chown opensearch:opensearch /etc/opensearch/config/certs/*

retreive DN for admin and node1

openssl x509 -subject -nameopt RFC2253 -noout -in node1.pem
openssl x509 -subject -nameopt RFC2253 -noout -in admin.pem

add them in opensearch.yml ; here is my transport section

plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/config/certs/node1.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/config/certs/node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/config/certs/root-ca.pem
plugins.security.authcz.admin_dn:
-“CN=xx,OU=xx,O=xx,L=xx,ST=xx,C=xx”
plugins.security.nodes_dn:
-“CN=xx,OU=xx,O=xx,L=xx,ST=xx,C=xx”
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.ssl.transport.enforce_hostname_verification: false

Restart opensearch service = OK
access https is still working on my browser and user/password isn’t working

execute commands :

/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh   -cd /etc/opensearch/opensearch-security/   -h 127.0.0.1 -p 9300   -cacert /etc/opensearch/config/certs/root-ca.pem   -cert /etc/opensearch/config/certs/admin.pem   -key /etc/opensearch/config/certs/admin-key.pem   -icl -nhnv

Keep falling in error with various messages and corrective tests :face_with_steam_from_nose:

action : try to add root-ca.pem inside /usr/share/opensearch/jdk/lib/security/cacerts = KO
Result : Caused by: java.security.KeyManagementException: problem accessing trust store or Unable to find a valid certification path to requested target

Try to add it in a new jks and call them as this part of official page “Sample script to convert PEM certificates to keystore and truststore files” this part is blocking me

/etc/opensearch/config/certs# ./trustore_keystore.sh
Enter password for node1-cert.p12
Enter Export Password:
Verifying - Enter Export Password:
4097C5AD4B730000:error:05800074:x509 certificate routines:X509_check_private_key:key values mismatch:../crypto/x509/x509_cmp.c:408:
Enter password for keystore.jks
Import du fichier de clés node1-cert.p12 vers keystore.jks…
Entrez le mot de passe du fichier de clés de destination :
Ressaisissez le nouveau mot de passe :
Entrez le mot de passe du fichier de clés source :
erreur keytool : java.io.IOException: keystore password was incorrect
Enter password for admin-cert.p12

So i tried by another way :

keytool -import -file root-ca.pem -alias root-ca     -keystore admin-truststore.jks     -storepass storeit -noprompt
openssl pkcs12 -export -in admin.pem -inkey admin-key.pem     -out admin.p12 -name admin-cert     -passout pass:storeit
keytool -importkeystore -destkeystore admin-keystore.jks     -srckeystore admin.p12 -srcstoretype PKCS12     -srcstorepass storeit -deststorepass storeit
export JAVA_OPTS=“-Djavax.net.ssl.trustStore=/etc/opensearch/config/certs/admin-truststore.jks
-Djavax.net.ssl.trustStorePassword=storeit
-Djavax.net.ssl.keyStore=/etc/opensearch/config/certs/admin-keystore.jks
-Djavax.net.ssl.keyStorePassword=storeit”
/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh   -cd /etc/opensearch/opensearch-security/   -h 127.0.0.1 -p 9300   -ks /etc/opensearch/config/certs/admin-keystore.jks -kspass storeit   -ts /etc/opensearch/config/certs/admin-truststore.jks -tspass storeit   -icl -nhnv

result : Caused by: org.apache.http.ParseException: Not a valid protocol version: This is not an HTTP port

why is this so difficult to secure communication and apply certificates :face_with_raised_eyebrow:

Finaly it works !!
i tried, restart service and change configuration so many times :face_exhaling:

my review :

first → you cannot mix own pki and self signed certificate, trust me, i test it for you :sweat_smile:
second → no need of keystore/truststore.jks as long as you work with pem certificate : PEM (cert + key + CA) or keystore/truststore JKS / PKCS12 ; i didn’t understand that was 2 distincts part of configuration.
third don’t forget to remove all demo certificates !
last → admin.pem must respect clientauth eku

Execute these commands :
vi admin.cnf

[ req ]
default_bits       = 2048
prompt             = no
default_md         = sha256
distinguished_name = dn
req_extensions     = req_ext

[ dn ]
C  = xx
ST = xx
L  = xx
O  = xx
OU = xx
CN = admin

[ req_ext ]
extendedKeyUsage = clientAuth
keyUsage = digitalSignature, keyEncipherment
chown opensearch:opensearch admin.cnf
openssl x509 -req -in admin.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial   -out admin.pem -days 730 -extfile admin.cnf -extensions req_ext
openssl x509 -in admin.pem -text -noout | grep -A5 “Extended Key Usage”

change your opensearch.yml :

#TLS
plugins.security.disabled: false
plugins.security.allow_default_init_securityindex: true
plugins.security.allow_unsafe_democertificates: false

plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/config/certs/node1.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/config/certs/node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/config/certs/root-ca.pem

plugins.security.authcz.admin_dn:
-“CN=admin,OU=xx,O=xx,L=xx,ST=xx,C=xx”
plugins.security.nodes_dn:
“CN=opensearch-node1,OU=xx,O=xx,L=xx,ST=xx,C=xx”
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.ssl.transport.enforce_hostname_verification: false

#HTTPS
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/config/certs/node1.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/config/certs/node1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/config/certs/root-ca.pem

same certificates for both

to apply security settings target the 9200 port not 9300 (seems to be blocked since OpenSearch 2.12

/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh   -cd /etc/opensearch/opensearch-security/   -h 127.0.0.1   -p 9200   -nhnv   -cert /etc/opensearch/config/certs/admin.pem   -key /etc/opensearch/config/certs/admin-key.pem   -cacert /etc/opensearch/config/certs/root-ca.pem

access to your browser, put your user/password and taddaammm :partying_face:

{
“name” : “xxx”,
“cluster_name” : “xxxx”,
“cluster_uuid” : “A5-GWPJkSYqKctOUico6Sw”,
“version” : {
“distribution” : “opensearch”,
“number” : “2.19.4”,
“build_type” : “deb”,
“build_hash” : “e2e89961c9a327daf514a7ce1320a6189bfd08cd”,
“build_date” : “2025-11-01T02:35:34.092584983Z”,
“build_snapshot” : false,
“lucene_version” : “9.12.3”,
“minimum_wire_compatibility_version” : “7.10.0”,
“minimum_index_compatibility_version” : “7.0.0”
},
“tagline” : “The OpenSearch Project: https://opensearch.org/”
}

Hope all of my research/commands will help some of you.

if you want to use your own pki for SSL/TLS i think it must be working with good eku (serverauth/clientauth). Maybe someone can confirm it works this way too ?

Expiriment is part of the process and progress :grin:

@pablo i’m still waiting for answers about what you said in your reply, i’m curious to know about it.

This must be set as an environmental variable. Setting it will prevent the creation of the demo security configuration in opensearch.yml and demo certificates.

That is correct, both options are environmental variables.

To run securityadmin.sh you must always use a superadmin certificate which DN is listed in plugins.security.authcz.admin_dn and was signed by a CA that is in plugins.security.ssl.http.pemtrustedcas_filepath.
Also, the superadmin certificate must be only a client certificate (Client Authentication in EKU)

There is no need to modify this option as it is not used by the securitydamin.sh script. This script bypasses all security.

This has changed in 2.x. As per the release notes of 2.0.0, the Transport Client Authentication/Authorization has been deprecated. That’s why securityadmin.sh has been moved to 9200 and HTTPS must be enabled.