Received fatal alert: unknown_ca while docker installation

OpenSearch
,
1

Versions opensearch 2.5.0 ubuntu 22.04 OCI shell:

Hi guys,

I am trying to install Openseach version: 2.5.0 for Magento

Describe the issue:

I get:

opensearch-project-os01-1 |     at java.lang.Thread.run(Thread.java:833) [?:?]
opensearch-project-os01-1 | **Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca**
opensearch-project-os01-1 |     at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
opensearch-project-os01-1 |     at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
opensearch-project-os01-1 |     at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?]
opensearch-project-os01-1 |     at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
opensearch-project-os01-1 |     at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
opensearch-project-os01-1 |     at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
opensearch-project-os01-1 |     at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
opensearch-project-os01-1 |     at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
opensearch-project-os01-1 |     at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
opensearch-project-os01-1 |     at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
opensearch-project-os01-1 |     at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
opensearch-project-os01-1 |     at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296) ~[netty-handler-4.1.86.Final.jar:4.1.86.Final]
opensearch-project-os01-1 |     at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343) ~[netty-handler-4.1.86.Final.jar:4.1.86.Final]
opensearch-project-os01-1 |     at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[netty-handler-4.1.86.Final.jar:4.1.86.Final]
opensearch-project-os01-1 |     at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.86.Final.jar:4.1.86.Final]
opensearch-project-os01-1 |     at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.86.Final.jar:4.1.86.Final]

Configuration:

sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sudo swapoff -a
Mem:           23987         909       18219           7        4858       22746
Swap:              0           0           0

cat /proc/sys/vm/max_map_count
262144

mkdir -p ~/opensearch-project/certs; cd ~/opensearch-project

touch docker-compose.yml opensearch.yml opensearch_dashboards.yml internal_users.yml

mkdir -p certs/{ca,os-dashboards}

export MYDN="/C=BR/ST=Sao Paulo/L=Sao Paulo/O=tsl"

echo $MYDN
/C=BR/ST=Sao Paulo/L=Sao Paulo/O=tsl

openssl genrsa -out certs/ca/ca.key 2048
openssl req -new -x509 -sha256 -days 1095 -subj "$MYDN/CN=CA" -key certs/ca/ca.key -out certs/ca/ca.pem
openssl genrsa -out certs/ca/admin-temp.key 2048
openssl pkcs8 -inform PEM -outform PEM -in certs/ca/admin-temp.key -topk8 -nocrypt -v1 PBE-SHA1-3DES -out certs/ca/admin.key
openssl req -new -subj "$MYDN/CN=ADMIN" -key certs/ca/admin.key -out certs/ca/admin.csr
openssl x509 -req -in certs/ca/admin.csr -CA certs/ca/ca.pem -CAkey certs/ca/ca.key -CAcreateserial -sha256 -out certs/ca/admin.pem
openssl genrsa -out certs/os-dashboards/os-dashboards-temp.key 2048
openssl pkcs8 -inform PEM -outform PEM -in certs/os-dashboards/os-dashboards-temp.key -topk8 -nocrypt -v1 PBE-SHA1-3DES -out certs/os-dashboards/os-dashboards.key
openssl req -new -subj "$MYDN/CN=osdashboards" -key certs/os-dashboards/os-dashboards.key -out certs/os-dashboards/os-dashboards.csr
openssl x509 -req -in certs/os-dashboards/os-dashboards.csr -CA certs/ca/ca.pem -CAkey certs/ca/ca.key -CAcreateserial -sha256 -out certs/os-dashboards/os-dashboards.pem
rm certs/os-dashboards/os-dashboards-temp.key certs/os-dashboards/os-dashboards.csr

for NODE_NAME in "os01" "os02" 
do
    mkdir "certs/${NODE_NAME}"
    openssl genrsa -out "certs/$NODE_NAME/$NODE_NAME-temp.key" 2048
    openssl pkcs8 -inform PEM -outform PEM -in "certs/$NODE_NAME/$NODE_NAME-temp.key" -topk8 -nocrypt -v1 PBE-SHA1-3DES -out "certs/$NODE_NAME/$NODE_NAME.key"
    openssl req -new -subj "$MYDN/CN=$NODE_NAME" -key "certs/$NODE_NAME/$NODE_NAME.key" -out "certs/$NODE_NAME/$NODE_NAME.csr"
    openssl x509 -req -extfile <(printf "subjectAltName=DNS:localhost,IP:127.0.0.1,DNS:$NODE_NAME") -in "certs/$NODE_NAME/$NODE_NAME.csr" -CA certs/ca/ca.pem -CAkey certs/ca/ca.key -CAcreateserial -sha256 -out "certs/$NODE_NAME/$NODE_NAME.pem"
    rm "certs/$NODE_NAME/$NODE_NAME-temp.key" "certs/$NODE_NAME/$NODE_NAME.csr"
done

done
Certificate request self-signature ok
subject=C = BR, ST = Sao Paulo, L = Sao Paulo, O = tsl, CN = os01
Certificate request self-signature ok
subject=C = BR, ST = Sao Paulo, L = Sao Paulo, O = tsl, CN = os02

sudo chown -R 1000:1000 certs/

 - 

keytool -importcert -file certs/ca/ca.pem -keystore keystore.jks -alias root
Enter keystore password:  
Owner: CN=CA, O=tsl, L=Sao Paulo, ST=Sao Paulo, C=BR
Issuer: CN=CA, O=tsl, L=Sao Paulo, ST=Sao Paulo, C=BR
Serial number: 5aa0c7d557a2a4dca33812f8fe6f9b3aa6442c95
Valid from: Fri Dec 29 20:06:47 UTC 2023 until: Mon Dec 28 20:06:47 UTC 2026
Certificate fingerprints:
         SHA1: B8:8A:A0:45:61:FD:2A:2A:C5:98:F4:E1:4B:5E:BC:89:89:D3:5E:B6
         SHA256: 47:AB:35:5D:B9:95:66:74:01:CB:D1:9D:F9:37:03:C4:E3:A8:C0:2D:96:0E:B6:92:BB:FC:D6:44:B7:0F:01:51
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions: 

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 5F 7E 11 1A 23 1D 49 6E   0F 39 A4 11 D7 17 2A D2  _...#.In.9....*.
0010: 2F A0 A8 42                                        /..B
]
]

#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen: no limit
]

#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 5F 7E 11 1A 23 1D 49 6E   0F 39 A4 11 D7 17 2A D2  _...#.In.9....*.
0010: 2F A0 A8 42                                        /..B
]
]

Trust this certificate? [no]:  yes
Certificate was added to keystore

chmod u+rwx keystore.jks
sudo chown 1000:1000 keystore.jks


#generate hash keys for users:
docker run -it --rm opensearchproject/opensearch:2.5.0 sh -c "/usr/share/opensearch/plugins/opensearch-security/tools/hash.sh"

vim internal_users.yml
---
# This is the internal user database
# The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh

_meta:
  type: "internalusers"
  config_version: 2

# Define your internal users here

admin:
  hash: "password"
  reserved: true
  backend_roles:
  - "admin"
  description: "Admin User"

kibanaserver:
  hash: "password"
  reserved: true
  description: "OpenSearch Dashboards user"

vim docker-compose.yml


version: '3.7'

services:

  os01:
    restart: always
    image: opensearchproject/opensearch:2.5.0
    environment:
      OPENSEARCH_JAVA_OPTS: "-Xms4g -Xmx4g" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      node.name: os01
      discovery.seed_hosts: os01,os02
      cluster.initial_master_nodes: os01,os02
      plugins.security.ssl.transport.pemkey_filepath: certificates/os01/os01.key # relative path
      plugins.security.ssl.transport.pemcert_filepath: certificates/os01/os01.pem
      plugins.security.ssl.http.pemkey_filepath: certificates/os01/os01.key
      plugins.security.ssl.http.pemcert_filepath: certificates/os01/os01.pem
     
      DISABLE_INSTALL_DEMO_CONFIG: "true"
      JAVA_HOME: /usr/share/opensearch/jdk
      bootstrap.memory_lock: "true" # along with the memlock settings below, disables swapping
      network.host: "0.0.0.0"
    ulimits: 
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test: ["CMD-SHELL", "curl --silent --fail https://localhost:9200/_cluster/health || exit 1"]
      interval: 1m30s
      timeout: 10s
      retries: 3
      start_period: 40s
    volumes:
      - "./opensearch.yml:/usr/share/opensearch/config/opensearch.yml"
      - "./internal_users.yml:/usr/share/opensearch/config/opensearch-security/internal_users.yml"
      - "os-data1:/usr/share/opensearch/data"
      - "./certs:/usr/share/opensearch/config/certificates:ro"
      
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
  
  os02:
    restart: always
    image: opensearchproject/opensearch:2.5.0
    
    environment:
      OPENSEARCH_JAVA_OPTS: "-Xms4g -Xmx4g" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      node.name: os02
      discovery.seed_hosts: os01,os02
      cluster.initial_master_nodes: os01,os02
      plugins.security.ssl.transport.pemkey_filepath: certificates/os02/os02.key # relative path
      plugins.security.ssl.transport.pemcert_filepath: certificates/os02/os02.pem
      plugins.security.ssl.http.pemkey_filepath: certificates/os02/os02.key
      plugins.security.ssl.http.pemcert_filepath: certificates/os02/os02.pem
      
      DISABLE_INSTALL_DEMO_CONFIG: "true"
      JAVA_HOME: /usr/share/opensearch/jdk
      bootstrap.memory_lock: "true" # along with the memlock settings below, disables swapping
      network.host: "0.0.0.0"
    ulimits: 
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test: ["CMD-SHELL", "echo QUIT | openssl s_client -connect localhost:9200 &> /dev/null || exit 1"]
      interval: 1m30s
      timeout: 10s
      retries: 3
      start_period: 40s
    
    volumes:
      - "./opensearch.yml:/usr/share/opensearch/config/opensearch.yml"
      - "./internal_users.yml:/usr/share/opensearch/config/opensearch-security/internal_users.yml"
      - "os-data2:/usr/share/opensearch/data"
      - "./certs:/usr/share/opensearch/config/certificates:ro"
  
  osdashboards:
    restart: always
    image: opensearchproject/opensearch:2.5.0
    
    ports:
      - 5601:5601
    volumes:
      - "./certs:/usr/share/opensearch-dashboards/config/certificates:ro"
      - "./opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml"
    environment:
      OPENSEARCH_HOSTS: '["https://os01:9200","https://os02:9200"]' # must be a string with no spaces when specified as an environment variable
      DISABLE_INSTALL_DEMO_CONFIG: "true"
    healthcheck:
      test: ["CMD-SHELL", "echo QUIT | openssl s_client -connect localhost:9200 &> /dev/null || exit 1"]
      interval: 1m30s
      timeout: 10s
      retries: 3
      start_period: 40s
volumes:
  os-data1:
  os-data2:
  os-data3:

vim opensearch.yml
cluster.name: os-cluster
network.host: 0.0.0.0

bootstrap.memory_lock: "true" # along with the memlock settings below, disables swapping

plugins.security.allow_unsafe_democertificates: true
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemtrustedcas_filepath: certificates/ca/ca.pem
plugins.security.ssl.http.pemcert_filepath: certificates/ca/admin.pem
plugins.security.ssl.http.pemkey_filepath: certificates/ca/admin.key
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.pemtrustedcas_filepath: certificates/ca/ca.pem
plugins.security.ssl.transport.pemcert_filepath: certificates/ca/admin.pem
plugins.security.ssl.transport.pemkey_filepath: certificates/ca/admin.key
plugins.security.ssl.transport.enforce_hostname_verification: false

plugins.security.authcz.admin_dn:
 - 'CN=ADMIN,O=tsl,L=Sao Paulo,ST=Sao Paulo,C=BR'
plugins.security.nodes_dn:
 - 'CN=os00,O=tsl,L=Sao Paulo,ST=Sao Paulo,C=BR'
 - 'CN=os01,O=tsl,L=Sao Paulo,ST=Sao Paulo,C=BR'
 - 'CN=os02,O=tsl,L=Sao Paulo,ST=Sao Paulo,C=BR'
 - 'CN=os03,O=tsl,L=Sao Paulo,ST=Sao Paulo,C=BR'

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
opendistro_security.audit.config.disabled_rest_categories: NONE
opendistro_security.audit.config.disabled_transport_categories: NONE


vim opensearch_dashboards.yml

server.name: osdashboards
server.host: "0.0.0.0"
opensearch.username: kibanaserver
opensearch.password: password

opensearch.requestHeadersWhitelist: [authorization, securitytenant]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]

# Use this setting if you are running opensearch-dashboards without https
opensearch_security.cookie.secure: false

# Encrypt traffic between the browser and OpenSearch-Dashboards
server.ssl.enabled: false
	#server.ssl.certificate: "/usr/share/opensearch-dashboards/config/certificates/os-dashboards/os-dashboards.pem"
	#server.ssl.key: "/usr/share/opensearch-dashboards/config/certificates/os-dashboards/os-dashboards.key"

# Encrypt traffic between OpenSearch-Dashboards and Opensearch
opensearch.ssl.certificateAuthorities: ["/usr/share/opensearch-dashboards/config/certificates/ca/ca.pem"]
opensearch.ssl.verificationMode: full

..
└── opensearch-project
    ├── certs
    │   ├── ca
    │   │   ├── admin-temp.key
    │   │   ├── admin.csr
    │   │   ├── admin.key
    │   │   ├── admin.pem
    │   │   ├── ca.key
    │   │   └── ca.pem
    │   ├── os-dashboards
    │   │   ├── os-dashboards.key
    │   │   └── os-dashboards.pem
    │   ├── os01
    │   │   ├── os01.key
    │   │   └── os01.pem
    │   └── os02
    │       ├── os02.key
    │       └── os02.pem
    ├── docker-compose.yml
    ├── internal_users.yml
    ├── keystore.jks
    ├── opensearch.yml
    └── opensearch_dashboards.yml
docker compose up -d
docker compose ps

#validade securityadmin users
docker compose exec os01 bash -c "chmod +x plugins/opensearch-security/tools/securityadmin.sh && bash plugins/opensearch-security/tools/securityadmin.sh -cd config/opensearch-security -icl -nhnv -cacert config/certificates/ca/ca.pem -cert config/certificates/ca/admin.pem -key config/certificates/ca/admin.key -h localhost"

Relevant Logs or Screenshots:
It gives the same error whether or not I use keystore and declare it on the docker-compose environment.
The root is:

openssl x509 -in ~/opensearch-project/certs/ca/ca.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            5a:a0:c7:d5:57:a2:a4:dc:a3:38:12:f8:fe:6f:9b:3a:a6:44:2c:95
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = BR, ST = Sao Paulo, L = Sao Paulo, O = tsl, CN = CA
        Validity
            Not Before: Dec 29 20:06:47 2023 GMT
            Not After : Dec 28 20:06:47 2026 GMT
        Subject: C = BR, ST = Sao Paulo, L = Sao Paulo, O = tsl, CN = CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:93:b9:d8:0f:fb:bb:7f:0a:83:dd:57:6e:72:df:
                    e5:e9:c5:da:70:cd:61:47:9b:97:6a:d6:fc:47:c6:
                    77:8d:af:83:15:a1:cb:e5:98:72:a5:fb:b8:60:22:
                    2a:90:f6:81:af:aa:db:23:1f:60:3d:08:dc:4a:59:
                    32:10:0f:c7:ce:92:8d:f3:f3:57:9c:11:e4:1b:f6:
                    fe:12:47:5c:c3:56:c6:36:6d:c8:fc:0c:0c:4b:31:
                    b4:e7:dd:c3:cb:13:5c:33:f8:63:55:61:a7:9d:f8:
                    a6:fc:b1:27:4f:a7:b2:b2:58:a7:17:29:bb:ef:91:
                    cc:ec:63:be:ab:c6:b4:e6:cd:14:b7:c8:fd:0d:4b:
                    e3:34:ba:cc:29:a5:f9:46:29:bb:fe:3a:0f:67:46:
                    81:6a:d4:66:a0:07:59:c7:b2:07:36:d5:1a:d5:67:
                    5f:c0:3d:bf:0d:91:86:d1:95:7c:3a:19:b9:ba:ac:
                    bd:fe:3a:b7:d1:88:25:2f:a9:10:3f:ff:48:9d:d6:
                    3c:c2:0f:96:6c:ba:87:f9:e7:7e:5c:c6:65:25:f5:
                    92:08:1a:10:be:ee:8f:5f:d4:9c:87:f1:b6:37:a2:
                    2c:fc:a2:69:a4:e0:fa:50:58:39:e0:cb:9c:7e:7a:
                    86:17:ef:07:17:a9:8b:3e:f3:4a:78:d9:5b:cf:25:
                    e2:e9
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                5F:7E:11:1A:23:1D:49:6E:0F:39:A4:11:D7:17:2A:D2:2F:A0:A8:42
            X509v3 Authority Key Identifier: 
                5F:7E:11:1A:23:1D:49:6E:0F:39:A4:11:D7:17:2A:D2:2F:A0:A8:42
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        67:14:36:89:46:e2:76:8b:7a:3b:f1:4b:f0:8c:7c:14:85:24:
        02:e0:1b:26:f9:43:a6:8c:a9:05:c4:9d:96:0f:26:73:36:eb:
        90:e5:df:ef:90:06:1c:d0:2e:66:6f:df:c3:40:ba:ed:7f:c6:
        d1:d0:02:79:32:29:57:75:68:39:d0:1e:27:89:38:de:37:2b:
        c3:92:c5:80:fe:15:34:9d:7c:16:77:75:92:c8:60:ca:d0:36:
        10:7b:77:ea:1f:51:00:ef:78:fc:b0:e8:e6:37:61:be:70:c8:
        f7:fd:42:ce:be:76:d1:b6:f3:d0:4c:90:b2:87:fe:e3:92:97:
        c7:10:b0:0a:97:8c:9e:d6:b5:6d:91:79:cb:bd:14:08:08:40:
        b1:e2:b8:43:78:46:b3:84:48:b1:ed:19:0c:5b:10:b1:2c:22:
        29:66:df:3f:0d:1e:4a:9c:93:a2:4a:4d:13:ac:5c:8c:7f:d9:
        f0:33:12:93:4b:a8:0b:fa:05:56:e4:06:bd:65:a8:ab:fc:cb:
        f6:aa:22:7e:ae:54:67:6a:e9:38:3b:74:ce:dd:cc:7b:67:69:
        52:72:78:b8:d2:12:0e:37:65:bd:7c:26:ce:98:2a:fc:a5:02:
        33:20:00:62:28:96:29:d1:2a:e1:54:30:78:eb:09:03:2e:85:
        5a:68:de:6f

All the talks here related to this error have been read, resoluitions are not very clear…

Is the alternative: “use keystore and truststore files in JKS or PKCS12/PFX format” the only way? What am I missing please… thanks in advance!

2

I switched to the alternative path:

cat opensearch.yml
cluster.name: os-cluster
network.host: 0.0.0.0

bootstrap.memory_lock: "true" # along with the memlock settings below, disables swapping

plugins.security.allow_unsafe_democertificates: true
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false

plugins.security.ssl.transport.keystore_filepath: certs/os01/os01.p12
plugins.security.ssl.transport.keystore_password:
plugins.security.ssl.transport.truststore_filepath: certs/ca/ca.p12
plugins.security.ssl.transport.truststore_password: 

plugins.security.ssl.http.keystore_filepath: certs/os01/os01.p12
plugins.security.ssl.http.keystore_password:
plugins.security.ssl.http.truststore_filepath: certs/ca/ca.p12
plugins.security.ssl.http.truststore_password: 

plugins.security.authcz.admin_dn:
 - 'CN=ADMIN,O=tsl,L=Sao Paulo,ST=Sao Paulo,C=BR'
plugins.security.nodes_dn:
 - 'CN=os00,O=tsl,L=Sao Paulo,ST=Sao Paulo,C=BR'
 - 'CN=os01,O=tsl,L=Sao Paulo,ST=Sao Paulo,C=BR'
 - 'CN=os02,O=tsl,L=Sao Paulo,ST=Sao Paulo,C=BR'
 - 'CN=os03,O=tsl,L=Sao Paulo,ST=Sao Paulo,C=BR'

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
opendistro_security.audit.config.disabled_rest_categories: NONE
opendistro_security.audit.config.disabled_transport_categories: NONE

version: '3.7'

services:

  os01:
    restart: always
    image: opensearchproject/opensearch:2.5.0
    environment:
      OPENSEARCH_JAVA_OPTS: "-Xms4g -Xmx4g" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      node.name: os01
      discovery.seed_hosts: os01
      cluster.initial_master_nodes: os01
      plugins.security.ssl.transport.keystore_filepath: certs/os01/os01.p12
      plugins.security.ssl.transport.keystore_password: 
      plugins.security.ssl.http.keystore_filepath: certs/os01/os01.p12
      plugins.security.ssl.http.keystore_password: 
      DISABLE_INSTALL_DEMO_CONFIG: "true"
      JAVA_HOME: /usr/share/opensearch/jdk
      bootstrap.memory_lock: "true" # along with the memlock settings below, disables swapping
      network.host: "0.0.0.0" 
    ulimits: 
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test: ["CMD-SHELL", "curl --silent --fail https://localhost:9200/_cluster/health || exit 1"]
      interval: 1m30s
      timeout: 10s
      retries: 3
      start_period: 40s
    volumes:
      - "./opensearch.yml:/usr/share/opensearch/config/opensearch.yml"
      - "./internal_users.yml:/usr/share/opensearch/config/opensearch-security/internal_users.yml"
      - "os-data1:/usr/share/opensearch/data"
      - "./certs:/usr/share/opensearch/config/certificates:ro"
      
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer

volumes:
  os-data1:
  os-data2:
  os-data3:

java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[Unable to read /usr/share/opensearch/config/certs/os01/os01.p12 (/usr/share/opensearch/config/certs/os01/os01.p12). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.keystore_filepath]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.checkPath(DefaultSecurityKeyStore.java:983)
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.resolve(DefaultSecurityKeyStore.java:235)
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:284)
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:256)
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:177)
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218)
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:263)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:782)
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731)
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533)
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195)
        at org.opensearch.node.Node.<init>(Node.java:429)
        at org.opensearch.node.Node.<init>(Node.java:356)
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.Command.main(Command.java:101)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)

openssl pkcs12 -info -in certs/os01/os01.p12 -nokeys
Enter Import Password:
MAC: sha256, Iteration 2048
MAC length: 32, salt length: 8
PKCS7 Encrypted data: PBES2, PBKDF2, AES-256-CBC, Iteration 2048, PRF hmacWithSHA256
Certificate bag
Bag Attributes
    friendlyName: os01
    localKeyID: 37 4D 1B 70 EB D3 26 E3 23 2B 42 7D 93 C1 3A B3 A9 F2 75 A1 
subject=C = BR, ST = Sao Paulo, L = Sao Paulo, O = tsl, CN = os01
issuer=C = BR, ST = Sao Paulo, L = Sao Paulo, O = tsl, CN = CA
-----BEGIN CERTIFICATE-----
MIIDlDCCAnygAwIBAgIUWC6+vSiUoaqZno06O1oreH+7q6EwDQYJKoZIhvcNAQEL
BQAwUDELMAkGA1UEBhMCQlIxEjAQBgNVBAgMCVNhbyBQYXVsbzESMBAGA1UEBwwJ
U2FvIFBhdWxvMQwwCgYDVQQKDAN0c2wxCzAJBgNVBAMMAkNBMB4XDTIzMTIyOTIw
MDk1NloXDTI0MDEyODIwMDk1NlowUjELMAkGA1UEBhMCQlIxEjAQBgNVBAgMCVNh
byBQYXVsbzESMBAGA1UEBwwJU2FvIFBhdWxvMQwwCgYDVQQKDAN0c2wxDTALBgNV
BAMMBG9zMDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDi3YLhehli
K3uewjMyEaHbFNajTYmQWhQ+Hj3ca9BC+vcxiBaWbS12+h1AtPureF+KoZ07vGXH
T7WilRRxQCEf2RyNpJPGYh5T12tTBpXTCrPBLCOCGV/37bwQdXnaXjzwYgi79uk7
9AMDXuy4mnMHmzkpRkVxt4Vkicu20eK6TLwYTzhikWL7TSMF35nVEf+krezzQZYu
Eqg7Wx99aRIYD42A03Wp8pf0FBPPwieCN7jaFQ3XqlAbm8SvNUIy6uL4z9q8PI/T
AHY/1yGXH8oXOn0cqUY6Aaug1LYfgCaLE9HmoJMohlSgatQOyBopEqHU+O5k2i27
+Im+tkJXSWinAgMBAAGjZDBiMCAGA1UdEQQZMBeCCWxvY2FsaG9zdIcEfwAAAYIE
b3MwMTAdBgNVHQ4EFgQU4Lqr4n4bkMiE6IrZGiI472/yY/MwHwYDVR0jBBgwFoAU
X34RGiMdSW4POaQR1xcq0i+gqEIwDQYJKoZIhvcNAQELBQADggEBAC/4M8pFQnE/
KQUrrCi+ITg09YkDs3XQTOWQ+d5ZlwrY0wtTqo0dZaAFkJkXV5i9UB//k1+hV7TE
UWpDZO+cMxdTPsdOYxpr28+K/cbjyWqKx2eLdpBIruDCFtC/eX5HL8R48XdaxG4B
v+c7hOXkreGqmKHEtBzLWSDGr6EGxge0PLhSkNSfTHJGGA9m2S8y3qXd4C7AWDh1
40604MYQiJuWn4XhbRANB266SN86lQuATXSWp1OMevp/QPAt/LbVBIbJTzWft6Yi
e8Im0qtONNEpM0vx9AISPZ5OX4tkCYSzweOJQJ2I5YdI+R5Tp3QMgERiXb4Hcibs
9EKwEmCpEnU=
-----END CERTIFICATE-----
PKCS7 Data
Shrouded Keybag: PBES2, PBKDF2, AES-256-CBC, Iteration 2048, PRF hmacWithSHA256

ls -l certs/os01/os01.p12
-rw-r–r-- 1 deborah deborah 2716 Jan 5 01:21 certs/os01/os01.p12

and I gave permissions: sudo chmod 644 certs/os01/os01.p12

I have been in this for more than 3 weeks, have mercy :wink:

3

Hi @Deborah ,

have you tried setting plugins.security.ssl.transport.keystore_type to PKCS12/PFX:

plugins.security.ssl.transport.keystore_type: "PKCS12/PFX"

for more details:

best,
mj

4

Hi @Mantas

I had switched from PEM using:


sudo chmod 755 certs/os01


openssl pkcs12 -export -in certs/os01/os01.pem -inkey certs/os01/os01.key -name os01 -CAfile certs/ca/ca.pem -caname root -out certs/os01/os01.p12

sudo chmod 644 certs/os01/os01.p12

Received the error mentioned above.

After following your suggestion to opensearch.yml

I get:

uncaught exception in thread [main]
os01_1  | java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
os01_1  | Likely root cause: OpenSearchException[Unable to read /usr/share/opensearch/config/certs/os01/os01.p12 (/usr/share/opensearch/config/certs/os01/os01.p12). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.keystore_filepath]
os01_1  |       at org.opensearch.security.ssl.DefaultSecurityKeyStore.checkPath(DefaultSecurityKeyStore.java:983)
os01_1  |       at org.opensearch.security.ssl.DefaultSecurityKeyStore.resolve(DefaultSecurityKeyStore.java:235)
os01_1  |       at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:284)
os01_1  |       at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:256)
os01_1  |       at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:177)
os01_1  |       at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218)
os01_1  |       at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:263)
os01_1  |       at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
os01_1  |       at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
os01_1  |       at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
os01_1  |       at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
os01_1  |       at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
os01_1  |       at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:782)
os01_1  |       at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731)
os01_1  |       at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533)
os01_1  |       at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195)
os01_1  |       at org.opensearch.node.Node.<init>(Node.java:429)
os01_1  |       at org.opensearch.node.Node.<init>(Node.java:356)
os01_1  |       at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
os01_1  |       at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
os01_1  |       at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
os01_1  |       at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
os01_1  |       at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
os01_1  |       at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
os01_1  |       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
os01_1  |       at org.opensearch.cli.Command.main(Command.java:101)
os01_1  |       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
os01_1  |       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
os01_1  | For complete error details, refer to the log at /usr/share/opensearch/logs/os-cluster.log

thank you!

5

Could please run chown opensearch:opensearch /usr/share/opensearch/config/certs/os01/os01.p12 and share ls -l /usr/share/opensearch/config/certs/os01/ output?

Thanks,
mj

1 Like
6

holly cow it’s empty!

deborah@webserver-magento-poc:~$ ls -l /usr/share/opensearch/certs
total 0

I’ll fix it and let you know.

7

The way I was generating certificates was not healthy. The values for SAN (MYDN) where not being captured correctly. So I enhanced a bit the script given on the documentation and generated new PEM files. Using PKCS12 or JKS was not the solution as passwording was a bit more complicated for a simple docker installation. So to solve the installation with PEM I first abstracted from the blog: (howtoforge), used the documentation to generate certs. Than, I used opensearch.yml files (one per node) to pass values for certs, not just root-ca. I used the sample yml file given in the documentation ALTERING the hostnames so they were aligned with certs SAN. I opened more ports, according to the docs (were missing 9300 and 9250). Included an environment to deny demo certs on docker-compose.yml. Finally adjusted permission: 1000:1000 (opensearch on docker) for the file path with 750 for directory and 644 for all files. So concentrating on the official docs rather than on blogs, reading the forum with example files and using gpt4 to “translate” the verbiage was the way to go. :star_struck: