PKIX path building failed when trying to connect Opensearch to minio with https

OpenSearch
, ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): OS version 3.3.2, dashboard: 3.3.0

Describe the issue: While trying to connect minio (on https), following error occurs: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Configuration: OS installed using helm with customized values.yml, tried to add the ca cert of minio (i.e. ca crt of k3s, which is the issuer of minio’s cert) in OS’s truststore using command
keytool -importcert
-alias k3s-ca
-file /tmp/k3s-ca.crt
-keystore /usr/share/opensearch/config/truststore.jks
-storepass changeit
-noprompt
Certificate was added to keystore`

Did this in all nodes, then did helm upgrade but OS is still not recognizing the minio’s cert…

Relevant Logs or Screenshots:
When trying to run command on console:
PUT _snapshot/minio-repo
{
“type”: “s3”,
“settings”: {
“bucket”: “opensearch-snapshots”,
“base_path”: “snapshots”,
“client”: “default”,
“endpoint”: “``https://soc-minio-pool-0-0.soc-minio-hl.minio-tenant.svc.cluster.local:9000``”,
“protocol”: “https”,
“path_style_access”: true,
“compress”: true,
“region”: “us-east-1”
}
}
Error is:
{
“error”: {
“root_cause”: [
{
“type”: “repository_verification_exception”,
“reason”: “[minio-repo] path [snapshots] is not accessible on cluster-manager node”
}
],
“type”: “repository_verification_exception”,
“reason”: “[minio-repo] path [snapshots] is not accessible on cluster-manager node”,
“caused_by”: {
“type”: “i_o_exception”,
“reason”: “Unable to upload object [snapshots/tests-lZUX43fUSM2K8Wjl8hKs8g/master.dat] using a single upload”,
“caused_by”: {
“type”: “sdk_client_exception”,
“reason”: “sdk_client_exception: Unable to execute HTTP request: (certificate_unknown) PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (SDK Attempt Count: 4)”,
“caused_by”: {
“type”: “i_o_exception”,
“reason”: “(certificate_unknown) PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”,
“caused_by”: {
“type”: “validator_exception”,
“reason”: “validator_exception: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”,
“caused_by”: {
“type”: “sun_cert_path_builder_exception”,
“reason”: “sun_cert_path_builder_exception: unable to find valid certification path to requested target”
}
}
},
“suppressed”: [
{
“type”: “sdk_client_exception”,
“reason”: “sdk_client_exception: Request attempt 1 failure: Unable to execute HTTP request: (certificate_unknown) PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”
},
{
“type”: “sdk_client_exception”,
“reason”: “sdk_client_exception: Request attempt 2 failure: Unable to execute HTTP request: (certificate_unknown) PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”
},
{
“type”: “sdk_client_exception”,
“reason”: “sdk_client_exception: Request attempt 3 failure: Unable to execute HTTP request: (certificate_unknown) PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”
}
]
}
}
},
“status”: 500
}

2

@yashg Have you tried adding Minio SSL certificate to Java’s keystore?

/usr/share/opensearch/jdk/lib/security/cacerts

keytool -import -noprompt -trustcacerts -alias minio -file <minio.crt> -keystore /usr/share/opensearch/jdk/lib/security/cacerts -storepass changeit
3

Hi @pablo, the java’s keystore is by-default read only, so I tried making a custom-truststore, adding minio’s ca cert there and then customizing the following field in helm’s values.yml:
opensearchJavaOpts: “-Xmx2048M -Xms2048M -Djavax.net.ssl.trustStore=/truststore/custom-truststore.jks -Djavax.net.ssl.trustStorePassword=changeit -Dcom.amazonaws.sdk.disableCertChecking=false”

But it didn’t work.
Still got the same error…

4

How do you deploy your cluster?

5

I’ve got it from your first message. So K3 with Official OS charts.

6

Using helm, as per the docs instructions.

My hot-values.yml is:

clusterName: “opensearch-cluster”
nodeGroup: “hot”
masterService: “opensearch-cluster-hot”
replicas: 3
roles:

cluster_manager

ingest

data

opensearchJavaOpts: "-Xmx2048M -Xms2048M "

resources:
requests:
cpu: “1000m”
memory: “3Gi”
limits:
cpu: “1000m”
memory: “3Gi”
persistence:
enabled: true
size: “300Gi”
storageClass: “local-path”
config:
opensearch.yml: |
cluster.name: opensearch-cluster
network.host: 0.0.0.0
node.attr.temp: hot
s3.client.default.path_style_access: true
s3.client.default.protocol: https

discovery.seed_hosts:

“opensearch-cluster-hot-0.opensearch.svc.cluster.local:9300”

“opensearch-cluster-hot-1.opensearch.svc.cluster.local:9300”

“opensearch-cluster-hot-2.opensearch.svc.cluster.local:9300”
cluster.initial_cluster_manager_nodes:

“opensearch-cluster-hot-0”

“opensearch-cluster-hot-1”

“opensearch-cluster-hot-2”
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.pemcert_filepath: /usr/share/opensearch/config/certs/tls.crt
plugins.security.ssl.transport.pemkey_filepath: /usr/share/opensearch/config/certs/tls.key
plugins.security.ssl.transport.pemtrustedcas_filepath: /usr/share/opensearch/config/certs/ca.crt
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /usr/share/opensearch/config/certs/tls.crt
plugins.security.ssl.http.pemkey_filepath: /usr/share/opensearch/config/certs/tls.key
plugins.security.ssl.http.pemtrustedcas_filepath: /usr/share/opensearch/config/certs/ca.crt
plugins.security.allow_default_init_securityindex: true
plugins.security.nodes_dn:

“CN=cluster-hot-0,OU=Node,O=OpenSearch-System,L=City,ST=State,C=IN”
tls:
enabled: true
generateNodeCerts: false
usePemCerts: true
pemCerts:
secret:
name: opensearch-certs
caFilename: “ca.crt”
certFilename: “tls.crt”
keyFilename: “tls.key”
plugins:
enabled: true
installList:

repository-s3
extraEnvs:

name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: opensearch-secrets
key: OPENSEARCH_INITIAL_ADMIN_PASSWORD

keystore:

secretName: opensearch-s3-secrets

protocol: https
httpPort: 9200
transportPort: 9300
securityConfig:
enabled: true
config:
securityConfigSecret: “”
dataComplete: true

extraVolumes:

name: opensearch-tls
secret:
secretName: opensearch-certs

extraVolumeMounts:

name: opensearch-tls
mountPath: /usr/share/opensearch/config/certs
readOnly: true

Similar is the warm-values.yml with the changes in nodeGroup, replicas, and roles

Please ignore the indentations here.

7

@yashg You could try copying /usr/share/opensearch/jdk/lib/security/cacerts from the OpenSearch binary and adding the cert with the mentioned keytool command.

https://artifacts.opensearch.org/releases/bundle/opensearch/3.3.2/opensearch-3.3.2-linux-x64.tar.gz

Then map it with extraVolume.

8

Thanks a lot, this seems to be working…

The thing I did is copy keystore to /tmp folder, add the cert, copy it back.
And with the new cert, created secret and create volumeMount from that.
extraVolumes:

  • name: custom-cacerts
    secret:
    secretName: opensearch-custom-cacerts

extraVolumeMounts:

  • name: custom-cacerts
    mountPath: /usr/share/opensearch/jdk/lib/security/cacerts
    subPath: cacerts
9

The Only Issue I can see is the slow speed of the connection/process, which was almost instant when I connected Docker-contained OS with minio without HTTPS.
Can I do something for this?
Would increasing RAM to OS or to JVM work?

Like when I do this:
PUT _snapshot/minio-repo/snapshot-logs-check-2023-11-26?wait_for_completion=true
{
“indices”: “*logs-check*”,
“ignore_unavailable”: true,
“include_global_state”: false
}

Given that the datastream is only 2kb in size.
It keeps running, snapshot is created at minio, but the error shows here at console:
{
“error”: {
“root_cause”: [
{
“type”: “invalid_snapshot_name_exception”,
“reason”: “[minio-repo:snapshot-logs-2023-11-26] Invalid snapshot name [snapshot-logs-2023-11-26], snapshot with the same name is already in-progress”
}
],
“type”: “invalid_snapshot_name_exception”,
“reason”: “[minio-repo:snapshot-logs-2023-11-26] Invalid snapshot name [snapshot-logs-2023-11-26], snapshot with the same name is already in-progress”
},
“status”: 400
}

10

@yashg I had no issues with creating snapshots to Minio. I haven’t used HTTPS, and this shouldn’t be the case here.
This is my working configuration for Minio and official OpenSearch Helm charts.

config:
  # Values must be YAML literal style scalar / YAML multiline string.
  # <filename>: |
  #   <formatted-value(s)>
  # log4j2.properties: |
  #   status = error
  #
  #   appender.console.type = Console
  #   appender.console.name = console
  #   appender.console.layout.type = PatternLayout
  #   appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
  #
  #   rootLogger.level = info
  #   rootLogger.appenderRef.console.ref = console
  opensearch.yml: |
    cluster.name: opensearch-cluster
    # Bind to all interfaces because we don't know what IP address Docker will assign to us.
    network.host: 0.0.0.0
    s3.client.default.endpoint: dockerhub.pablo.local:9000
    s3.client.default.protocol: http
    s3.client.default.region: eu-west-1
    s3.client.default.path_style_access: true
keystore:
# To add secrets to the keystore:
#  - secretName: opensearch-encryption-key
  - secretName: aws-credentials
## Enable to add 3rd Party / Custom plugins not offered in the default OpenSearch image.
plugins:
  enabled: true
  installList:
  - repository-s3

extraEnvs: #if the storage endpoint is custom S3 i.e. MinIO
  - name: AWS_EC2_METADATA_DISABLED
    value: "true"