PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.13

Describe the issue:
I am trying to configure S3data Source and i am getting below error when the job is automatically submitted to EMR serveless.
I am using the demo configuration

24/04/23 01:50:38 ERROR FlintREPL: Session error: Failed to execute update request on index: .query_execution_request_opss3test, id: ZVFhZ0JRVmpob09QU1MzVGVzdA==
java.lang.RuntimeException: Failed to execute update request on index: .query_execution_request_opss3test, id: ZVFhZ0JRVmpob09QU1MzVGVzdA==
at org.opensearch.flint.core.storage.OpenSearchUpdater.updateDocument(OpenSearchUpdater.java:80) ~[org.opensearch_opensearch-spark-standalone_2.12-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at org.opensearch.flint.core.storage.OpenSearchUpdater.upsert(OpenSearchUpdater.java:35) ~[org.opensearch_opensearch-spark-standalone_2.12-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at org.apache.spark.sql.FlintREPL$.setupFlintJob(FlintREPL.scala:433) [org.opensearch_opensearch-spark-sql-application_2.12-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at org.apache.spark.sql.FlintREPL$.setupFlintJobWithExclusionCheck(FlintREPL.scala:307) [org.opensearch_opensearch-spark-sql-application_2.12-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at org.apache.spark.sql.FlintREPL$.main(FlintREPL.scala:176) [org.opensearch_opensearch-spark-sql-application_2.12-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at org.apache.spark.sql.FlintREPL.main(FlintREPL.scala) [org.opensearch_opensearch-spark-sql-application_2.12-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) [spark-core_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1075) [spark-core_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194) [spark-core_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217) [spark-core_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) [spark-core_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1167) [spark-core_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1176) [spark-core_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) [spark-core_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.

Configuration:

Relevant Logs or Screenshots:

2

Hey @pravinrc

That might be your issue, have you tried not using the demo certifications and create you own?

3

I am not sure how java trust store is used by opensearch, but i have a small class file SSLPoke which i have downloaded from internet, i have checked the class by passing truststore paramater, wrong file name produce the same issue i am having, correct cacerts location have no issue.

Now i have updated /etc/opensearch/jvm.options with -Djavax.net.ssl.trustStore=/usr/share/opensearch/jdk/lib/security/cacerts

and restared opensearch but i am still having the same issue,not sure if trustfile is overridden by some other location.
Do we know which file opensearch uses ?

root@awuavopsa01 rpm]# /usr/share/opensearch/jdk/bin/java -Djavax.net.ssl.trustStore=/usr/share/opensearch/jdk/lib/security/cacerts.pc SSLPoke X.X.X.X 9200
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:388)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:271)
at java.base/sun.security.validator.Validator.validate(Validator.java:256)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:230)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1302)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:447)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:201)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:922)
at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1291)
at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1263)
at SSLPoke.main(SSLPoke.java:31)
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:383)
… 19 more
[root@awuavopsa01 rpm]# /usr/share/opensearch/jdk/bin/java -Djavax.net.ssl.trustStore=/usr/share/opensearch/jdk/lib/security/cacerts SSLPoke x.x.x.x 9200
Successfully connected
[root@awuavopsa01 rpm]#

4

I may try to regenerate self signed certificate but not sure it will resolve the issue unless we know the correct location.

5

Hi @pravinrc

Could you share your opensearch.yml file?

6

######## Start OpenSearch Security Demo Configuration ########

WARNING: revise all the lines below before you go into production

network.host: 10.0.100.107
discovery.type: single-node
plugins.security.disabled: false
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn: [‘CN=kirk,OU=client,O=client,L=test,C=de’]
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [all_access, security_rest_api_access]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [.plugins-ml-agent, .plugins-ml-config, .plugins-ml-connector,
.plugins-ml-controller, .plugins-ml-model-group, .plugins-ml-model, .plugins-ml-task,
.plugins-ml-conversation-meta, .plugins-ml-conversation-interactions, .plugins-ml-memory-meta,
.plugins-ml-memory-message, .plugins-ml-stop-words, .opendistro-alerting-config,
.opendistro-alerting-alert*, .opendistro-anomaly-results*, .opendistro-anomaly-detector*,
.opendistro-anomaly-checkpoints, .opendistro-anomaly-detection-state, .opendistro-reports-,
.opensearch-notifications-, .opensearch-notebooks, .opensearch-observability, .ql-datasources,
.opendistro-asynchronous-search-response*, .replication-metadata-store, .opensearch-knn-models,
.geospatial-ip2geo-data*, .plugins-flow-framework-config, .plugins-flow-framework-templates,
.plugins-flow-framework-state]
node.max_local_storage_nodes: 3
plugins.query.datasources.encryption.masterkey: b0ae8ea6e048f9502027f56d
Plugins.query.executionengine.spark.config: ‘{"applicationId":"00fikl658a6mml29","executionRoleARN":"arn:aws:iam::xxxxx:role/EMRServerlessS3RuntimeRoleOPS","region":"ap-southeast-2a", "sparkSubmitParameters": "–conf spark.dynamicAllocation.enabled=false"}’
plugins.query.executionengine.spark.config: ‘{“applicationId”:“00fikl658a6mml29”,“executionRoleARN”:“arn:aws:iam::xxxxx:role/EMRServerlessS3RuntimeRoleOPS”,“region”:“ap-southeast-2”,“sparkSubmitParameters”:“–conf spark.dynamicAllocation.enabled=false”}’
Plugins.query.executionengine.spark.config: ‘{“applicationId”:“00fikl658a6mml29”,“executionRoleARN”:“arn:aws:iam::xxxxx:role/EMRServerlessS3RuntimeRoleOPS”,“region”:“”,“sparkSubmitParameters”:“–confspark.dynamicAllocation.enabled=false”}’
######## End OpenSearch Security Demo Configuration ########

7

looks like the issue is in EMR Serverless, possibly i will build custom image for EMR serverless with same java version which ultimately have the same cacerts, i think that should solve the problem.

Will appreciate if any other solutions please.