I’m trying to bring up the high level REST client, as documented in the closed issue ‘Open Distro Java High Level REST Client #64’, on a Windows 10 development machine using localhost.
I’m getting a javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
The above post provided a solution by forum user aetter as: keytool -importcert -file client-certificate.pem -keystore keystore.jks -alias “test”, but without the detail on generating the client-certificate.pem.
For the purposes of testing I used the demo certs and performed the following:
Construted a jks keystore via:
keytool -keystore client.jks -storepass changeit -keypass changeit -genkey -alias kms -validity 365 -dname /
“CN=resthost, O=acme.net, L=NY, S=NY, C=US, OU=IS” /
-ext “SAN=DNS:localhost,IP:127.0.0.1” -ext “ExtendedKeyUsage=serverAuth,clientAuth”
In reseaching the issue, I found this post, Client / node certificate - #2 by manishatanwar, indicating that the yml file should include the following entry:
opendistro_security.nodes_dn, which I entered as:
opendistro_security.nodes_dn:
- CN=resthost,O=acme.net, L=NY, S=NY, C=US, OU=IS"
Note that I have tried the steps without this entry with the same error result.
My initial attempt was to generate the keystore with a root certificate, with the generated CSR request that included the SAN entries. However, the SAN entries did not carry over due to the root being self signed.
My understanding is that by adding a self signed cert (with no signing root) to the truststore was sufficient, and was the only approach I could determine to include the SAN and ExtendedKeyUsage settings in the keystore.
I exported the cert from the keystore via:
keytool -export -alias kms -keystore client.jks -rfc -file client.cert
and imported it into the cacerts store in both the embedded JDK provided via the opendistro download and my java home JDK (the only two JDK/JREs on the system):
keytool -import -alias kms -keystore “C:\Program Files\Java\jdk11.0.4_10\lib\security\cacerts” -file client.cert
keytool -import -alias kms -keystore “N:\opendistro\jdk\lib\security\cacerts” -file client.cert
Per the code provided in the #64 post, the RestClientBuilder and RestHighLevelClient constructors execute cleanly, with the first access attempt the above exception is thrown on
boolean exists = getRestClient().indices().exists(request, RequestOptions.DEFAULT);
Thanks in advance for any assistance. When resolved, I will update with the correct scripts and settings.
This is the client cert content via: keytool -printcert -v -file client.cert
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 1 entry
Alias name: kms
Creation date: Aug 5, 2020
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=resthost, O=acme.net, L=NY, ST=NY, C=US, OU=IS
Issuer: CN=resthost, O=acme.net, L=NY, ST=NY, C=US, OU=IS
Serial number: 114967f
Valid from: Wed Aug 05 09:29:47 CDT 2020 until: Thu Aug 05 09:29:47 CDT 2021
Certificate fingerprints:
SHA1: 8C:BF:EE:41:FE:E5:A0:96:DF:12:E4:1E:10:2B:41:4F:67:4D:69:04
SHA256: 6C:27:D2:6F:F3:8A:CA:53:E2:BA:14:D9:96:36:B4:FA:C9:EF:A9:3C:14:1D:B6:1E:C4:1F:94:CD:79:DE:F7:BB
Signature algorithm name: SHA256withDSA
Subject Public Key Algorithm: 2048-bit DSA key
Version: 3
Extensions:
#1: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
serverAuth
clientAuth
]
#2: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: localhost
IPAddress: 127.0.0.1
]
#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: BE 1D 48 72 56 8E 42 CB 53 F7 34 87 BB E2 BC BE …HrV.B.S.4…
0010: 93 EC 17 44 …D
]
]
This is the startup log entries:
N:\opendistro>.\bin\elasticsearch.bat
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2020-08-05T11:10:45,831][INFO ][o.e.n.Node ] [KMS] version[7.8.0], pid[14720], build[oss/zip/757314695644ea9a1dc2fecd26d1a43856725e65/2020-06-14T19:35:50.234439Z], OS[Windows 10/10.0/amd64], JVM[Amazon.com Inc./OpenJDK 64-Bit Server VM/11.0.4/11.0.4+11-LTS]
[2020-08-05T11:10:45,837][INFO ][o.e.n.Node ] [KMS] JVM home [C:\Program Files\Java\jdk11.0.4_10]
[2020-08-05T11:10:45,840][INFO ][o.e.n.Node ] [KMS] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=C:\Users\micha\AppData\Local\Temp\elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Delasticsearch, -Des.path.home=N:\opendistro, -Des.path.conf=N:\opendistro\config, -Des.distribution.flavor=oss, -Des.distribution.type=zip, -Des.bundled_jdk=true]
[2020-08-05T11:10:47,201][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [KMS] ES Config path is N:\opendistro\config
[2020-08-05T11:10:47,321][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [KMS] JVM supports TLSv1.3
[2020-08-05T11:10:47,324][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [KMS] Config directory is N:\opendistro\config/, from there the key- and truststore files are resolved relatively
[2020-08-05T11:10:47,923][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [KMS] TLS Transport Client Provider : JDK
[2020-08-05T11:10:47,924][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [KMS] TLS Transport Server Provider : JDK
[2020-08-05T11:10:47,925][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [KMS] TLS HTTP Provider : JDK
[2020-08-05T11:10:47,934][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [KMS] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2, TLSv1.1]
[2020-08-05T11:10:47,937][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [KMS] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2, TLSv1.1]
[2020-08-05T11:10:48,192][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [KMS] Clustername: elasticsearch
[2020-08-05T11:10:48,352][INFO ][c.a.o.j.JobSchedulerPlugin] [KMS] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2020-08-05T11:10:48,526][INFO ][c.a.o.j.JobSchedulerPlugin] [KMS] Loaded scheduler extension: opendistro-managed-index, index: .opendistro-ism-config
[2020-08-05T11:10:48,590][INFO ][o.e.p.PluginsService ] [KMS] loaded module [aggs-matrix-stats]
[2020-08-05T11:10:48,591][INFO ][o.e.p.PluginsService ] [KMS] loaded module [analysis-common]
[2020-08-05T11:10:48,592][INFO ][o.e.p.PluginsService ] [KMS] loaded module [geo]
[2020-08-05T11:10:48,599][INFO ][o.e.p.PluginsService ] [KMS] loaded module [ingest-common]
[2020-08-05T11:10:48,600][INFO ][o.e.p.PluginsService ] [KMS] loaded module [ingest-geoip]
[2020-08-05T11:10:48,600][INFO ][o.e.p.PluginsService ] [KMS] loaded module [ingest-user-agent]
[2020-08-05T11:10:48,601][INFO ][o.e.p.PluginsService ] [KMS] loaded module [kibana]
[2020-08-05T11:10:48,603][INFO ][o.e.p.PluginsService ] [KMS] loaded module [lang-expression]
[2020-08-05T11:10:48,603][INFO ][o.e.p.PluginsService ] [KMS] loaded module [lang-mustache]
[2020-08-05T11:10:48,611][INFO ][o.e.p.PluginsService ] [KMS] loaded module [lang-painless]
[2020-08-05T11:10:48,615][INFO ][o.e.p.PluginsService ] [KMS] loaded module [mapper-extras]
[2020-08-05T11:10:48,624][INFO ][o.e.p.PluginsService ] [KMS] loaded module [parent-join]
[2020-08-05T11:10:48,630][INFO ][o.e.p.PluginsService ] [KMS] loaded module [percolator]
[2020-08-05T11:10:48,639][INFO ][o.e.p.PluginsService ] [KMS] loaded module [rank-eval]
[2020-08-05T11:10:48,644][INFO ][o.e.p.PluginsService ] [KMS] loaded module [reindex]
[2020-08-05T11:10:48,654][INFO ][o.e.p.PluginsService ] [KMS] loaded module [repository-url]
[2020-08-05T11:10:48,659][INFO ][o.e.p.PluginsService ] [KMS] loaded module [tasks]
[2020-08-05T11:10:48,667][INFO ][o.e.p.PluginsService ] [KMS] loaded module [transport-netty4]
[2020-08-05T11:10:48,674][INFO ][o.e.p.PluginsService ] [KMS] loaded plugin [opendistro-anomaly-detection]
[2020-08-05T11:10:48,676][INFO ][o.e.p.PluginsService ] [KMS] loaded plugin [opendistro-job-scheduler]
[2020-08-05T11:10:48,684][INFO ][o.e.p.PluginsService ] [KMS] loaded plugin [opendistro_alerting]
[2020-08-05T11:10:48,690][INFO ][o.e.p.PluginsService ] [KMS] loaded plugin [opendistro_index_management]
[2020-08-05T11:10:48,692][INFO ][o.e.p.PluginsService ] [KMS] loaded plugin [opendistro_security]
[2020-08-05T11:10:48,699][INFO ][o.e.p.PluginsService ] [KMS] loaded plugin [opendistro_sql]
[2020-08-05T11:10:48,734][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [KMS] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting ‘http.compression: true’ in elasticsearch.yml
[2020-08-05T11:10:48,757][INFO ][o.e.e.NodeEnvironment ] [KMS] using [1] data paths, mounts [[NCS (N:)]], net usable_space [111gb], net total_space [119.2gb], types [NTFS]
[2020-08-05T11:10:48,759][INFO ][o.e.e.NodeEnvironment ] [KMS] heap size [990.7mb], compressed ordinary object pointers [true]
[2020-08-05T11:10:48,888][INFO ][o.e.n.Node ] [KMS] node name [KMS], node ID [qO6D-zkDRkKcQ0tNDbu-jA], cluster name [elasticsearch]
[2020-08-05T11:10:52,364][WARN ][c.a.o.s.c.Salt ] [KMS] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2020-08-05T11:10:52,397][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Auditing on REST API is enabled.
[2020-08-05T11:10:52,399][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] [GRANTED_PRIVILEGES, AUTHENTICATED] are excluded from REST API auditing.
[2020-08-05T11:10:52,400][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Auditing on Transport API is enabled.
[2020-08-05T11:10:52,406][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] [GRANTED_PRIVILEGES, AUTHENTICATED] are excluded from Transport API auditing.
[2020-08-05T11:10:52,412][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Auditing of request body is enabled.
[2020-08-05T11:10:52,413][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Bulk requests resolution is disabled during request auditing.
[2020-08-05T11:10:52,420][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Index resolution is enabled during request auditing.
[2020-08-05T11:10:52,421][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Sensitive headers auditing is enabled.
[2020-08-05T11:10:52,423][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Auditing requests from kibanaserver users is disabled.
[2020-08-05T11:10:52,503][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Auditing of external configuration is disabled.
[2020-08-05T11:10:52,506][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Auditing of internal configuration is disabled.
[2020-08-05T11:10:52,509][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Auditing only metadata information for read request is disabled.
[2020-08-05T11:10:52,515][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Auditing will watch {} for read requests.
[2020-08-05T11:10:52,528][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Auditing read operation requests from kibanaserver users is disabled.
[2020-08-05T11:10:52,531][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Auditing only metadata information for write request is disabled.
[2020-08-05T11:10:52,558][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Auditing diffs for write requests is disabled.
[2020-08-05T11:10:52,561][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Auditing write operation requests from kibanaserver users is disabled.
[2020-08-05T11:10:52,568][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Auditing will watch for write requests.
[2020-08-05T11:10:52,572][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] .opendistro_security is used as internal security index.
[2020-08-05T11:10:52,573][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Internal index used for posting audit logs is null
[2020-08-05T11:10:52,796][INFO ][c.a.o.s.a.i.AuditLogImpl ] [KMS] Message routing enabled: true
[2020-08-05T11:10:53,039][INFO ][c.a.o.s.f.OpenDistroSecurityFilter] [KMS] indices are made immutable.
[2020-08-05T11:10:53,117][INFO ][c.a.o.a.b.ADCircuitBreakerService] [KMS] Registered memory breaker.
[2020-08-05T11:10:53,485][INFO ][o.e.d.DiscoveryModule ] [KMS] using discovery type [zen] and seed hosts providers [settings]
[2020-08-05T11:10:54,129][INFO ][o.e.n.Node ] [KMS] initialized
[2020-08-05T11:10:54,129][INFO ][o.e.n.Node ] [KMS] starting …
[2020-08-05T11:10:54,830][INFO ][o.e.t.TransportService ] [KMS] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2020-08-05T11:10:55,040][WARN ][o.e.b.BootstrapChecks ] [KMS] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2020-08-05T11:10:55,042][INFO ][o.e.c.c.Coordinator ] [KMS] cluster UUID [nz6KOHJHS9OWzch_jy1h4w]
[2020-08-05T11:10:55,056][INFO ][o.e.c.c.ClusterBootstrapService] [KMS] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2020-08-05T11:10:55,210][INFO ][o.e.c.s.MasterService ] [KMS] elected-as-master ([1] nodes joined)[{KMS}{qO6D-zkDRkKcQ0tNDbu-jA}{NaZA-t7EToWt7U-7gWFQ4A}{127.0.0.1}{127.0.0.1:9300}{dimr} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 5, version: 24, delta: master node changed {previous , current [{KMS}{qO6D-zkDRkKcQ0tNDbu-jA}{NaZA-t7EToWt7U-7gWFQ4A}{127.0.0.1}{127.0.0.1:9300}{dimr}]}
[2020-08-05T11:10:55,287][INFO ][o.e.c.s.ClusterApplierService] [KMS] master node changed {previous , current [{KMS}{qO6D-zkDRkKcQ0tNDbu-jA}{NaZA-t7EToWt7U-7gWFQ4A}{127.0.0.1}{127.0.0.1:9300}{dimr}]}, term: 5, version: 24, reason: Publication{term=5, version=24}
[2020-08-05T11:10:55,299][INFO ][c.a.o.a.c.ADClusterEventListener] [KMS] CLuster is not recovered yet.
[2020-08-05T11:10:55,395][INFO ][o.e.g.GatewayService ] [KMS] recovered [1] indices into cluster_state
[2020-08-05T11:10:55,790][INFO ][o.e.h.AbstractHttpServerTransport] [KMS] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2020-08-05T11:10:55,792][INFO ][o.e.n.Node ] [KMS] started
[2020-08-05T11:10:55,798][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [KMS] Node started
[2020-08-05T11:10:55,799][INFO ][c.a.o.s.c.ConfigurationRepository] [KMS] Will attempt to create index .opendistro_security and default configs if they are absent
[2020-08-05T11:10:55,800][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [KMS] 4 Open Distro Security modules loaded so far: [Module [type=UNKNOWN, implementing class=com.amazon.opendistroforelasticsearch.security.compliance.ComplianceIndexingOperationListenerImpl], Module [type=AUDITLOG, implementing class=com.amazon.opendistroforelasticsearch.security.auditlog.impl.AuditLogImpl], Module [type=MULTITENANCY, implementing class=com.amazon.opendistroforelasticsearch.security.configuration.PrivilegesInterceptorImpl], Module [type=REST_MANAGEMENT_API, implementing class=com.amazon.opendistroforelasticsearch.security.dlic.rest.api.OpenDistroSecurityRestApiActions]]
[2020-08-05T11:10:55,800][INFO ][c.a.o.s.c.ConfigurationRepository] [KMS] Background init thread started. Install default config?: true
[2020-08-05T11:10:55,839][INFO ][c.a.o.s.c.ConfigurationRepository] [KMS] Index .opendistro_security already exists
[2020-08-05T11:10:55,998][INFO ][o.e.c.r.a.AllocationService] [KMS] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.opendistro_security][0]]]).
[2020-08-05T11:10:56,329][INFO ][stdout ] [KMS] [FINE] No subscribers registered for event class com.amazon.opendistroforelasticsearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl
[2020-08-05T11:10:56,331][INFO ][stdout ] [KMS] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent
[2020-08-05T11:10:56,331][INFO ][c.a.o.s.c.ConfigurationRepository] [KMS] Node ‘KMS’ initialized
[2020-08-05T11:15:55,050][INFO ][c.a.o.j.s.JobSweeper ] [KMS] Running full sweep