Handling multidomain certifiates

Security
1

Versions Opensearch: 2.6.0

Describe the issue:
Ability to install multidomain certificate

Can someone help me to understand what type of certs can be used for this setup?
I have multi-domain cert. That supposed to handle all three nodes. But I am getting handshake error between nodes. Same cert/key combination is installed on the nodes. I am actually using nfs mount for config folder where everything is shared amongst the nodes and it’s very handy. I have used the same approach before. I am using tar distribution.

The way plugins.security.nodes_dn is asked to be configured I don’t quiet understand, because it only would make sense in the case of individual node certs which I find retarded to do. So if I have multidomain cert which should be able to handle things, it’s just a java app after all, how do I define items for plugins.security.nodes_dn

[ERROR][o.o.s.t.SecurityRequestHandler] [nodem01.blah.com-alpha] OpenSearchException[Transport client authentication no longer supported.]
[WARN ][o.o.d.HandshakingTransportAddressConnector] [nodem03.blah.com-alpha] handshake failed for [connectToRemoteMasterNode[10.5.xxx.35:9300]]
org.opensearch.transport.RemoteTransportException: [nodem03.blah.com-alpha][10.5.xxx.35:9300][internal:transport/handshake]
Caused by: org.opensearch.OpenSearchException: Transport client authentication no longer supported.

Configuration:
cluster.name: opensearch-alpha-st1
node.name: ${HOSTNAME}-alpha
node.roles: [cluster_manager]
path.data: ${PWD}/data
path.logs: ${PWD}/logs
path.repo: [“/mnt/opensearch_st1_repo/alpha”]
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: [“10.5.xxx.33”,“10.5.xxx.34”,“10.5.xxx.35”]
cluster.initial_cluster_manager_nodes: [“nodem01.blah.com-alpha”,“nodem02.blah.com-alpha”,“nodem03.blah.com-alpha”]
plugins.security.ssl.transport.pemcert_filepath: opensearch_hosts_cert.pem
plugins.security.ssl.transport.pemkey_filepath: opensearch_hosts_key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: Private_Issuing_CA.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: opensearch_hosts_cert.pem
plugins.security.ssl.http.pemkey_filepath: opensearch_hosts_key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: Private_Issuing_CA.pem
plugins.security.nodes_dn:

  • ‘CN=blahm01.corp.pvt, OU=Enterprise SSL’
  • ‘CN=blahm02.corp.pvt, OU=Enterprise SSL’
  • ‘CN=blahm03.corp.pvt, OU=Enterprise SSL’
    plugins.security.system_indices.enabled: true

Thanks in advance

Relevant Logs or Screenshots:

type or paste code here
2

I think I have found the answer to my own question :smiley:
Should hae read the article:

plugins.security.nodes_dn can be ignored
If your node certificates have an Object ID (OID) identifier in the SAN section, you can omit this configuration.

1 Like
4

Have the following question

plugins.security.ssl.transport.pemcert_filepath: does this contain just end domain cert or end domain and Intermediate cert?
plugins.security.ssl.transport.pemkey_filepath: this only contains the key to end domain cert?
plugins.security.ssl.transport.pemtrustedcas_filepath: this contains the root cert?

I have domain/Intermediate/Root certs

5

@stecino
plugins.security.ssl.transport.pemcert_filepath - contains node + Intermediate (if exists)
plugins.security.ssl.transport.pemkey_filepath - private key of the node certificate
plugins.security.ssl.transport.pemtrustedcas_filepath - RootCA certificate/s - this regards any RootCAs i.e. remote cluster, admin cert, other nodes in the cluster if different

6

@pablo so I append admin cert after RootCA? What about the admin key, do I need to make a reference to it in the config or file?

7

@stecino. You have to add DN of admin certificate to the admin_dn.

https://opensearch.org/docs/latest/security/configuration/tls/#configuring-node-certificates

Then you can use admin cert and key with every API call using curl or other client. Please keep in mind that the admin cert is very powerful and should be used only for security settings management.

If your admin cert was signed by a different RootCA than the nodes then you must add that RootCA to the file defined in the plugins.security.ssl.transport.pemtrustedcas_filepath and plugins.security.ssl.http.pemtrustedcas_filepath.

8

@pablo here is what I have

plugins.security.disabled: false
plugins.security.ssl.transport.pemcert_filepath: combined.pem
plugins.security.ssl.transport.pemkey_filepath: opensearch_hosts_key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: private_root_ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: combined.pem
plugins.security.ssl.http.pemkey_filepath: opensearch_hosts_key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: private_root_ca.pem
plugins.security.authcz.admin_dn:
  - 'CN=opensearchadmin.example.pvt,OU=Enterprise SSL,O=Thryv\, Inc'
plugins.security.nodes_dn:
  - 'CN=opensearch.example.pvt,OU=Enterprise SSL,O=Thryv\, Inc'
  - 'CN=opensearch.example.pvt,OU=Enterprise SSL,O=Thryv\, Inc'
plugins.security.allow_default_init_securityindex: true
plugins.security.system_indices.enabled: true

where

combined.pem is the domain cert with Intermediate cert
private_root_ca.pem is the root cert
I haven’t added the admin cert to the private_root_ca.pem should I

After the restart

I am seeing these on the nodem01.example.pvt-alpha

elected-as-cluster-manager ([1] nodes joined)[{nodem01.example.pvt-alpha}{WDVD-hQ5QkCeWp6T5R73xA}{8iXeNlOOQlufICYMlOdhOg}{10.5.xxx.33}{10.5.xxx.33:9300}{m}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 4, version: 49, delta: cluster-manager node changed {previous [], current [{nodem01.example.pvt-alpha}{WDVD-hQ5QkCeWp6T5R73xA}{8iXeNlOOQlufICYMlOdhOg}{10.5.xxx.33}{10.5.xxx.33:9300}{m}{shard_indexing_pressure_enabled=true}]}

OpenSearchException[Transport client authentication no longer supported

on the nodem02.example.pvt-alpha

handshake failed for [connectToRemoteMasterNode[10.5.xxx.33:9300]]
org.opensearch.transport.RemoteTransportException: [nodem01.example.pvt-alpha][10.5.xxx.33:9300][internal:transport/handshake]
Caused by: org.opensearch.OpenSearchException: Transport client authentication no longer supported.

[nodem02.example.pvt-alpha] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];

cluster-manager not discovered or elected yet, an election requires a node with id [rD5Sy9ogScy_14yoCSdMUQ], have discovered [{nodem02.example.pvt-alpha}{dhj7j-J5Q6uRbhQjcZ8IIw}{gjmQKaOzTDuiw5hFu7MGjQ}{10.5.xxx.34}{10.5.xxx.34:9300}{m}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [10.5.xxx.33:9300, 10.5.xxx.28:9300] from hosts providers and [{nodem02.example.pvt-alpha}{dhj7j-J5Q6uRbhQjcZ8IIw}{gjmQKaOzTDuiw5hFu7MGjQ}{10.5.xxx.34}{10.5.xxx.34:9300}{m}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 3, last-accepted version 39 in term 3
[2023-05-02T15:57:36,085][ERROR][o.o.s.c.ConfigurationRepository] [nodem02.example.pvt-alpha] Cannot apply default config (this is maybe not an error!)

9

These two nodes were bootstrapped and had joined the cluster. Things were working fine prior to the security plugin install/enable

10

Made an adjustment to http protcol

plugins.security.ssl.http.pemcert_filepath: opensearch_admin_cert.pem
plugins.security.ssl.http.pemkey_filepath: opensearch_admin_key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: private_root_ca.pem
11

I also made a curl request to one of the nodes to query API

curl -vk -GET --key opensearch_admin_key.pem --cert opensearch_admin_cert.pem https://nodem02.example.pvt:9200/

I am presented by the admin proper cert but at the end this is the exception I am getting
[WARN ][o.o.h.AbstractHttpServerTransport] [nodem02.example.pvt-alpha] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.5.xxx.34:9200, remoteAddress=/10.5.xxx.55:58738}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.87.Final.jar:4.1.87.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.87.Final.jar:4.1.87.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

12

Ran the securityadmin

Security Admin v7
Will connect to nodem01.example.pvt:9200 … done
ERR: An unexpected SSLHandshakeException occured: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
See Java high-level REST client - OpenSearch documentation for troubleshooting.
Trace:
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
See Java high-level REST client - OpenSearch documentation for troubleshooting.
at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:947)
at org.opensearch.client.RestClient.performRequest(RestClient.java:332)
at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:462)
at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:159)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:288)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:356)

13

@stecino This is how it should look like.

plugins.security.ssl.transport.pemcert_filepath: opensearch_hosts_cert.pem (node cert + Intermediate)
plugins.security.ssl.transport.pemkey_filepath: opensearch_hosts_key.pem (node key)
plugins.security.ssl.transport.pemtrustedcas_filepath: Private_Issuing_CA.pem (node RootCA, admin's RootCA)
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: opensearch_hosts_cert.pem (node cert + Intermediate)
plugins.security.ssl.http.pemkey_filepath: opensearch_hosts_key.pem (node key)
plugins.security.ssl.http.pemtrustedcas_filepath: Private_Issuing_CA.pem  (node RootCA, admin's RootCA)

As per my previous comment, you only configure admin cert’s DN in admin_dn. The plugin.ssl.security settings requires node certificates for both transport and http.

14

@pablo made the changes as you suggested. I have renamed some pem files, but essentially this is what I have. PLEASE NOTE: both my admin and node certs are generated by the same CA, so I only have one cert for root-ca

plugins.security.disabled: false
plugins.security.ssl.transport.pemcert_filepath: combined.pem  <---- (node and intermediate certs)
plugins.security.ssl.transport.pemkey_filepath: opensearch_hosts_key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: private_root_ca.pem <---- (private CA root cert)
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: combined.pem <---- (node and intermediate certs)
plugins.security.ssl.http.pemkey_filepath: opensearch_hosts_key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: private_root_ca.pem <---- (private CA root cert)
plugins.security.authcz.admin_dn:
  - 'CN=opensearchadmin.exmaple.pvt,OU=Enterprise\ SSL,O=Example\,\ Inc'
plugins.security.nodes_dn:
  - 'CN=opensearch.example.pvt,OU=Enterprise\ SSL,O=Example\,\ Inc'
  - 'CN=opensearch.example.pvt,OU=Enterprise\ SSL,O=Example\,\ Inc'
plugins.security.allow_default_init_securityindex: true
plugins.security.system_indices.enabled: true

This is in the logs:

[INFO ] Config directory is /opt/opensearch-2.6.0-alpha/config/, from there the key- and truststore files are resolved relatively

[ERROR] Default endpoint could not be created, auditlog will not work properly

Still getting handshake errors

15

Hey @stecino

I noticed that error in your file, Not sure if I have over looked this post but by chance you using JAVA default keystore or a custom one?
The reason I ask is because when I see that error sometimes is permission on the certificates or Opensearch cant find the keystore /w the certificates install and/or the wrong cert/s one is installed.

Example here

16

I don’t thinks so I am referencing cert and keys explicitly in my yaml file

[INFO ] Config directory is /opt/opensearch-2.6.0-alpha/config/, from there the key- and truststore files are resolved relatively

That exception is being thrown when I make an API call to the node with admin cert/key via curl

1 Like
17

@pablo anything that may need to be done?
Thanks in advance

18

@stecino How does your node certificate look like inside? Does it follow this order?

-----BEGIN CERTIFICATE-----
<node certificate>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<intermediate certificate>
-----END CERTIFICATE-----

Do you get transport layer node handshake errors? Could you share the output?
If you start just one node, can you connect with securityadmin.sh script?

19

@pablo that’s correct node certificate looks like that
I did what you suggested, just staring off with one node, it loads. This is the log output

[2023-05-09T19:44:00,802][INFO ][o.o.n.Node               ] [nodem03.example.com-alpha] version[2.6.0], pid[93872], build[tar/7203a5af21a8a009aece1474446b437a3c674db6/2023-02-24T18:57:04.388618985Z], OS[Linux/5.4.17-2136.316.7.el8uek.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.6/17.0.6+10]
[2023-05-09T19:44:00,804][INFO ][o.o.n.Node               ] [nodem03.example.com-alpha] JVM home [/opt/opensearch-2.6.0-alpha/jdk], using bundled JDK [true]
[2023-05-09T19:44:00,805][INFO ][o.o.n.Node               ] [nodem03.example.com-alpha] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms2g, -Xmx2g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/opt/opensearch-2.6.0-alpha/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/opt/opensearch-2.6.0/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=1073741824, -Dopensearch.path.home=/opt/opensearch-2.6.0-alpha, -Dopensearch.path.conf=/opt/opensearch-2.6.0-alpha/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2023-05-09T19:44:01,742][WARN ][stderr                   ] [nodem03.example.com-alpha] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2023-05-09T19:44:01,743][WARN ][stderr                   ] [nodem03.example.com-alpha] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2023-05-09T19:44:01,743][WARN ][stderr                   ] [nodem03.example.com-alpha] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2023-05-09T19:44:01,753][INFO ][o.o.s.s.t.SSLConfig      ] [nodem03.example.com-alpha] SSL dual mode is disabled
[2023-05-09T19:44:01,754][INFO ][o.o.s.OpenSearchSecurityPlugin] [nodem03.example.com-alpha] OpenSearch Config path is /opt/opensearch-2.6.0-alpha/config
[2023-05-09T19:44:01,991][INFO ][o.o.s.s.DefaultSecurityKeyStore] [nodem03.example.com-alpha] JVM supports TLSv1.3
[2023-05-09T19:44:01,993][INFO ][o.o.s.s.DefaultSecurityKeyStore] [nodem03.example.com-alpha] Config directory is /opt/opensearch-2.6.0-alpha/config/, from there the key- and truststore files are resolved relatively
[2023-05-09T19:44:02,513][INFO ][o.o.s.s.DefaultSecurityKeyStore] [nodem03.example.com-alpha] TLS Transport Client Provider : JDK
[2023-05-09T19:44:02,514][INFO ][o.o.s.s.DefaultSecurityKeyStore] [nodem03.example.com-alpha] TLS Transport Server Provider : JDK
[2023-05-09T19:44:02,514][INFO ][o.o.s.s.DefaultSecurityKeyStore] [nodem03.example.com-alpha] TLS HTTP Provider             : JDK
[2023-05-09T19:44:02,514][INFO ][o.o.s.s.DefaultSecurityKeyStore] [nodem03.example.com-alpha] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2023-05-09T19:44:02,514][INFO ][o.o.s.s.DefaultSecurityKeyStore] [nodem03.example.com-alpha] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2]
[2023-05-09T19:44:02,525][INFO ][o.o.s.OpenSearchSecurityPlugin] [nodem03.example.com-alpha] Clustername: opensearch-alpha-st1
[2023-05-09T19:44:03,281][INFO ][o.o.i.r.ReindexPlugin    ] [nodem03.example.com-alpha] ReindexPlugin reloadSPI called
[2023-05-09T19:44:03,282][INFO ][o.o.i.r.ReindexPlugin    ] [nodem03.example.com-alpha] Unable to find any implementation for RemoteReindexExtension
[2023-05-09T19:44:03,337][INFO ][o.o.j.JobSchedulerPlugin ] [nodem03.example.com-alpha] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2023-05-09T19:44:03,340][INFO ][o.o.j.JobSchedulerPlugin ] [nodem03.example.com-alpha] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2023-05-09T19:44:03,342][INFO ][o.o.j.JobSchedulerPlugin ] [nodem03.example.com-alpha] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2023-05-09T19:44:03,366][INFO ][o.o.j.JobSchedulerPlugin ] [nodem03.example.com-alpha] Loaded scheduler extension: observability, index: .opensearch-observability-job
[2023-05-09T19:44:03,372][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [aggs-matrix-stats]
[2023-05-09T19:44:03,372][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [analysis-common]
[2023-05-09T19:44:03,372][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [geo]
[2023-05-09T19:44:03,373][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [ingest-common]
[2023-05-09T19:44:03,373][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [ingest-geoip]
[2023-05-09T19:44:03,373][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [ingest-user-agent]
[2023-05-09T19:44:03,373][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [lang-expression]
[2023-05-09T19:44:03,373][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [lang-mustache]
[2023-05-09T19:44:03,373][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [lang-painless]
[2023-05-09T19:44:03,374][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [mapper-extras]
[2023-05-09T19:44:03,374][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [opensearch-dashboards]
[2023-05-09T19:44:03,374][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [parent-join]
[2023-05-09T19:44:03,374][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [percolator]
[2023-05-09T19:44:03,374][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [rank-eval]
[2023-05-09T19:44:03,374][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [reindex]
[2023-05-09T19:44:03,375][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [repository-url]
[2023-05-09T19:44:03,375][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [systemd]
[2023-05-09T19:44:03,375][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded module [transport-netty4]
[2023-05-09T19:44:03,375][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-alerting]
[2023-05-09T19:44:03,376][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-anomaly-detection]
[2023-05-09T19:44:03,376][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-asynchronous-search]
[2023-05-09T19:44:03,376][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-cross-cluster-replication]
[2023-05-09T19:44:03,376][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-geospatial]
[2023-05-09T19:44:03,376][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-index-management]
[2023-05-09T19:44:03,376][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-job-scheduler]
[2023-05-09T19:44:03,377][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-knn]
[2023-05-09T19:44:03,377][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-ml]
[2023-05-09T19:44:03,377][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-neural-search]
[2023-05-09T19:44:03,377][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-notifications]
[2023-05-09T19:44:03,377][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-notifications-core]
[2023-05-09T19:44:03,377][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-observability]
[2023-05-09T19:44:03,377][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-reports-scheduler]
[2023-05-09T19:44:03,377][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-security]
[2023-05-09T19:44:03,378][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-security-analytics]
[2023-05-09T19:44:03,378][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] loaded plugin [opensearch-sql]
[2023-05-09T19:44:03,420][INFO ][o.o.s.OpenSearchSecurityPlugin] [nodem03.example.com-alpha] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2023-05-09T19:44:03,442][INFO ][o.o.e.NodeEnvironment    ] [nodem03.example.com-alpha] using [1] data paths, mounts [[/opt (/dev/mapper/ol-opt)]], net usable_space [8.4gb], net total_space [9.9gb], types [xfs]
[2023-05-09T19:44:03,443][INFO ][o.o.e.NodeEnvironment    ] [nodem03.example.com-alpha] heap size [2gb], compressed ordinary object pointers [true]
[2023-05-09T19:44:03,522][INFO ][o.o.n.Node               ] [nodem03.example.com-alpha] node name [nodem03.example.com-alpha], node ID [L2QC4B7BQX-XJAfSxERlXg], cluster name [opensearch-alpha-st1], roles [cluster_manager]
[2023-05-09T19:44:06,499][WARN ][o.o.s.c.Salt             ] [nodem03.example.com-alpha] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2023-05-09T19:44:06,544][ERROR][o.o.s.a.s.SinkProvider   ] [nodem03.example.com-alpha] Default endpoint could not be created, auditlog will not work properly.
[2023-05-09T19:44:06,545][WARN ][o.o.s.a.r.AuditMessageRouter] [nodem03.example.com-alpha] No default storage available, audit log may not work properly. Please check configuration.
[2023-05-09T19:44:06,545][INFO ][o.o.s.a.i.AuditLogImpl   ] [nodem03.example.com-alpha] Message routing enabled: false
[2023-05-09T19:44:06,570][INFO ][o.o.s.f.SecurityFilter   ] [nodem03.example.com-alpha] <NONE> indices are made immutable.
[2023-05-09T19:44:06,865][INFO ][o.o.a.b.ADCircuitBreakerService] [nodem03.example.com-alpha] Registered memory breaker.
[2023-05-09T19:44:07,186][INFO ][o.o.m.b.MLCircuitBreakerService] [nodem03.example.com-alpha] Registered ML memory breaker.
[2023-05-09T19:44:07,186][INFO ][o.o.m.b.MLCircuitBreakerService] [nodem03.example.com-alpha] Registered ML disk breaker.
[2023-05-09T19:44:07,187][INFO ][o.o.m.b.MLCircuitBreakerService] [nodem03.example.com-alpha] Registered ML native memory breaker.
[2023-05-09T19:44:07,284][INFO ][o.r.Reflections          ] [nodem03.example.com-alpha] Reflections took 36 ms to scan 1 urls, producing 13 keys and 33 values
[2023-05-09T19:44:08,218][INFO ][o.o.t.NettyAllocator     ] [nodem03.example.com-alpha] creating NettyAllocator with the following configs: [name=opensearch_configured, chunk_size=256kb, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=1mb}]
[2023-05-09T19:44:08,304][INFO ][o.o.d.DiscoveryModule    ] [nodem03.example.com-alpha] using discovery type [zen] and seed hosts providers [settings]
[2023-05-09T19:44:08,716][WARN ][o.o.g.DanglingIndicesState] [nodem03.example.com-alpha] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-05-09T19:44:09,167][INFO ][o.o.n.Node               ] [nodem03.example.com-alpha] initialized
[2023-05-09T19:44:09,168][INFO ][o.o.n.Node               ] [nodem03.example.com-alpha] starting ...
[2023-05-09T19:44:09,268][INFO ][o.o.t.TransportService   ] [nodem03.example.com-alpha] publish_address {10.5.xxx.35:9300}, bound_addresses {0.0.0.0:9300}
[2023-05-09T19:44:09,430][INFO ][o.o.b.BootstrapChecks    ] [nodem03.example.com-alpha] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-05-09T19:44:09,434][INFO ][o.o.c.c.Coordinator      ] [nodem03.example.com-alpha] cluster UUID [xp6VVjl6QEGZAPWk01bNfA]
[2023-05-09T19:44:09,491][INFO ][o.o.c.s.MasterService    ] [nodem03.example.com-alpha] elected-as-cluster-manager ([1] nodes joined)[{nodem03.example.com-alpha}{L2QC4B7BQX-XJAfSxERlXg}{7T43q730SoWe0tdUnkkmpw}{10.5.xxx.35}{10.5.106.35:9300}{m}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 4, version: 27, delta: cluster-manager node changed {previous [], current [{nodem03.example.com-alpha}{L2QC4B7BQX-XJAfSxERlXg}{7T43q730SoWe0tdUnkkmpw}{10.5.106.35}{10.5.106.35:9300}{m}{shard_indexing_pressure_enabled=true}]}
[2023-05-09T19:44:09,532][INFO ][o.o.c.s.ClusterApplierService] [nodem03.example.com-alpha] cluster-manager node changed {previous [], current [{nodem03.example.com-alpha}{L2QC4B7BQX-XJAfSxERlXg}{7T43q730SoWe0tdUnkkmpw}{10.5.xxx.35}{10.5.106.35:9300}{m}{shard_indexing_pressure_enabled=true}]}, term: 4, version: 27, reason: Publication{term=4, version=27}
[2023-05-09T19:44:09,539][INFO ][o.o.a.c.ADClusterEventListener] [nodem03.example.com-alpha] Cluster is not recovered yet.
[2023-05-09T19:44:09,545][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [nodem03.example.com-alpha] Detected cluster change event for destination migration
[2023-05-09T19:44:09,564][INFO ][o.o.i.i.ManagedIndexCoordinator] [nodem03.example.com-alpha] Cache cluster manager node onClusterManager time: 1683679449564
[2023-05-09T19:44:09,576][INFO ][o.o.d.PeerFinder         ] [nodem03.example.com-alpha] setting findPeersInterval to [1s] as node commission status = [true] for local node [{nodem03.example.com-alpha}{L2QC4B7BQX-XJAfSxERlXg}{7T43q730SoWe0tdUnkkmpw}{10.5.xxx.35}{10.5.106.35:9300}{m}{shard_indexing_pressure_enabled=true}]
[2023-05-09T19:44:09,581][INFO ][o.o.h.AbstractHttpServerTransport] [nodem03.example.com-alpha] publish_address {10.5.xxx.35:9200}, bound_addresses {0.0.0.0:9200}
[2023-05-09T19:44:09,581][INFO ][o.o.n.Node               ] [nodem03.example.com-alpha] started
[2023-05-09T19:44:09,585][INFO ][o.o.s.OpenSearchSecurityPlugin] [nodem03.example.com-alpha] Node started
[2023-05-09T19:44:09,586][INFO ][o.o.s.c.ConfigurationRepository] [nodem03.example.com-alpha] Will attempt to create index .opendistro_security and default configs if they are absent
[2023-05-09T19:44:09,589][INFO ][o.o.s.OpenSearchSecurityPlugin] [nodem03.example.com-alpha] 0 OpenSearch Security modules loaded so far: []
[2023-05-09T19:44:09,590][INFO ][o.o.s.c.ConfigurationRepository] [nodem03.example.com-alpha] Background init thread started. Install default config?: true
[2023-05-09T19:44:09,591][INFO ][o.o.c.r.a.AllocationService] [nodem03.example.com-alpha] Cluster health status changed from [YELLOW] to [RED] (reason: [state recovered]).
[2023-05-09T19:44:09,609][INFO ][o.o.c.s.ClusterSettings  ] [nodem03.example.com-alpha] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
[2023-05-09T19:44:09,611][INFO ][o.o.a.c.HashRing         ] [nodem03.example.com-alpha] Node added: [L2QC4B7BQX-XJAfSxERlXg]
[2023-05-09T19:44:09,613][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [nodem03.example.com-alpha] Detected cluster change event for destination migration
[2023-05-09T19:44:09,615][INFO ][o.o.a.c.HashRing         ] [nodem03.example.com-alpha] All nodes with known AD version: {L2QC4B7BQX-XJAfSxERlXg=ADNodeInfo{version=2.6.0, isEligibleDataNode=false}}
[2023-05-09T19:44:09,615][INFO ][o.o.a.c.HashRing         ] [nodem03.example.com-alpha] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
[2023-05-09T19:44:09,615][INFO ][o.o.a.c.HashRing         ] [nodem03.example.com-alpha] Build AD version hash ring successfully
[2023-05-09T19:44:09,616][INFO ][o.o.a.c.ADClusterEventListener] [nodem03.example.com-alpha] Init AD version hash ring successfully
[2023-05-09T19:44:09,640][INFO ][o.o.g.GatewayService     ] [nodem03.example.com-alpha] recovered [2] indices into cluster_state
[2023-05-09T19:44:09,642][WARN ][o.o.o.i.ObservabilityIndex] [nodem03.example.com-alpha] message: index [.opensearch-observability/j9-GyJKQS9qh3Ft8NLTB8A] already exists
[2023-05-09T19:44:09,642][INFO ][o.o.o.i.ObservabilityTracesIndex] [nodem03.example.com-alpha] observability:createMappingTemplate sso_trace_template API called
[2023-05-09T19:44:09,642][INFO ][o.o.s.c.ConfigurationRepository] [nodem03.example.com-alpha] Index .opendistro_security already exists
[2023-05-09T19:44:09,643][INFO ][o.o.s.c.ConfigurationRepository] [nodem03.example.com-alpha] Node started, try to initialize it. Wait for at least yellow cluster state....
[2023-05-09T19:44:09,664][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] PluginService:onIndexModule index:[validate-template-qtkufpgqqai8tcwpb63uvw/eIGGPRJQSaO9BDOLgi2IWQ]
[2023-05-09T19:44:09,795][INFO ][o.o.c.m.MetadataIndexTemplateService] [nodem03.example.com-alpha] updating index template [sso_trace_template] for index patterns [sso_traces-*-*]
[2023-05-09T19:44:09,854][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [nodem03.example.com-alpha] Detected cluster change event for destination migration
[2023-05-09T19:44:09,856][INFO ][o.o.o.i.ObservabilityTracesIndex] [nodem03.example.com-alpha] observability:Mapping Template sso_trace_template creation Acknowledged
[2023-05-09T19:44:09,856][INFO ][o.o.o.i.ObservabilityMetricsIndex] [nodem03.example.com-alpha] observability:createMappingTemplate sso_metric_template API called
[2023-05-09T19:44:09,861][INFO ][o.o.p.PluginsService     ] [nodem03.example.com-alpha] PluginService:onIndexModule index:[validate-template-rt6xyqrbtbye_snz5rrnla/zrnffK1vQ82Tb9fXbj9d-g]
[2023-05-09T19:44:09,878][INFO ][o.o.c.m.MetadataIndexTemplateService] [nodem03.example.com-alpha] updating index template [sso_metric_template] for index patterns [sso_metrics-*-*]
[2023-05-09T19:44:09,907][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [nodem03.example.com-alpha] Detected cluster change event for destination migration
[2023-05-09T19:44:09,908][INFO ][o.o.o.i.ObservabilityMetricsIndex] [nodem03.example.com-alpha] observability:Mapping Template sso_metric_template creation Acknowledge

When I try to run an api call against it I get certificate unknown error

curl -vk -GET --key opensearch_admin_key.pem --cert opensearch_admin_cert.pem https://nodem03.example.pvt:9200/_cat/nodes

LibreSSL SSL_read: LibreSSL/3.3.6: error:1404C416:SSL routines:ST_OK:sslv3 alert certificate unknown, errno 0
* Closing connection 0
curl: (56) LibreSSL SSL_read: LibreSSL/3.3.6: error:1404C416:SSL routines:ST_OK:sslv3 alert certificate unknown, errno 0
20

@pablo any thoughts?