**Is your feature request related to a problem?**
My current goal is provide ou…r developers ability to watch over time some conditions and when reached matching criteria, they will be noticed via Prometheus AlertManager. Unfortunately, we would like to have more secure infrastructure, so internal AlertManager endpoint is secured via SSL certificate, which is signed by our internal CA. And here is a problem with supporting that scenario.
**What solution would you like?**
Have some settings (in the OS Dashboard GUI preferable, but settings in config files are fine too) to specify custom CA, which is used to signed different endpoints for webhook targets.
**What alternatives have you considered?**
A some way, how to set or specify custom CA certificate for webhook endpoint.
**Do you have any additional context?**
If the endpoint isnt secured, everything working fine, but we need rise security.
I asked [on the forum](https://forum.opensearch.org/t/custom-webhook-from-opensearch-dashboard-to-alertmanager/14940) and I was pointed to the [old closed issue](https://github.com/opensearch-project/alerting/issues/65#issuecomment-853402473) with workaround. Unfortunately, with OpenSearch version 2.7.0 the workaround not working. If I made trustore with `keytool` utility from the Docker image, JKS trustore looks good for the utility, but OpenSearch refuse start properly with bunch of warnings and errors. Looks like:
```
{"type": "logging", "timestamp": "2023-07-14T12:06:53,170Z", "level": "ERROR", "component": "o.o.b.OpenSearchUncaughtExceptionHandler", "cluster.name": "logging", "node.name": "ofd-client-77f4d94d7c-84nsr", "message": "uncaught exception in thread [main]",
"stacktrace": ["org.opensearch.bootstrap.StartupException: org.opensearch.common.ssl.SslConfigException: failed to initialize a TrustManager for the system keystore",
"at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.7.0.jar:2.7.0]",
"at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.7.0.jar:2.7.0]",
"Caused by: org.opensearch.common.ssl.SslConfigException: failed to initialize a TrustManager for the system keystore",
"at org.opensearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:83) ~[?:?]",
"at org.opensearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:155) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:160) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:130) ~[?:?]",
"at org.opensearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:129) ~[?:?]",
"at org.opensearch.node.Node.lambda$new$16(Node.java:788) ~[opensearch-2.7.0.jar:2.7.0]",
"at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]",
"at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]",
"at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]",
"at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]",
"at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]",
"at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]",
"at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]",
"at org.opensearch.node.Node.<init>(Node.java:802) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.node.Node.<init>(Node.java:375) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.7.0.jar:2.7.0]",
"... 6 more",
"Caused by: java.security.KeyStoreException: problem accessing trust store",
"at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:73) ~[?:?]",
"at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282) ~[?:?]",
"at org.opensearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:168) ~[?:?]",
"at org.opensearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:81) ~[?:?]",
"at org.opensearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:155) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:160) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:130) ~[?:?]",
"at org.opensearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:129) ~[?:?]",
"at org.opensearch.node.Node.lambda$new$16(Node.java:788) ~[opensearch-2.7.0.jar:2.7.0]",
"at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]",
uncaught exception in thread [main]
"at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]",
"at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]",
"at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]",
"at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]",
"at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]",
"at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]",
"at org.opensearch.node.Node.<init>(Node.java:802) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.node.Node.<init>(Node.java:375) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.7.0.jar:2.7.0]",
"... 6 more",
"Caused by: java.io.IOException: keystore password was incorrect",
"at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159) ~[?:?]",
"at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:242) ~[?:?]",
"at java.security.KeyStore.load(KeyStore.java:1473) ~[?:?]",
"at sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:390) ~[?:?]",
"at sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:336) ~[?:?]",
"at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:57) ~[?:?]",
"at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]",
"at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282) ~[?:?]",
"at org.opensearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:168) ~[?:?]",
"at org.opensearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:81) ~[?:?]",
"at org.opensearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:155) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:160) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:130) ~[?:?]",
"at org.opensearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:129) ~[?:?]",
"at org.opensearch.node.Node.lambda$new$16(Node.java:788) ~[opensearch-2.7.0.jar:2.7.0]",
"at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]",
"at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]",
"at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]",
"at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]",
"at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]",
"at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]",
"at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]",
"at org.opensearch.node.Node.<init>(Node.java:802) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.node.Node.<init>(Node.java:375) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.7.0.jar:2.7.0]",
"... 6 more",
"Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.",
"at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159) ~[?:?]",
"at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:242) ~[?:?]",
"at java.security.KeyStore.load(KeyStore.java:1473) ~[?:?]",
"at sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:390) ~[?:?]",
"at sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:336) ~[?:?]",
"at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:57) ~[?:?]",
"at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]",
"at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282) ~[?:?]",
"at org.opensearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:168) ~[?:?]",
"at org.opensearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:81) ~[?:?]",
"at org.opensearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:155) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:160) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:130) ~[?:?]",
"at org.opensearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:129) ~[?:?]",
"at org.opensearch.node.Node.lambda$new$16(Node.java:788) ~[opensearch-2.7.0.jar:2.7.0]",
"at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]",
"at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]",
"at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]",
"at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]",
"at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]",
"at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]",
"at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]",
"at org.opensearch.node.Node.<init>(Node.java:802) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.node.Node.<init>(Node.java:375) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.7.0.jar:2.7.0]",
"... 6 more"] }
org.opensearch.common.ssl.SslConfigException: failed to initialize a TrustManager for the system keystore
Likely root cause: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159)
at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:242)
at java.base/java.security.KeyStore.load(KeyStore.java:1473)
at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:390)
at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:336)
at java.base/sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:57)
at java.base/sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49)
at java.base/javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282)
at org.opensearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:168)
at org.opensearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:81)
at org.opensearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:155)
at org.opensearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:160)
at org.opensearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:130)
at org.opensearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:129)
at org.opensearch.node.Node.lambda$new$16(Node.java:788)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.opensearch.node.Node.<init>(Node.java:802)
at org.opensearch.node.Node.<init>(Node.java:375)
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
<<<truncated>>>
For complete error details, refer to the log at /usr/share/opensearch/logs/logging.log
```
For generating truststore I used the command:
```
/usr/share/opensearch/jdk/bin/keytool \
-importcert -trustcacerts -noprompt -file /tmp/devel-ca.pem \
-alias company-ca -keystore /shared/truststore.jks \
-storepass password123
```
Java settings for OpenSearch Client node looks like:
```
-Djavax.net.ssl.trustStore=/usr/share/opensearch/config/truststore.jks
-Djavax.net.ssl.trustStoreType=JKS
-Djavax.net.ssl.trustStorePassword="password123"
```
If I check truststore with `keytool`, everything looks good:
```
/usr/share/opensearch/jdk/bin/keytool -list -v -keystore /shared/truststore.jks -storepass password123
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 1 entry
Alias name: company-ca
Creation date: Jul 14, 2023
Entry type: trustedCertEntry
Owner: CN=cluster:devel
Issuer: CN=fake-ca
Serial number: 5a70d3abd4521488e4f702589377772cdcae0c2d
Valid from: Fri Jun 23 07:18:00 UTC 2023 until: Sat Jun 21 07:18:00 UTC 2031
Certificate fingerprints:
SHA1: 9E:BA:EA:44:A5:D0:2D:B4:7C:3A:95:AD:D7:A1:9E:26:AA:CB:F3:87
SHA256: D3:C8:E9:8B:3D:1E:53:1D:15:55:3D:26:A1:7D:F1:CE:01:F3:E2:01:55:2C:2E:94:D4:2C:EA:3E:EA:FC:BC:12
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
Extensions:
#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 78 C4 D4 ED 99 07 BF D0 A3 18 11 1F 43 A8 18 BA x...........C...
0010: 5F 9C 11 B8 _...
]
]
#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:0
]
#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
serverAuth
clientAuth
]
#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Key_Encipherment
Key_CertSign
Crl_Sign
]
#5: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 70 91 5C 95 80 1F 77 14 DE 7C 19 3A 48 0B D6 FA p.\...w....:H...
0010: 1E 67 69 DC .gi.
]
]
*******************************************
*******************************************
```
If you need some additional information, please, let me know.