Error in OpenSearch SSL Configuration

Security
1

Hi @all,
We are trying to configure the OpenSearch using tarball.
Now we are trying to integrate the self signed SSL certificate with the OpenSearch.
Here we are using letsencrypt SSL certificate.

Server Details:
OS: Rocky Linux (version: 8.5)
OpenSearch Version:1.2.4

Java : bundled jdk with OpenSearch

java home: /oss/bin/opensearch/jdk
openjdk version "15.0.1" 2020-10-20
OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.1+9, mixed mode, sharing)

Here is the OpenSearch configurarion file:

plugins.security.ssl.transport.pemcert_filepath: /oss/bin/opensearch/config/cert1.pem
plugins.security.ssl.transport.pemkey_filepath: /oss/bin/opensearch/config/privkey1.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /oss/bin/opensearch/config/fullchain1.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=test, C=de

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3

Now after execute the opensearch-tar-install.sh file, we geetting these error:

OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /oss/bin/opensearch
OpenSearch install type: .tar.gz on Rocky Linux release 8.5 (Green Obsidian)
OpenSearch config dir: /oss/bin/opensearch/config
OpenSearch config file: /oss/bin/opensearch/config/opensearch.yml
OpenSearch bin dir: /oss/bin/opensearch/bin
OpenSearch plugins dir: /oss/bin/opensearch/plugins
OpenSearch lib dir: /oss/bin/opensearch/lib
Detected OpenSearch Version: x-content-1.2.4
Detected OpenSearch Security Version: 1.2.4.0
/oss/bin/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
done security
done plugins
k-NN libraries not found in LD_LIBRARY_PATH. Updating path to: :/oss/bin/opensearch/plugins/opensearch-knn/knn                                                              lib.
Starting OpenSearch
[2022-03-24T14:15:07,408][INFO ][o.o.n.Node               ] [node-1] version[1.2.4], pid[41516], build[tar/e505b10357c03ae8d26d675172402f2f2144ef0f/2022-01-14T03:38:06.881862Z], OS[Linux/4.18.0-348.20.1.el8_5.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2022-03-24T14:15:07,413][INFO ][o.o.n.Node               ] [node-1] JVM home [/oss/bin/opensearch/jdk], using bundled JDK [true]
[2022-03-24T14:15:07,413][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-12117070065417233059, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/oss/bin/opensearch/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/oss/bin/opensearch, -Dopensearch.path.conf=/oss/bin/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2022-03-24T14:15:08,846][INFO ][o.o.s.s.t.SSLConfig      ] [node-1] SSL dual mode is disabled
[2022-03-24T14:15:08,847][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] OpenSearch Config path is /oss/bin/opensearch/config
[2022-03-24T14:15:09,191][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] JVM supports TLSv1.3
[2022-03-24T14:15:09,194][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Config directory is /oss/bin/opensearch/config/, from there the key- and truststore files are resolved relatively
[2022-03-24T14:15:09,679][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:182) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-1.2.4.jar:1.2.4]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-1.2.4.jar:1.2.4]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101) ~[opensearch-1.2.4.jar:1.2.4]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:790) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:726) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:528) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:194) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.2.4.jar:1.2.4]
        ... 6 more
Caused by: java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:726) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:528) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:194) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.2.4.jar:1.2.4]
        ... 6 more
Caused by: org.opensearch.OpenSearchException: plugins.security.ssl.http.keystore_filepath or plugins.security.ssl.http.pemkey_filepath must be set if http ssl is requested.
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initHttpSSLConfig(DefaultSecurityKeyStore.java:541) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:264) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:252) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:726) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:528) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:194) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.2.4.jar:1.2.4]
        ... 6 more
uncaught exception in thread [main]
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[plugins.security.ssl.http.keystore_filepath or plugins.security.ssl.http.pemkey_filepath must be set if http ssl is requested.]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initHttpSSLConfig(DefaultSecurityKeyStore.java:541)
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:264)
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179)
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218)
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:252)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781)
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:726)
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:528)
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:194)
        at org.opensearch.node.Node.<init>(Node.java:396)
        at org.opensearch.node.Node.<init>(Node.java:319)
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412)
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178)
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169)
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.Command.main(Command.java:101)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101)
For complete error details, refer to the log at /oss/logs/opensearch/Subhasree-Reporting.log

If anyone could point me in the right direction that would be great help.
Thanks

2

@avnishs You specified plugins.security.ssl.http.enabled: true, meaning you would like to use https to call opensearch, however you have not specified where the certificates are located.

You only mentioned where the certs used for transport are located.

Try to add the below lines to the config.

plugins.security.ssl.http.pemcert_filepath: /oss/bin/opensearch/config/cert1.pem
plugins.security.ssl.http.pemkey_filepath: /oss/bin/opensearch/config/privkey1.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /oss/bin/opensearch/config/fullchain1.pem

1 Like
3

First you may want to look and read carefully document here: Generate certificates - OpenSearch documentation

and second take a look in:

plugins.security.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=test, C=de

Above is the default when you use default security config, you should change its information to your LE cert, it takes me about an hour to understand a little configuration of it.

4

Thanks @Anthony problem was solved with your assistance.