Using Helm chart, demo install with default certs fails to start error

abhyankar · April 7, 2024, 2:13pm

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch: chart: opensearch-2.19.0
Virtualization: Oracle Virtualbox 7.0.12
k8s system: minikube
Virtual OS: Ubuntu 22.04.3 LTS
Base/bare metal OS: Windows 10
Available disk space in Ubuntu: 6.2GB [ after initialization of the opensearch deployment]

Describe the issue:

Path: /home/amey/k8s-projects/opensearch

output of pwdcommand: values.yaml

Installation command:
helm install my-deployment opensea rch/opensearch --set extraEnvs[0].name=OPENSEARCH_INITIAL_ADMIN_PASSWORD,extraEn vs[0].value=$OPENSEARCH_INITIAL_ADMIN_PASSWORD=q4r9lwnWDm3hrjI

Changes in values.yaml =

Added variable with password
persistent storage disabled in values.yaml file as per hxxps://forum.opensearch.org/t/using-helm-chart-demo-install-with-default-certs-fails-to-start/18506
Using default ssl certs

Configuration:

amey@ubuntu22-test:~/k8s-projects/opensearch$ k get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-5dd5756b68-p8rqx 1/1 Running 27 (6d19h ago) 72d
etcd-minikube 1/1 Running 27 (6d19h ago) 72d
kindnet-dvdq5 1/1 Running 20 (6d19h ago) 48d
kube-apiserver-minikube 1/1 Running 27 (6d19h ago) 72d
kube-controller-manager-minikube 1/1 Running 27 (6d19h ago) 72d
kube-proxy-sstfh 1/1 Running 27 (6d19h ago) 72d
kube-scheduler-minikube 1/1 Running 27 (6d19h ago) 72d
metrics-server-7c66d45ddc-kgltl 1/1 Running 42 (6d19h ago) 69d
storage-provisioner 1/1 Running 58 (7h41m ago) 72d

Relevant Logs or Screenshots:

Defaulted container “opensearch” out of: opensearch, fsgroup-volume (init), configfile (init)
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for ‘admin’ user.
Please define an environment variable ‘OPENSEARCH_INITIAL_ADMIN_PASSWORD’ with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: Docker - OpenSearch Documentation

OpenSearch Security Demo Installer

Warning: Do not use on production or public reachable systems

OpenSearch install type: rpm/deb on Linux 6.5.0-26-generic amd64

[2024-04-07T13:54:58,299][INFO ][o.o.i.r.ReindexPlugin ] [opensearch-cluster-master-0] Unable to find any implementation for RemoteReindexExtension
[2024-04-07T13:54:58,411][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2024-04-07T13:54:58,416][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2024-04-07T13:54:58,417][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2024-04-07T13:54:58,419][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
[2024-04-07T13:54:58,420][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: opensearch_sap_job, index: .opensearch-sap–job
[2024-04-07T13:54:58,442][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [aggs-matrix-stats]

Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: OpenSearchException[Unable to read /usr/share/opensearch/config/esnode.pem (/usr/share/opensearch/config/esnode.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:484) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:298) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:204) ~[?:?]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:235) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:295) ~[?:?]
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:794) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:743) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.(PluginsService.java:196) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.node.Node.(Node.java:490) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.node.Node.(Node.java:417) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.13.0.jar:2.13.0]
… 6 more
Caused by: org.opensearch.OpenSearchException: Unable to read /usr/share/opensearch/config/esnode.pem (/usr/share/opensearch/config/esnode.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath
at org.opensearch.security.ssl.DefaultSecurityKeyStore.checkPath(DefaultSecurityKeyStore.java:1135) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.resolve(DefaultSecurityKeyStore.java:276) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:454) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:298) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:204) ~[?:?]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:235) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:295) ~[?:?]
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:794) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:743) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.(PluginsService.java:196) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.node.Node.(Node.java:490) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.node.Node.(Node.java:417) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.13.0.jar:2.13.0]
… 6 more
uncaught exception in thread [main]
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[Unable to read /usr/share/opensearch/config/esnode.pem (/usr/share/opensearch/config/esnode.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.checkPath(DefaultSecurityKeyStore.java:1135)
at org.opensearch.security.ssl.DefaultSecurityKeyStore.resolve(DefaultSecurityKeyStore.java:276)
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:454)
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:298)
at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:204)
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:235)
at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:295)
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:794)
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:743)
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544)
at org.opensearch.plugins.PluginsService.(PluginsService.java:196)
at org.opensearch.node.Node.(Node.java:490)
at org.opensearch.node.Node.(Node.java:417)
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242)
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)
For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch-cluster.log

Gsmitt · April 7, 2024, 6:58pm

Hey @abhyankar

It looks like OS/OSD cant find/read you certificates. If there there, I would check permissions first.

Mantas · April 8, 2024, 2:14pm

Hi @abhyankar,

Would you mind sharing your full values.yaml file?

Thanks,
mj

abhyankar · April 13, 2024, 5:49am

Sure.

I am taking values from = helm-charts/charts/opensearch/values.yaml at main · opensearch-project/helm-charts · GitHub

Modifications = line number 105, line number 110.

Thanks,
Amey.

Mantas · April 15, 2024, 11:00am

Hi @abhyankar,

Could you run (on one of your OpenSearch nodes) ls -l /usr/share/opensearch/config/ and share the output?

Thanks,
mj

abhyankar · May 4, 2024, 8:07pm

Now the master node keeps crashing.

I added config in values.yaml from other post = Multiple exceptions related to plugins.security - #2 by Mantas

Error:

Blockquote
Defaulted container “opensearch” out of: opensearch, fsgroup-volume (init), configfile (init)
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for ‘admin’ user.
Please define an environment variable ‘OPENSEARCH_INITIAL_ADMIN_PASSWORD’ with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: Docker - OpenSearch Documentation

OpenSearch Security Demo Installer

Warning: Do not use on production or public reachable systems

OpenSearch install type: rpm/deb on Linux 6.5.0-26-generic amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.13.0
Detected OpenSearch Security Version: 2.13.0.0
/usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.13.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
May 04, 2024 8:05:27 PM sun.util.locale.provider.LocaleProviderAdapter
WARNING: COMPAT locale provider will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.13.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2024-05-04T20:05:28,417][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] version[2.13.0], pid[36], build[tar/7ec678d1b7c87d6e779fdef94e33623e1f1e2647/2024-03-26T00:02:39.659767978Z], OS[Linux/6.5.0-26-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.2/21.0.2+13-LTS]
[2024-05-04T20:05:28,419][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2024-05-04T20:05:28,420][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-3769741321000128473, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx512M, -Xms512M, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2024-05-04T20:05:29,790][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-cluster-master-0] SSL dual mode is disabled
[2024-05-04T20:05:29,791][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] OpenSearch Config path is /usr/share/opensearch/config
[2024-05-04T20:05:30,071][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] JVM supports TLSv1.3
[2024-05-04T20:05:30,073][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2024-05-04T20:05:30,087][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-master-0] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.13.0.jar:2.13.0]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.13.0.jar:2.13.0]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:803) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:743) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.(PluginsService.java:196) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.node.Node.(Node.java:490) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.node.Node.(Node.java:417) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.13.0.jar:2.13.0]
… 6 more
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:794) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:743) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.(PluginsService.java:196) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.node.Node.(Node.java:490) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.node.Node.(Node.java:417) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.13.0.jar:2.13.0]
… 6 more
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: OpenSearchException[Unable to read /usr/share/opensearch/config/esnode.pem (/usr/share/opensearch/config/esnode.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:484) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:298) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:204) ~[?:?]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:235) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:295) ~[?:?]
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:794) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:743) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.(PluginsService.java:196) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.node.Node.(Node.java:490) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.node.Node.(Node.java:417) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.13.0.jar:2.13.0]
… 6 more
Caused by: org.opensearch.OpenSearchException: Unable to read /usr/share/opensearch/config/esnode.pem (/usr/share/opensearch/config/esnode.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath
at org.opensearch.security.ssl.DefaultSecurityKeyStore.checkPath(DefaultSecurityKeyStore.java:1135) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.resolve(DefaultSecurityKeyStore.java:276) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:454) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:298) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:204) ~[?:?]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:235) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:295) ~[?:?]
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:794) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:743) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.plugins.PluginsService.(PluginsService.java:196) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.node.Node.(Node.java:490) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.node.Node.(Node.java:417) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.13.0.jar:2.13.0]
… 6 more
uncaught exception in thread [main]
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[Unable to read /usr/share/opensearch/config/esnode.pem (/usr/share/opensearch/config/esnode.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.checkPath(DefaultSecurityKeyStore.java:1135)
at org.opensearch.security.ssl.DefaultSecurityKeyStore.resolve(DefaultSecurityKeyStore.java:276)
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:454)
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:298)
at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:204)
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:235)
at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:295)
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:794)
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:743)
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544)
at org.opensearch.plugins.PluginsService.(PluginsService.java:196)
at org.opensearch.node.Node.(Node.java:490)
at org.opensearch.node.Node.(Node.java:417)
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242)
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)
For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch-cluster.log

abhyankar · May 5, 2024, 1:02pm

Quik update.

After adding content from helm-charts/charts/opensearch/ci/ci-values.yaml at main · opensearch-project/helm-charts · GitHub my opensearch pod is running fine without any issues.

I did not configure following block =

data: {}
# config.yml: |-
# internal_users.yml: |-
# roles.yml: |-
# roles_mapping.yml: |-
# action_groups.yml: |-
# tenants.yml: |-

I am not sure why the helm dpeloyment fails for 3 node cluster with following values.yaml = helm-charts/charts/opensearch/values.yaml at main · opensearch-project/helm-charts · GitHub

rawshaan · June 13, 2024, 6:16am

hi @abhyankar
Can you please elaborate on what you meant by adding content from the ci-values.yaml file? I see that this file already exists in the helm chart of Opensearch when downloaded from Git Hub? what extra was added to it that made it work for you?

abhyankar · June 18, 2024, 6:48am

I mean the content from line number 100 to 104 from helm-charts/charts/opensearch/ci/ci-values.yaml at main · opensearch-project/helm-charts · GitHub

These lines are commented in other chart and if I try to use some random password, that was not working.

I did not try to put the content from line number 100 to 104 as it is and put in to the other yaml and deploy as my cluster is single node cluster.

walchinsky · September 28, 2024, 1:46am

The specific section of code that I needed to run to get the pods running without error in k3s was to make it a single node

singleNode: true

So basically I was able to do a successful deploy with the following override on the default values.
helm install my-deployment opensearch/opensearch -f override.yaml

---
singleNode: true
extraEnvs:
  - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
    value: myStrongPassword123@456

If I didn’t do that then I’d get the error at the start of the thread.
“Unable to read /usr/share/opensearch/config/esnode.pem (/usr/share/opensearch/config/esnode.pem”

Hopefully that helps others!

Topic		Replies	Views
Opensearch fails to start with default configuration (demo install) OpenSearch	4	1056	June 25, 2024
Helm Chart demo install with demo certs fails to start OpenSearch install	11	556	July 10, 2024
Using Helm chart, demo install with default certs fails to start OpenSearch install	9	1889	July 2, 2024
Error installing with default configuration OpenSearch troubleshoot , install	1	890	June 22, 2023
How do I configure demo-replacement certificate for helm chart? OpenSearch install	6	67	October 2, 2024

Using Helm chart, demo install with default certs fails to start error

OpenSearch Security Demo Installer

** Warning: Do not use on production or public reachable systems **

OpenSearch Security Demo Installer

** Warning: Do not use on production or public reachable systems **

Related topics

Warning: Do not use on production or public reachable systems

Warning: Do not use on production or public reachable systems