Official Ansible role fails to start OpenSearch

m_mlk · June 10, 2022, 3:02pm

Hi all!

I’m using the official Ansible playbook/role to deploy an OpenSearch cluster on newly created VMs which I destroy and recreate for each deployment attempt to make sure that no remains interfere in any way.

I have observed some issues so far:

The task:

TASK [opensearch : Security Plugin configuration | Copy the node & admin certificates to opensearch nodes]

…sometimes copies the certificates, sometimes it does not. I had to comment the "when: " statement to force the copy…

More important: OpenSearch fails to start and throws a Java exception:

Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 opensearch[4461]: Jun 10 14:21:16 os1 systemd[1]: Jun 10 14:21:16 os1 systemd[1]: uncaught exception in thread [main]
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:422)
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258)
at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:179)
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:218)
at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:255)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781)
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730)
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532)
at org.opensearch.plugins.PluginsService.(PluginsService.java:195)
at org.opensearch.node.Node.(Node.java:413)
at org.opensearch.node.Node.(Node.java:336)
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:244)
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:244)
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:414)
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
For complete error details, refer to the log at /usr/share/opensearch/logs/os-cluster.log
opensearch.service: Main process exited, code=exited, status=1/FAILURE
opensearch.service: Failed with result ‘exit-code’.

I have seen these 2 other entries but none of the solutions seem to apply:

OpenSearch ssl exception (tar) ← aren’t certs automatically generated?

Tarball install security error ← this should be handled by the playbook as well, or?

Any ideas as to why this is happening?

Thanks in advance!

Best

m_mlk · June 10, 2022, 3:51pm

I have now deployed 2 CentOS7 VMs because of the banner shown here Redirecting…

The Ansible playbook only supports deployment of OpenSearch and OpenSearch Dashboards to CentOS7 hosts.

Unfortunately, I get the same results:

Jun 10 15:48:19 localhost systemd: Started opensearch.
Jun 10 15:48:23 localhost opensearch: WARNING: A terminally deprecated method in java.lang.System has been called
Jun 10 15:48:23 localhost opensearch: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.0.0.jar)
Jun 10 15:48:23 localhost opensearch: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Jun 10 15:48:23 localhost opensearch: WARNING: System::setSecurityManager will be removed in a future release
Jun 10 15:48:24 localhost ansible-ansible.legacy.command: Invoked with creates=None executable=None _uses_shell=False strip_empty_ends=True _raw_params=sed -i ‘/cluster.initial_master_nodes
/d’ “/usr/share/opensearch/config/opensearch.yml” removes=None argv=None warn=False chdir=None stdin_add_newline=True stdin=None
Jun 10 15:48:25 localhost ansible-ansible.legacy.systemd: Invoked with no_block=False force=None name=opensearch daemon_reexec=False enabled=True daemon_reload=False state=started masked=No
ne scope=system
Jun 10 15:48:26 localhost ansible-ansible.legacy.command: Invoked with creates=None executable=None _uses_shell=False strip_empty_ends=True _raw_params=/usr/share/opensearch/bin/opensearch-
plugin list removes=None argv=None warn=False chdir=None stdin_add_newline=True stdin=None
Jun 10 15:48:29 localhost opensearch: WARNING: A terminally deprecated method in java.lang.System has been called
Jun 10 15:48:29 localhost opensearch: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.0.0.jar)
Jun 10 15:48:29 localhost opensearch: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Jun 10 15:48:29 localhost opensearch: WARNING: System::setSecurityManager will be removed in a future release
Jun 10 15:48:31 localhost ansible-wait_for: Invoked with active_connection_states=[‘ESTABLISHED’, ‘FIN_WAIT1’, ‘FIN_WAIT2’, ‘SYN_RECV’, ‘SYN_SENT’, ‘TIME_WAIT’] state=started connect_timeou
t=1 delay=5 search_regex=None host=192.168.56.40 sleep=1 timeout=300 exclude_hosts=None msg=None path=None port=9200
Jun 10 15:48:32 localhost opensearch: uncaught exception in thread [main]
Jun 10 15:48:32 localhost opensearch: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Jun 10 15:48:32 localhost opensearch: Likely root cause: OpenSearchException[plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and p
lugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.]
Jun 10 15:48:32 localhost opensearch: at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:422)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:179)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:218)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:255)
Jun 10 15:48:32 localhost opensearch: at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Jun 10 15:48:32 localhost opensearch: at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
Jun 10 15:48:32 localhost opensearch: at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Jun 10 15:48:32 localhost opensearch: at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
Jun 10 15:48:32 localhost opensearch: at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.plugins.PluginsService.(PluginsService.java:195)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.node.Node.(Node.java:413)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.node.Node.(Node.java:336)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:244)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:244)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:414)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.cli.Command.main(Command.java:101)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Jun 10 15:48:32 localhost opensearch: For complete error details, refer to the log at /usr/share/opensearch/logs/btcs-cluster.log
Jun 10 15:48:32 localhost systemd: opensearch.service: main process exited, code=exited, status=1/FAILURE
Jun 10 15:48:32 localhost systemd: Unit opensearch.service entered failed state.
Jun 10 15:48:32 localhost systemd: opensearch.service failed.

Your assistance is much appreciated.

Have a good weekend!

m_mlk · June 13, 2022, 2:13pm

Hello again!

I tried the manual installation path described here: Redirecting…

Same errors:

root@os1:~/opensearch-2.0.0# ./opensearch-tar-install.sh

** This tool will be deprecated in the next major release of OpenSearch **
** [DEPRECATION] Security Plugin Tools will be replaced · Issue #1755 · opensearch-project/security · GitHub **

OpenSearch Security Demo Installer
** Warning: Do not use on production or public reachable systems **
Basedir: /root/opensearch-2.0.0
OpenSearch install type: .tar.gz on DISTRIB_ID=Ubuntu
OpenSearch config dir: /root/opensearch-2.0.0/config
OpenSearch config file: /root/opensearch-2.0.0/config/opensearch.yml
OpenSearch bin dir: /root/opensearch-2.0.0/bin
OpenSearch plugins dir: /root/opensearch-2.0.0/plugins
OpenSearch lib dir: /root/opensearch-2.0.0/lib
Detected OpenSearch Version: x-content-2.0.0
Detected OpenSearch Security Version: 2.0.0.0
/root/opensearch-2.0.0/config/opensearch.yml seems to be already configured for Security. Quit.
done security
done plugins
k-NN libraries not found in LD_LIBRARY_PATH. Updating path to: :/root/opensearch-2.0.0/plugins/opensearch-knn/lib.
Starting OpenSearch
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/root/opensearch-2.0.0/lib/opensearch-2.0.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
[2022-06-13T09:32:10,581][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [os1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.RuntimeException: can not run opensearch as root
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.0.0.jar:2.0.0]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.0.0.jar:2.0.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.0.0.jar:2.0.0]
Caused by: java.lang.RuntimeException: can not run opensearch as root
at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:126) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:193) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:414) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.0.0.jar:2.0.0]
… 6 more
uncaught exception in thread [main]
java.lang.RuntimeException: can not run opensearch as root
at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:126)
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:193)
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:414)
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
For complete error details, refer to the log at /root/opensearch-2.0.0/logs/opensearch.log
root@os1:~/opensearch-2.0.0#

Ideas?

Thanks!

m_mlk · June 13, 2022, 2:14pm

The Java exception occurs at this point:

TASK [linux/opensearch : Make sure opensearch is started] ********************************************************************************************************************************************
ok: [os1]
ok: [os2]

TASK [linux/opensearch : Get all the installed ES plugins] *******************************************************************************************************************************************
changed: [os1]
changed: [os2]

TASK [linux/opensearch : Show all the installed ES plugins] ******************************************************************************************************************************************
ok: [os1] => {
“msg”: “opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-ml\nopensearch-notifications\nopensearch-notifications-core\nopensearch-observability\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql”
}
ok: [os2] => {
“msg”: “opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-ml\nopensearch-notifications\nopensearch-notifications-core\nopensearch-observability\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql”
}

TASK [linux/opensearch : Wait for opensearch to startup] *********************************************************************************************************************************************

Your assistance with this is much appreciated!

TIA!

m_mlk · June 30, 2022, 7:19am

Replying to myself once again

I solved the issue by:
a) using CentOS 7.9 as the operating system in the VMs
b) replacing the original installation method from tarball and using the RPM instead.

Original code snippet from tasks/opensearch.yml:

- name: OpenSearch Install | Download opensearch {{ os_version }}
  get_url:
    url: "{{ os_download_url }}/{{ os_version }}/opensearch-{{ os_version }}-linux-x64.tar.gz"
    dest: "/tmp/opensearch.tar.gz"
  register: download

- name: OpenSearch Install | Create opensearch user
  user:
    name: "{{ os_user }}"
    state: present
    shell: /bin/bash
  when: download.changed

- name: OpenSearch Install | Create home directory
  file:
    path: "{{ os_home }}"
    state: directory
    owner: "{{ os_user }}"
    group: "{{ os_user }}"
  when: download.changed

- name: OpenSearch Install | Extract the tar file
  command: chdir=/tmp/ tar -xvzf opensearch.tar.gz -C "{{ os_home }}" --strip-components=1
  when: download.changed

Replaced with:

- name: OpenSearch Install | Import PGP key
  rpm_key:
    state: present
    key: https://artifacts.opensearch.org/publickeys/opensearch.pgp

- name: OpenSearch Install | Add OpenSearch 2.x repository
  get_url:
    url: https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/opensearch-2.x.repo
    dest: /etc/yum.repos.d/opensearch-2.x.repo
    owner: root
    group: root
    mode: 0644

- name: OpenSearch Install | Add OpenSearch Dashboards 2.x repository
  get_url:
    url: https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/2.x/opensearch-dashboards-2.x.repo
    dest: /etc/yum.repos.d/opensearch-dashboards-2.x.repo
    owner: root
    group: root
    mode: 0644

- name: OpenSearch Install | Install packages
  package:
    name:
      - opensearch
      - opensearch-dashboards
      - firewalld
    state: latest

- name: OpenSearch Install | Start firewalld
  systemd:
    name: firewalld
    daemon_reload: yes
    state: started

OpenSearch starts as expected.

Topic		Replies	Views
OpenSearch ssl exception (tar) OpenDistro	2	7698	December 9, 2021
Unable to start opensearch from command line (Error: plugins.security.ssl.transport.keystore_filepath) OpenSearch configure , security-issue	4	1016	July 9, 2024
Using Helm chart, demo install with default certs fails to start OpenSearch install	9	1834	July 2, 2024
I am planning to setup Opensearch master nodes in OKD as containers and data nodes as VM. Does this kind of setup supported by Opensearch? or all the nodes should be either containers or VM OpenSearch discuss , troubleshoot , configure , install	3	26	September 12, 2024
Need Support to use open search in windows OpenSearch troubleshoot , install	4	517	October 7, 2023

Official Ansible role fails to start OpenSearch

Related topics