Official Ansible role fails to start OpenSearch

Hi all!

I’m using the official Ansible playbook/role to deploy an OpenSearch cluster on newly created VMs which I destroy and recreate for each deployment attempt to make sure that no remains interfere in any way.

I have observed some issues so far:

The task:

TASK [opensearch : Security Plugin configuration | Copy the node & admin certificates to opensearch nodes]

…sometimes copies the certificates, sometimes it does not. I had to comment the "when: " statement to force the copy…

More important: OpenSearch fails to start and throws a Java exception:

Jun 10 14:21:16 os1 opensearch[4461]: uncaught exception in thread [main]
Jun 10 14:21:16 os1 opensearch[4461]: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Jun 10 14:21:16 os1 opensearch[4461]: Likely root cause: OpenSearchException[plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.]
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:422)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:179)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:218)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:255)
Jun 10 14:21:16 os1 opensearch[4461]: at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Jun 10 14:21:16 os1 opensearch[4461]: at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
Jun 10 14:21:16 os1 opensearch[4461]: at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Jun 10 14:21:16 os1 opensearch[4461]: at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
Jun 10 14:21:16 os1 opensearch[4461]: at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.plugins.PluginsService.(PluginsService.java:195)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.node.Node.(Node.java:413)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.node.Node.(Node.java:336)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:244)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:244)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:414)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.cli.Command.main(Command.java:101)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Jun 10 14:21:16 os1 opensearch[4461]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Jun 10 14:21:16 os1 opensearch[4461]: For complete error details, refer to the log at /usr/share/opensearch/logs/os-cluster.log
Jun 10 14:21:16 os1 systemd[1]: opensearch.service: Main process exited, code=exited, status=1/FAILURE
Jun 10 14:21:16 os1 systemd[1]: opensearch.service: Failed with result ‘exit-code’.

I have seen these 2 other entries but none of the solutions seem to apply:

OpenSearch ssl exception (tar) ← aren’t certs automatically generated?

Tarball install security error ← this should be handled by the playbook as well, or?

Any ideas as to why this is happening?

Thanks in advance!

Best

I have now deployed 2 CentOS7 VMs because of the banner shown here Ansible playbook - OpenSearch documentation

The Ansible playbook only supports deployment of OpenSearch and OpenSearch Dashboards to CentOS7 hosts.

Unfortunately, I get the same results:

Jun 10 15:48:19 localhost systemd: Started opensearch.
Jun 10 15:48:23 localhost opensearch: WARNING: A terminally deprecated method in java.lang.System has been called
Jun 10 15:48:23 localhost opensearch: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.0.0.jar)
Jun 10 15:48:23 localhost opensearch: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Jun 10 15:48:23 localhost opensearch: WARNING: System::setSecurityManager will be removed in a future release
Jun 10 15:48:24 localhost ansible-ansible.legacy.command: Invoked with creates=None executable=None _uses_shell=False strip_empty_ends=True _raw_params=sed -i ‘/cluster.initial_master_nodes
/d’ “/usr/share/opensearch/config/opensearch.yml” removes=None argv=None warn=False chdir=None stdin_add_newline=True stdin=None
Jun 10 15:48:25 localhost ansible-ansible.legacy.systemd: Invoked with no_block=False force=None name=opensearch daemon_reexec=False enabled=True daemon_reload=False state=started masked=No
ne scope=system
Jun 10 15:48:26 localhost ansible-ansible.legacy.command: Invoked with creates=None executable=None _uses_shell=False strip_empty_ends=True _raw_params=/usr/share/opensearch/bin/opensearch-
plugin list removes=None argv=None warn=False chdir=None stdin_add_newline=True stdin=None
Jun 10 15:48:29 localhost opensearch: WARNING: A terminally deprecated method in java.lang.System has been called
Jun 10 15:48:29 localhost opensearch: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.0.0.jar)
Jun 10 15:48:29 localhost opensearch: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Jun 10 15:48:29 localhost opensearch: WARNING: System::setSecurityManager will be removed in a future release
Jun 10 15:48:31 localhost ansible-wait_for: Invoked with active_connection_states=[‘ESTABLISHED’, ‘FIN_WAIT1’, ‘FIN_WAIT2’, ‘SYN_RECV’, ‘SYN_SENT’, ‘TIME_WAIT’] state=started connect_timeou
t=1 delay=5 search_regex=None host=192.168.56.40 sleep=1 timeout=300 exclude_hosts=None msg=None path=None port=9200
Jun 10 15:48:32 localhost opensearch: uncaught exception in thread [main]
Jun 10 15:48:32 localhost opensearch: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Jun 10 15:48:32 localhost opensearch: Likely root cause: OpenSearchException[plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and p
lugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.]
Jun 10 15:48:32 localhost opensearch: at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:422)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:179)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:218)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:255)
Jun 10 15:48:32 localhost opensearch: at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Jun 10 15:48:32 localhost opensearch: at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
Jun 10 15:48:32 localhost opensearch: at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Jun 10 15:48:32 localhost opensearch: at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
Jun 10 15:48:32 localhost opensearch: at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.plugins.PluginsService.(PluginsService.java:195)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.node.Node.(Node.java:413)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.node.Node.(Node.java:336)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:244)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:244)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:414)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.cli.Command.main(Command.java:101)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Jun 10 15:48:32 localhost opensearch: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Jun 10 15:48:32 localhost opensearch: For complete error details, refer to the log at /usr/share/opensearch/logs/btcs-cluster.log
Jun 10 15:48:32 localhost systemd: opensearch.service: main process exited, code=exited, status=1/FAILURE
Jun 10 15:48:32 localhost systemd: Unit opensearch.service entered failed state.
Jun 10 15:48:32 localhost systemd: opensearch.service failed.

Your assistance is much appreciated.

Have a good weekend!

Hello again!

I tried the manual installation path described here: Tarball - OpenSearch documentation

Same errors:

root@os1:~/opensearch-2.0.0# ./opensearch-tar-install.sh


** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755 **


OpenSearch Security Demo Installer
** Warning: Do not use on production or public reachable systems **
Basedir: /root/opensearch-2.0.0
OpenSearch install type: .tar.gz on DISTRIB_ID=Ubuntu
OpenSearch config dir: /root/opensearch-2.0.0/config
OpenSearch config file: /root/opensearch-2.0.0/config/opensearch.yml
OpenSearch bin dir: /root/opensearch-2.0.0/bin
OpenSearch plugins dir: /root/opensearch-2.0.0/plugins
OpenSearch lib dir: /root/opensearch-2.0.0/lib
Detected OpenSearch Version: x-content-2.0.0
Detected OpenSearch Security Version: 2.0.0.0
/root/opensearch-2.0.0/config/opensearch.yml seems to be already configured for Security. Quit.
done security
done plugins
k-NN libraries not found in LD_LIBRARY_PATH. Updating path to: :/root/opensearch-2.0.0/plugins/opensearch-knn/lib.
Starting OpenSearch
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/root/opensearch-2.0.0/lib/opensearch-2.0.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
[2022-06-13T09:32:10,581][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [os1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.RuntimeException: can not run opensearch as root
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.0.0.jar:2.0.0]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.0.0.jar:2.0.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.0.0.jar:2.0.0]
Caused by: java.lang.RuntimeException: can not run opensearch as root
at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:126) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:193) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:414) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.0.0.jar:2.0.0]
… 6 more
uncaught exception in thread [main]
java.lang.RuntimeException: can not run opensearch as root
at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:126)
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:193)
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:414)
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
For complete error details, refer to the log at /root/opensearch-2.0.0/logs/opensearch.log
root@os1:~/opensearch-2.0.0#

Ideas?

Thanks!

The Java exception occurs at this point:

TASK [linux/opensearch : Make sure opensearch is started] ********************************************************************************************************************************************
ok: [os1]
ok: [os2]

TASK [linux/opensearch : Get all the installed ES plugins] *******************************************************************************************************************************************
changed: [os1]
changed: [os2]

TASK [linux/opensearch : Show all the installed ES plugins] ******************************************************************************************************************************************
ok: [os1] => {
“msg”: “opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-ml\nopensearch-notifications\nopensearch-notifications-core\nopensearch-observability\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql”
}
ok: [os2] => {
“msg”: “opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-ml\nopensearch-notifications\nopensearch-notifications-core\nopensearch-observability\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql”
}

TASK [linux/opensearch : Wait for opensearch to startup] *********************************************************************************************************************************************

Your assistance with this is much appreciated!

TIA!

Replying to myself once again :slight_smile:

I solved the issue by:
a) using CentOS 7.9 as the operating system in the VMs
b) replacing the original installation method from tarball and using the RPM instead.

Original code snippet from tasks/opensearch.yml:

- name: OpenSearch Install | Download opensearch {{ os_version }}
  get_url:
    url: "{{ os_download_url }}/{{ os_version }}/opensearch-{{ os_version }}-linux-x64.tar.gz"
    dest: "/tmp/opensearch.tar.gz"
  register: download

- name: OpenSearch Install | Create opensearch user
  user:
    name: "{{ os_user }}"
    state: present
    shell: /bin/bash
  when: download.changed

- name: OpenSearch Install | Create home directory
  file:
    path: "{{ os_home }}"
    state: directory
    owner: "{{ os_user }}"
    group: "{{ os_user }}"
  when: download.changed

- name: OpenSearch Install | Extract the tar file
  command: chdir=/tmp/ tar -xvzf opensearch.tar.gz -C "{{ os_home }}" --strip-components=1
  when: download.changed

Replaced with:

- name: OpenSearch Install | Import PGP key
  rpm_key:
    state: present
    key: https://artifacts.opensearch.org/publickeys/opensearch.pgp

- name: OpenSearch Install | Add OpenSearch 2.x repository
  get_url:
    url: https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/opensearch-2.x.repo
    dest: /etc/yum.repos.d/opensearch-2.x.repo
    owner: root
    group: root
    mode: 0644

- name: OpenSearch Install | Add OpenSearch Dashboards 2.x repository
  get_url:
    url: https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/2.x/opensearch-dashboards-2.x.repo
    dest: /etc/yum.repos.d/opensearch-dashboards-2.x.repo
    owner: root
    group: root
    mode: 0644

- name: OpenSearch Install | Install packages
  package:
    name:
      - opensearch
      - opensearch-dashboards
      - firewalld
    state: latest

- name: OpenSearch Install | Start firewalld
  systemd:
    name: firewalld
    daemon_reload: yes
    state: started

OpenSearch starts as expected.