New install of OpenSearch 3.0 won't start on cluster that ran 2.19.1 before

OpenSearch
, ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 3.0.0
Ubuntu 24.04

Describe the issue:
I have previously installed 2.19.1 on my dev cluster with a self made Ansible playbook. To prepare for 3.0.0 I did apt purge opensearch and removed the remaining relevant directories on all cluster nodes.
To install I used the same playbook, but changed the repo and package versioning to refer to 3.x and 3.0.0.
Installation runs ok, but the opensearch service won’t start.
The most relevant log entry seems to be this:

[2025-05-30T11:08:50,998][ERROR][o.o.b.Bootstrap          ] [manager1] node validation exception
[1] bootstrap checks failed
[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

I guess it’s possible that there is some leftover configuration from the previous install, but I don’t know where that might be.

I can start a node by skipping system call filters by adding

bootstrap.system_call_filter: false

to opensearch.yml, but this is undesirable, since it compromises security.

Any tips would be appreciated

Configuration:

cluster.name: "dev-cl-01"
node.name: "manager1"
node.roles: ['cluster_manager']
cluster.initial_cluster_manager_nodes: ['manager1']
discovery.seed_hosts: ['dev-dno-vm02.example.net', 'dev-dno-vm01.example.net', 'dev-mgt-vm01.example.net']
path.data: /esdata
path.logs: /var/log/opensearch
network.host: 0.0.0.0
http.port: 9200
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/certs/node.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/certs/node.key
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/certs/root_ca.pem

plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/certs/node.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/certs/node.key
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/certs/root_ca.pem

plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - 'CN=A,OU=UNIT,O=ORG,L=SOMECITY,ST=SOMECOUNTY,C=SOMECOUNTRY'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [\"all_access\", \"security_rest_api_access\"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [.plugins-ml-agent, .plugins-ml-config, .plugins-ml-connector,
  .plugins-ml-controller, .plugins-ml-model-group, .plugins-ml-model, .plugins-ml-task,
  .plugins-ml-conversation-meta, .plugins-ml-conversation-interactions, .plugins-ml-memory-meta,
  .plugins-ml-memory-message, .plugins-ml-stop-words, .opendistro-alerting-config,
  .opendistro-alerting-alert*, .opendistro-anomaly-results*, .opendistro-anomaly-detector*,
  .opendistro-anomaly-checkpoints, .opendistro-anomaly-detection-state, .opendistro-reports-*,
  .opensearch-notifications-*, .opensearch-notebooks, .opensearch-observability, .ql-datasources,
  .opendistro-asynchronous-search-response*, .replication-metadata-store, .opensearch-knn-models,
  .geospatial-ip2geo-data*, .plugins-flow-framework-config, .plugins-flow-framework-templates,
  .plugins-flow-framework-state]

plugins.security.nodes_dn: ['CN=dev-dno-vm02.example.net,OU=UNIT,O=ORG,L=SOMECITY,ST=SOMECOUNTY,C=SOMECOUNTRY', 'CN=dev-dno-vm01.example.net,OU=UNIT,O=ORG,L=SOMECITY,ST=SOMECOUNTY,C=SOMECOUNTRY', 'CN=dev-mgt-vm01.example.net,OU=UNIT,O=ORG,L=SOMECITY,ST=SOMECOUNTY,C=SOMECOUNTRY']

Also, /etc/opensearch/jvm.options.d/heap_options.yml

-Xms2g
-Xmx2g

Relevant Logs or Screenshots:


[2025-05-30T11:08:45,578][DEBUG][o.o.n.NotificationPlugin ] [manager1] notifications:createComponents
[2025-05-30T11:08:45,579][DEBUG][o.o.n.s.PluginSettings   ] [manager1] notifications:opensearch.notifications.general.operation_timeout_ms -autoUpdatedTo-> 60000
[2025-05-30T11:08:45,579][DEBUG][o.o.n.s.PluginSettings   ] [manager1] notifications:opensearch.notifications.general.default_items_query_count -autoUpdatedTo-> 100
[2025-05-30T11:08:45,620][WARN ][o.o.s.p.SQLPlugin        ] [manager1] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterke
y config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encryptin
g-credential-information
[2025-05-30T11:08:46,227][INFO ][o.o.p.i.c.s.QueryInsightsService] [manager1] Setting query insights index template priority to [1847]
[2025-05-30T11:08:46,228][DEBUG][o.o.p.i.c.s.QueryInsightsService] [manager1] Updating query insights index template priority for top queries exporter to [1847]
[2025-05-30T11:08:46,288][DEBUG][o.o.m.j.JvmService       ] [manager1] using refresh_interval [1s]
[2025-05-30T11:08:47,328][DEBUG][o.o.n.NotificationPlugin ] [manager1] notifications:getActions
[2025-05-30T11:08:47,424][DEBUG][o.o.a.ActionModule       ] [manager1] Using REST wrapper from plugin org.opensearch.security.OpenSearchSecurityPlugin
[2025-05-30T11:08:47,435][DEBUG][o.o.n.r.t.AbstractAverageUsageTracker] [manager1] updated window size: 60
[2025-05-30T11:08:47,437][DEBUG][o.o.n.r.t.AbstractAverageUsageTracker] [manager1] updated window size: 60
[2025-05-30T11:08:47,438][DEBUG][o.o.n.r.t.AbstractAverageUsageTracker] [manager1] updated window size: 24
[2025-05-30T11:08:47,465][INFO ][o.o.t.NettyAllocator     ] [manager1] creating NettyAllocator with the following configs: [name=opensearch_configured, chunk_size=256kb, suggested_max_allocation_size=256k
b, factors={opensearch.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=1mb}]
[2025-05-30T11:08:47,466][DEBUG][o.o.h.n.Netty4HttpServerTransport] [manager1] using max_chunk_size[8kb], max_header_size[16kb], max_initial_line_length[4kb], max_content_length[100mb], receive_predictor[
64kb], max_composite_buffer_components[69905], pipelining_max_events[10000]
[2025-05-30T11:08:47,474][INFO ][o.o.s.s.t.SSLConfig      ] [manager1] SSL dual mode is disabled
[2025-05-30T11:08:47,475][DEBUG][o.o.h.n.s.SecureNetty4HttpServerTransport] [manager1] Using request decompressor provider: org.opensearch.security.ssl.OpenSearchSecureSettingsFactory$2$1@649d9cc5
[2025-05-30T11:08:47,697][DEBUG][o.o.d.SettingsBasedSeedHostsProvider] [manager1] using initial hosts [dev-dno-vm02.example.net, dev-dno-vm01.example.net, dev-mgt-vm01.example.net]
[2025-05-30T11:08:47,724][INFO ][o.o.d.DiscoveryModule    ] [manager1] using discovery type [zen] and seed hosts providers [settings]
[2025-05-30T11:08:48,763][WARN ][o.o.g.DanglingIndicesState] [manager1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be manage
d manually
[2025-05-30T11:08:49,098][DEBUG][o.o.m.j.JvmService       ] [manager1] using refresh_interval [1s]
[2025-05-30T11:08:49,568][DEBUG][o.o.m.j.JvmService       ] [manager1] using refresh_interval [1s]
[2025-05-30T11:08:49,682][DEBUG][o.o.n.Node               ] [manager1] initializing HTTP handlers ...
[2025-05-30T11:08:49,958][DEBUG][o.o.s.d.r.a.RestApiPrivilegesEvaluator] [manager1] Globally disabled endpoints: {}
[2025-05-30T11:08:49,959][DEBUG][o.o.s.d.r.a.RestApiPrivilegesEvaluator] [manager1] No disabled endpoints/methods for permitted role \"security_rest_api_access\" found, allowing all
[2025-05-30T11:08:49,959][DEBUG][o.o.s.d.r.a.RestApiPrivilegesEvaluator] [manager1] No disabled endpoints/methods for permitted role \"all_access\" found, allowing all
[2025-05-30T11:08:49,989][DEBUG][o.o.s.d.r.a.RestApiPrivilegesEvaluator] [manager1] Globally disabled endpoints: {}
[2025-05-30T11:08:49,989][DEBUG][o.o.s.d.r.a.RestApiPrivilegesEvaluator] [manager1] No disabled endpoints/methods for permitted role \"security_rest_api_access\" found, allowing all
[2025-05-30T11:08:49,989][DEBUG][o.o.s.d.r.a.RestApiPrivilegesEvaluator] [manager1] No disabled endpoints/methods for permitted role \"all_access\" found, allowing all
[2025-05-30T11:08:50,029][DEBUG][o.o.s.OpenSearchSecurityPlugin] [manager1] Added 26 rest handler(s)
[2025-05-30T11:08:50,325][DEBUG][o.o.n.NotificationPlugin ] [manager1] notifications:getRestHandlers
[2025-05-30T11:08:50,357][INFO ][o.o.n.Node               ] [manager1] initialized
[2025-05-30T11:08:50,357][INFO ][o.o.n.Node               ] [manager1] starting ...
[2025-05-30T11:08:50,469][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [windows_logtype.json] log type
[2025-05-30T11:08:50,470][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [waf_logtype.json] log type
[2025-05-30T11:08:50,470][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [vpcflow_logtype.json] log type
[2025-05-30T11:08:50,471][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [test_windows_logtype.json] log type
[2025-05-30T11:08:50,472][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [s3_logtype.json] log type
[2025-05-30T11:08:50,472][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [others_web_logtype.json] log type
[2025-05-30T11:08:50,472][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [others_proxy_logtype.json] log type
[2025-05-30T11:08:50,472][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [others_macos_logtype.json] log type
[2025-05-30T11:08:50,473][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [others_compliance_logtype.json] log type
[2025-05-30T11:08:50,473][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [others_cloud_logtype.json] log type
[2025-05-30T11:08:50,473][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [others_apt_logtype.json] log type
[2025-05-30T11:08:50,474][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [others_application_logtype.json] log type
[2025-05-30T11:08:50,474][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [okta_logtype.json] log type
[2025-05-30T11:08:50,476][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [network_logtype.json] log type
[2025-05-30T11:08:50,476][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [netflow_logtype.json] log type
[2025-05-30T11:08:50,477][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [m365_logtype.json] log type
[2025-05-30T11:08:50,477][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [linux_logtype.json] log type
[2025-05-30T11:08:50,477][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [gworkspace_logtype.json] log type
[2025-05-30T11:08:50,478][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [github_logtype.json] log type
[2025-05-30T11:08:50,479][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [dns_logtype.json] log type
[2025-05-30T11:08:50,480][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [cloudtrail_logtype.json] log type
[2025-05-30T11:08:50,480][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [azure_logtype.json] log type
[2025-05-30T11:08:50,481][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [apache_access_logtype.json] log type
[2025-05-30T11:08:50,481][INFO ][o.o.s.l.BuiltinLogTypeLoader] [manager1] Loaded [ad_ldap_logtype.json] log type
[2025-05-30T11:08:50,503][DEBUG][i.n.c.MultithreadEventLoopGroup] [manager1] -Dio.netty.eventLoopThreads: 4
[2025-05-30T11:08:50,519][DEBUG][i.n.u.c.GlobalEventExecutor] [manager1] -Dio.netty.globalEventExecutor.quietPeriodSeconds: 1
[2025-05-30T11:08:50,539][DEBUG][i.n.c.n.NioEventLoop     ] [manager1] -Dio.netty.noKeySetOptimization: true
[2025-05-30T11:08:50,539][DEBUG][i.n.c.n.NioEventLoop     ] [manager1] -Dio.netty.selectorAutoRebuildThreshold: 512
[2025-05-30T11:08:50,563][DEBUG][i.n.u.i.PlatformDependent] [manager1] org.jctools-core.MpscChunkedArrayQueue: unavailable
[2025-05-30T11:08:50,592][DEBUG][o.o.t.n.Netty4Transport  ] [manager1] using profile[default], worker_count[2], port[9300-9400], bind_host[[10.64.129.6]], publish_host[[]], receive_predictor[64kb->64kb]
[2025-05-30T11:08:50,606][DEBUG][o.o.t.TcpTransport       ] [manager1] binding server bootstrap to: [10.64.129.6]
[2025-05-30T11:08:50,620][DEBUG][i.n.c.DefaultChannelId   ] [manager1] -Dio.netty.processId: 32342 (auto-detected)
[2025-05-30T11:08:50,622][DEBUG][i.n.u.NetUtil            ] [manager1] -Djava.net.preferIPv4Stack: false
[2025-05-30T11:08:50,622][DEBUG][i.n.u.NetUtil            ] [manager1] -Djava.net.preferIPv6Addresses: false
[2025-05-30T11:08:50,624][DEBUG][i.n.u.NetUtilInitializations] [manager1] Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
[2025-05-30T11:08:50,625][DEBUG][i.n.u.NetUtil            ] [manager1] /proc/sys/net/core/somaxconn: 4096
[2025-05-30T11:08:50,626][DEBUG][i.n.c.DefaultChannelId   ] [manager1] -Dio.netty.machineId: 00:22:48:ff:fe:de:45:c8 (auto-detected)
[2025-05-30T11:08:50,657][DEBUG][i.n.b.ChannelInitializerExtensions] [manager1] -Dio.netty.bootstrap.extensions: null
[2025-05-30T11:08:50,674][DEBUG][o.o.t.TcpTransport       ] [manager1] Bound profile [default] to address {10.64.129.6:9300}
[2025-05-30T11:08:50,675][INFO ][o.o.t.TransportService   ] [manager1] publish_address {10.64.129.6:9300}, bound_addresses {10.64.129.6:9300}
[2025-05-30T11:08:50,950][DEBUG][o.o.g.PersistedClusterStateService] [manager1] writing cluster state took [200ms]; wrote full state with [4] indices
[2025-05-30T11:08:50,991][INFO ][o.o.b.BootstrapChecks    ] [manager1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2025-05-30T11:08:50,990][DEBUG][o.o.n.r.t.AverageCpuUsageTracker] [manager1] Recording cpu usage: 89%
[2025-05-30T11:08:50,993][DEBUG][o.o.n.r.t.AverageMemoryUsageTracker] [manager1] Recording memory usage: 12%
[2025-05-30T11:08:50,998][ERROR][o.o.b.Bootstrap          ] [manager1] node validation exception
[1] bootstrap checks failed
[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2025-05-30T11:08:51,000][INFO ][o.o.n.Node               ] [manager1] stopping ...
[2025-05-30T11:08:51,005][INFO ][o.o.s.a.r.AuditMessageRouter] [manager1] Closing AuditMessageRouter
[2025-05-30T11:08:51,005][INFO ][o.o.s.a.s.SinkProvider   ] [manager1] Closing InternalOpenSearchSink
[2025-05-30T11:08:51,005][INFO ][o.o.s.a.s.SinkProvider   ] [manager1] Closing DebugSink
[2025-05-30T11:08:51,019][INFO ][o.o.n.Node               ] [manager1] stopped
[2025-05-30T11:08:51,019][INFO ][o.o.n.Node               ] [manager1] closing ...
[2025-05-30T11:08:51,027][DEBUG][o.o.p.i.c.e.LocalIndexExporter] [manager1] Closing the LocalIndexExporter..
[2025-05-30T11:08:51,027][DEBUG][o.o.p.i.c.r.LocalIndexReader] [manager1] Closing the LocalIndexReader..
[2025-05-30T11:08:51,030][INFO ][o.o.s.a.i.AuditLogImpl   ] [manager1] Closing AuditLogImpl
[2025-05-30T11:08:51,031][DEBUG][o.o.s.a.i.AuditLogImpl   ] [manager1] Fail to unregister shutdown hook Thread[#30,Thread-4,5,]. Shutdown is in progress.
java.lang.IllegalStateException: Shutdown in progress
        at java.base/java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:83) ~[?:?]
        at java.base/java.lang.Runtime.removeShutdownHook(Runtime.java:281) ~[?:?]
        at org.opensearch.security.auditlog.impl.AuditLogImpl.removeShutdownHook(AuditLogImpl.java:103) ~[opensearch-security-3.0.0.0.jar:3.0.0.0]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
        at org.opensearch.security.auditlog.impl.AuditLogImpl.close(AuditLogImpl.java:114) [opensearch-security-3.0.0.0.jar:3.0.0.0]
        at org.opensearch.security.OpenSearchSecurityPlugin.close(OpenSearchSecurityPlugin.java:296) [opensearch-security-3.0.0.0.jar:3.0.0.0]
        at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) [opensearch-common-3.0.0.jar:3.0.0]
        at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) [opensearch-common-3.0.0.jar:3.0.0]
        at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114) [opensearch-common-3.0.0.jar:3.0.0]
        at org.opensearch.node.Node.close(Node.java:1965) [opensearch-3.0.0.jar:3.0.0]
        at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89) [opensearch-common-3.0.0.jar:3.0.0]
        at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131) [opensearch-common-3.0.0.jar:3.0.0]
        at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:81) [opensearch-common-3.0.0.jar:3.0.0]
        at org.opensearch.bootstrap.Bootstrap$4.run(Bootstrap.java:207) [opensearch-3.0.0.jar:3.0.0]
[2025-05-30T11:08:51,035][INFO ][o.o.n.Node               ] [manager1] closed
2

I installed 2.19.1 again, with the playbook, then tried to upgrade one node manually, using the 3.x repo and installing version 3.0.0
During the upgrade I accepted a new jvm.options file but kept my original opensearch.yml
(re)starting the service fails again with the same log messages as before

If I add inn

bootstrap.system_call_filter: false

to opensearch.yml, the service starts and the node joins the old cluster just fine

3

@mejrot Could you try setting the below option in opensearch.yml or as env variable?

bootstrap.memory_lock=true
4

Hi @pablo, and thanks for replying

I set the parameter in opensearch.yml,

bootstrap.memory_lock: true

but now I have 2 bootstrap checks failing:

[2025-05-31T09:14:30,291][DEBUG][o.o.t.TcpTransport       ] [data1] Bound profile [default] to address {10.64.129.4:9300}
[2025-05-31T09:14:30,292][INFO ][o.o.t.TransportService   ] [data1] publish_address {10.64.129.4:9300}, bound_addresses {10.64.129.4:9300}
[2025-05-31T09:14:30,564][DEBUG][o.o.g.PersistedClusterStateService] [data1] writing cluster state took [201ms]; wrote full state with [8] indices
[2025-05-31T09:14:30,595][INFO ][o.o.b.BootstrapChecks    ] [data1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2025-05-31T09:14:30,601][ERROR][o.o.b.Bootstrap          ] [data1] node validation exception
[2] bootstrap checks failed
[1]: memory locking requested for opensearch process but memory is not locked
[2]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
5

@mejrot I get the same error when memlock is not set properly.

This is my docker-compose configuration. The bootstrap memory lock must be set in conjunction with an unlimited value of ulimit memlock.

  opensearch-node1:
    image: opensearchproject/opensearch:${OS_VER}
    container_name: opensearch-node1_${OS_VER}
    environment:
      - cluster.name=opensearch-cluster1
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "OPENSEARCH_INITIAL_ADMIN_PASSWORD=Eliatra123"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536

If you run OpenSearch as a service, try setting the memlock unlimited value in /etc/security/limits.conf

This is an example config when OpenSearch is started with the opensearch user.

opensearch   soft   nofile   65536
opensearch   hard   nofile   65536
opensearch   soft   memlock  unlimited
opensearch   hard   memlock  unlimited
6

That didn’t seem to help, but I found something else.

When I run

sudo -u opensearch bash -c 'source /etc/default/opensearch && exec strace -f -e trace=seccomp /usr/share/opensearch/bin/opensearch'

strace shows this error before exiting:

/usr/share/opensearch/jdk/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
[pid  8480] +++ exited with 127 +++

The error doesn’t show up on the node running opensearch 2.19.1
It’s strange, because the library exists:

# ls -l /usr/share/opensearch/jdk/lib/libjli.so 
-rw-r--r-- 1 root root 104616 Oct 14  2022 /usr/share/opensearch/jdk/lib/libjli.so

# file /usr/share/opensearch/jdk/lib/libjli.so
/usr/share/opensearch/jdk/lib/libjli.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped

I tried copying the java bin bundled with 2.19.1 to /usr/share/opensearch/jdk/bin/java, and now the process continues, with a lot more output from strace. The systemd restart still fails, though it might be a different problem since I just dropped in a different java binary, but to me it seems there might be a problem with the java installation. At least on my system.

7

Try this : OpenSearch 3.0.0 fails to start - #6 by senthilengg

8

Thanks, but I already looked at that post. I’m using the jvm.options file that came with the 3.0.0 upgrade. I had a jvm.options.d/heap_options file to override heap memory settings, but I’ve tried without that as well

9

I am having the same issue with a bare metal install of 3.0.0. Using RPM installer
opensearch starts fine in single-node but cant get past boot-check.
“[2025-06-17T09:53:02,618][INFO ][o.o.s.l.BuiltinLogTypeLoader] [mccoy01] Loaded [ad_ldap_logtype.json] log type
[2025-06-17T09:53:02,722][INFO ][o.o.t.TransportService ] [mccoy01] publish_address {136.167.4.136:9300}, bound_addresses {136.167.4.136:9300}
[2025-06-17T09:53:02,838][INFO ][o.o.b.BootstrapChecks ] [mccoy01] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2025-06-17T09:53:02,842][ERROR][o.o.b.Bootstrap ] [mccoy01] node validation exception
[2] bootstrap checks failed
[1]: memory locking requested for opensearch process but memory is not locked
[2]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk”

10

upgrading from 2.19.1 to 3.1.0 with dnf install --best, opensearch fails with error java.lang.ClassNotFoundException

Fix: in my case, there was a jvm.options.rpmnew from the 3.1.0 install, replaced the 2.19.1 version with this one and it worked.