ava.lang.NullPointerException: Cannot invoke \"Object.getClass()\" because \"event\" is null

dbgtmaster · August 14, 2024, 10:21am

Hi,

i am using Opensearch 2.16 (lastet).

On all my clusters and all nodes, i get the following message every few seconds:

{"type": "server", "timestamp": "2024-08-14T10:10:53,763Z", "level": "ERROR", "component": "o.o.s.c.ConfigurationRepository", "cluster.name": "--clustername--", "node.name": "--nodename--", "message": "org.opensearch.security.securityconf.DynamicConfigFactory@57058a20 listener errored: java.lang.NullPointerException: Cannot invoke \"Object.getClass()\" because \"event\" is null", "cluster.uuid": "K4W6fQAdTC-Mtstu1VkECg", "node.id": "-38fMORoQhu7L0IUrIpP8Q" ,
"stacktrace": ["java.lang.NullPointerException: Cannot invoke \"Object.getClass()\" because \"event\" is null",
"at org.greenrobot.eventbus.EventBus.postSingleEvent(EventBus.java:387) ~[eventbus-java-3.3.1.jar:?]",
"at org.greenrobot.eventbus.EventBus.post(EventBus.java:275) ~[eventbus-java-3.3.1.jar:?]",
"at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:326) ~[opensearch-security-2.16.0.0.jar:2.16.0.0]",
"at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:570) [opensearch-security-2.16.0.0.jar:2.16.0.0]",
"at org.opensearch.security.configuration.ConfigurationRepository.notifyConfigurationListeners(ConfigurationRepository.java:559) [opensearch-security-2.16.0.0.jar:2.16.0.0]",
"at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:554) [opensearch-security-2.16.0.0.jar:2.16.0.0]",
"at org.opensearch.security.configuration.ConfigurationRepository.loadConfigurationWithLock(ConfigurationRepository.java:538) [opensearch-security-2.16.0.0.jar:2.16.0.0]",
"at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:531) [opensearch-security-2.16.0.0.jar:2.16.0.0]",
"at org.opensearch.security.configuration.ConfigurationRepository.initalizeClusterConfiguration(ConfigurationRepository.java:284) [opensearch-security-2.16.0.0.jar:2.16.0.0]",
"at org.opensearch.security.configuration.ConfigurationRepository.lambda$initOnNodeStart$10(ConfigurationRepository.java:439) [opensearch-security-2.16.0.0.jar:2.16.0.0]",
"at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]"] }

What does the error mean? How can i fix it?

Mantas · August 14, 2024, 10:27am

Hi @dbgtmaster,

Do you mind sharing your opensearch.yml content??

Thanks,
mj

dbgtmaster · August 14, 2024, 10:32am

Output of securityadmin.sh (looks fine):

$ sudo /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cd /etc/opensearch/opensearch-security -cert $ROOT_CERT -key $ROOT_KEY -cacert $CACERT -cn --clustername--
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to localhost:9200 ... done
Connected as "CN=securityadmin"
OpenSearch Version: 2.16.0
Contacting opensearch cluster '--clustername--' and wait for YELLOW clusterstate ...
Clustername: --clustername--
Clusterstate: GREEN
Number of nodes: 3
Number of data nodes: 3
.opendistro_security index already exists, so we do not need to create one.
Populate config from /etc/opensearch/opensearch-security/
Will update '/config' with /etc/opensearch/opensearch-security/config.yml
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/opensearch/opensearch-security/roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/opensearch/opensearch-security/roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/opensearch/opensearch-security/internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/opensearch/opensearch-security/action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/opensearch/opensearch-security/tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/opensearch/opensearch-security/nodes_dn.yml
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/opensearch/opensearch-security/whitelist.yml
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/opensearch/opensearch-security/audit.yml
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/opensearch/opensearch-security/allowlist.yml
   SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"]) due to: null
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"]) due to: null
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"]) due to: null
Done with success

my opensearch config:

---
action.destructive_requires_name: true
bootstrap.memory_lock: true
cluster.indices.replication.strategy: SEGMENT
cluster.max_shards_per_node: 50000
cluster.name: --clustername--
cluster.routing.allocation.awareness.attributes: location
cluster.routing.allocation.cluster_concurrent_rebalance: 4
cluster.routing.allocation.disk.watermark.flood_stage: 99%
cluster.routing.allocation.disk.watermark.high: 97%
cluster.routing.allocation.disk.watermark.low: 93%
cluster.routing.allocation.node_concurrent_incoming_recoveries: 20
cluster.routing.allocation.node_concurrent_outgoing_recoveries: 20
cluster.routing.allocation.node_initial_primaries_recoveries: 40
cluster.routing.rebalance.enable: all
discovery.seed_hosts:
- host1
- host2
- host3
network.host: 0.0.0.0
node.attr.location: xyz
node.name: nodename
path.data: "/var/lib/opensearch"
path.logs: "/var/log/opensearch"
plugins.replication.autofollow.fetch_poll_interval: 30s
plugins.replication.autofollow.retry_poll_interval: 60s
plugins.security.audit.type: internal_opensearch
plugins.security.authcz.admin_dn:
- CN=securityadmin
plugins.security.nodes_dn:
- CN=opensearch
- CN=opensearch-xxxxx
- CN=opensearch-yyyyy
plugins.security.restapi.endpoints_disabled.security_rest_api_access.ACTIONGROUPS:
- PUT
- POST
- DELETE
- PATCH
plugins.security.restapi.endpoints_disabled.security_rest_api_access.CACHE:
- PUT
- POST
- DELETE
- PATCH
plugins.security.restapi.endpoints_disabled.security_rest_api_access.CONFIG:
- PUT
- POST
- DELETE
- PATCH
plugins.security.restapi.endpoints_disabled.security_rest_api_access.INTERNALUSERS:
- PUT
- POST
- DELETE
- PATCH
plugins.security.restapi.endpoints_disabled.security_rest_api_access.ROLES:
- PUT
- POST
- DELETE
- PATCH
plugins.security.restapi.endpoints_disabled.security_rest_api_access.ROLESMAPPING:
- PUT
- POST
- DELETE
- PATCH
plugins.security.restapi.endpoints_disabled.security_rest_api_access.SYSTEMINFO:
- PUT
- POST
- DELETE
- PATCH
plugins.security.restapi.roles_enabled:
- security_rest_api_access
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: "/secrets/opensearch.pem"
plugins.security.ssl.http.pemkey_filepath: "/secrets/opensearch.pkcs8"
plugins.security.ssl.http.pemtrustedcas_filepath: "/secrets/opensearch-ca_full.crt"
plugins.security.ssl.transport.pemcert_filepath: "/secrets/opensearch.pem"
plugins.security.ssl.transport.pemkey_filepath: "/secrets/opensearch.pkcs8"
plugins.security.ssl.transport.pemtrustedcas_filepath: "/secrets/opensearch-ca_full.crt"
plugins.security.ssl_cert_reload_enabled: true
plugins.security.system_indices.enabled: true
s3.client.default.endpoint: minio-xyz:9000
s3.client.default.path_style_access: 'true'
transport.bind_host: 0.0.0.0
transport.publish_host: transportname-xyz

Mantas · August 14, 2024, 12:28pm

@dbgtmaster, Is this a fresh OS 2.16 install or have you upgraded your cluster from previous versions? If upgraded what was the original version?

Thanks,
mj

dbgtmaster · August 14, 2024, 12:33pm

The original version was 2.12 or 2.13.
I have about 15 clusters with different installed versions, and all clusters have this error.

Mantas · August 14, 2024, 1:12pm

@dbgtmaster, How do you deploy your Clusters?

Best,
mj

dbgtmaster · August 15, 2024, 6:42pm

@Mantas i am using debian 11.10, deployed with the opensearch debian packages. (*.deb).

merlinz01 · August 15, 2024, 8:18pm

I’m getting exactly the same error, fresh install of OpenSearch 2.16 using debian packages on a three-node cluster with Debian 12.

merlinz01 · August 15, 2024, 8:29pm

I ran into this exact error for 2.13. See below for a temporary solution.

github.com/opensearch-project/security

[BUG] Cannot invoke "Object.getClass()" because "event" is null

opened 01:08AM - 04 May 24 UTC

closed 01:31AM - 04 May 24 UTC

merlinz01

bug untriaged

**What is the bug?** I ran securityadmin.sh with an empty audit.yml file and no…w I'm getting null-pointer exceptions. **How can one reproduce the bug?** Retrieve currrent security config with securityadmin.sh. Rename all .yml filenames with dates to their normal names. Edit an unrelated setting in config.yml. Run securityadmin.sh to update the config. Observe this output from securityadmin.sh: ``` ERR: Seems ./audit.yml is not in OpenSearch Security 7 format: java.lang.NullPointerException: Cannot invoke "org.opensearch.security.securityconf.impl.SecurityDynamicConfiguration.get_meta()" because "sdc" is null ``` Edit audit.yml and observe that it is empty. Add these lines: ``` _meta: config_version: 2 type: "audit" ``` Run securityadmin.sh again and observe this output: ``` Will update '/audit' with ./audit.yml SUCC: Configuration for 'audit' created or updated ... FAIL: 1 nodes reported failures. Failure is /{"_nodes":{"total":1,"successful":0,"failed":1,"failures":[{"type":"failed_node_exception","reason":"Failed node [aAs3rkCcQ72TDfBPZ3rfyw]","node_id":"aAs3rkCcQ72TDfBPZ3rfyw","caused_by":{"type":"exception","reason":"java.lang.NullPointerException: Cannot invoke \"Object.getClass()\" because \"event\" is null","caused_by":{"type":"null_pointer_exception","reason":"Cannot invoke \"Object.getClass()\" because \"event\" is null"}}}]},"cluster_name":"prism-logs-cluster","configupdate_response":{"nodes":{},"node_size":0,"has_failures":true,"failures_size":1}} FAIL: Expected 1 nodes to return response, but got 0 Done with failures ``` Remove audit.yml and run securityadmin.sh again and observe the same output. See lots and lots of these in Opensearch logs: ``` [2024-05-04T01:04:12,908][INFO ][stdout ] [main-node] [FINE] No subscribers registered for event class org.opensearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl [2024-05-04T01:04:12,910][INFO ][stdout ] [main-node] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent [2024-05-04T01:04:12,912][ERROR][o.o.s.c.ConfigurationRepository] [main-node] org.opensearch.security.securityconf.DynamicConfigFactory@7a824730 listener errored: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "event" is null java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "event" is null at org.greenrobot.eventbus.EventBus.postSingleEvent(EventBus.java:387) ~[eventbus-java-3.3.1.jar:?] at org.greenrobot.eventbus.EventBus.post(EventBus.java:275) ~[eventbus-java-3.3.1.jar:?] at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:323) ~[opensearch-security-2.13.0.0.jar:2.13.0.0] at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:430) [opensearch-security-2.13.0.0.jar:2.13.0.0] at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:419) [opensearch-security-2.13.0.0.jar:2.13.0.0] at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:402) [opensearch-security-2.13.0.0.jar:2.13.0.0] at org.opensearch.security.configuration.ConfigurationRepository.initalizeClusterConfiguration(ConfigurationRepository.java:227) [opensearch-security-2.13.0.0.jar:2.13.0.0] at org.opensearch.security.configuration.ConfigurationRepository.lambda$initOnNodeStart$0(ConfigurationRepository.java:318) [opensearch-security-2.13.0.0.jar:2.13.0.0] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] ``` **What is the expected behavior?** No errors **What is your host/environment?** - OS: Debian 12 - Version 2.13.0 - Plugins ``` opensearch-alerting opensearch-anomaly-detection opensearch-asynchronous-search opensearch-cross-cluster-replication opensearch-custom-codecs opensearch-flow-framework opensearch-geospatial opensearch-index-management opensearch-job-scheduler opensearch-knn opensearch-ml opensearch-neural-search opensearch-notifications opensearch-notifications-core opensearch-observability opensearch-reports-scheduler opensearch-security opensearch-security-analytics opensearch-skills opensearch-sql ``` **Do you have any screenshots?** If applicable, add screenshots to help explain your problem. **Do you have any additional context?** Add any other context about the problem.

Mantas · August 16, 2024, 11:39am

@merlinz01, @dbgtmaster

Did adding the below to audit.yml stop the errors from appearing for you?

config:
  enabled: false

thanks,
mj

merlinz01 · August 16, 2024, 11:42am

Yes. The error is caused when that entry is missing; you can also set it to true.
I’m currently in the process of opening a PR to handle the error.

dbgtmaster · August 16, 2024, 2:50pm

Today, i only tested with “enabled: false”, which stop the errors.
On monday, i will try it with “enabled: true”.

Regards,
Thomas

merlinz01 · August 16, 2024, 3:00pm

github.com/opensearch-project/security

Handle the audit config being null

opensearch-project:main ← merlinz01:handle-null-audit-error

opened 11:45AM - 16 Aug 24 UTC

merlinz01

+2 -1

This was causing an error if the audit config was null and was posted to the eve…nt bus. ### Description * Category: Bugfix * Why these changes are required? An empty audit config causes the logs to be filled with an irrelevant error. * What is the old behavior before changes and new behavior after changes? Old behavior: Lots of errors when the audit config is missing New behavior: Audit config is set to the default when missing ### Issues Resolved #4315 https://forum.opensearch.org/t/ava-lang-nullpointerexception-cannot-invoke-object-getclass-because-event-is-null/20821/ Is this a backport? No. Do these changes introduce new permission(s) to be displayed in the static dropdown on the front-end? No. ### Testing None ### Check List - [ ] New functionality includes testing - [ ] New functionality has been documented - [ ] New Roles/Permissions have a corresponding security dashboards plugin PR - [ ] API changes companion pull request [created](https://github.com/opensearch-project/opensearch-api-specification/blob/main/DEVELOPER_GUIDE.md) - [x] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/security/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

Topic		Replies	Views
Exception java.lang.NullPointerException Security	5	1811	November 17, 2020
[security_analytics_exception] class java.lang.String cannot be cast to class java.util.Map (java.lang.String and java.util.Map are in module java.base of loader 'bootstrap') Security Analytics	1	182	June 15, 2024
I am getting the error while starting the opensearch using version 2.17.1 with jdk 11 OpenSearch	4	74	October 31, 2024
Searching does not work in one of the index OpenSearch Dashboards troubleshoot	7	4306	May 24, 2023
Grouping aggregated results with multi_term leads to NullPointerException General Feedback troubleshoot	2	532	August 29, 2023

ava.lang.NullPointerException: Cannot invoke \"Object.getClass()\" because \"event\" is null

Related topics