Invalid Key and role issues

landon_lslc · March 6, 2025, 5:12pm

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch: 2.19.1

Describe the issue:
I’ve been getting these errors in OpenSearch’s logs for a while now. They appear constantly when any nodes are restarting, and appear periodically otherwise. The frequency seems to correlate to any time any indices are yellow.
I’m not sure what these errors mean though, or how to fix them. Does anyone have any ideas?

It seems to cycle through all roles, with no seeming consistency or logic around which role it throws these alerts for.

In terms of the index it also throws alerts for, the index seems to consistantly be the write index for the log_win_microsoft-windows-security-auditing data stream. As that index rolls over, it throws alerts for the newest one.

Configuration:
Settings of that index:

{
  ".ds-log_win_microsoft-windows-security-auditing-000481": {
    "settings": {
      "index": {
        "replication": {
          "type": "DOCUMENT"
        },
        "routing": {
          "allocation": {
            "include": {
              "temp": "warm,hot"
            }
          }
        },
        "hidden": "true",
        "number_of_shards": "1",
        "provided_name": ".ds-log_win_microsoft-windows-security-auditing-000481",
        "creation_date": "1741262516421",
        "number_of_replicas": "1",
        "uuid": "G1VHIxvNQdKBzSIZ2mQtqg",
        "version": {
          "created": "136407927"
        }
      }
    }
  }
}

Relevant Logs or Screenshots:

[2025-03-06T08:03:17,041][ERROR][o.o.s.p.ActionPrivileges ] [osc01n02] Unexpected exception while processing role: USR_INFORMATION_TECHNOLOGY_DATABASE
Ignoring role.
java.lang.IllegalArgumentException: Invalid key .ds-log_win_microsoft-windows-security-auditing-000481; not present in keySuperSet
	at com.selectivem.collections.CompactMapGroupBuilder$MapBuilder.get(CompactMapGroupBuilder.java:139) ~[special-collections-complete-1.4.0.jar:?]
	at org.opensearch.security.privileges.ActionPrivileges$StatefulIndexPrivileges.<init>(ActionPrivileges.java:1029) [opensearch-security-2.19.1.0.jar:2.19.1.0]
	at org.opensearch.security.privileges.ActionPrivileges.updateStatefulIndexPrivileges(ActionPrivileges.java:247) [opensearch-security-2.19.1.0.jar:2.19.1.0]
	at org.opensearch.security.privileges.ActionPrivileges.updateClusterStateMetadata(ActionPrivileges.java:265) [opensearch-security-2.19.1.0.jar:2.19.1.0]
	at org.opensearch.security.privileges.ClusterStateMetadataDependentPrivileges.lambda$updateClusterStateMetadataAsync$0(ClusterStateMetadataDependentPrivileges.java:68) [opensearch-security-2.19.1.0.jar:2.19.1.0]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) [?:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) [opensearch-2.19.1.jar:2.19.1]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
[2025-03-06T08:03:18,974][INFO ][o.o.j.s.JobScheduler     ] [osc01n02] Will delay 87265 miliseconds for next execution of job .ds-log_dhcp-000007
[2025-03-06T08:03:20,583][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [osc01n02] Cancelling the migration process.
[2025-03-06T08:03:20,601][ERROR][o.o.s.p.ActionPrivileges ] [osc01n02] Unexpected exception while processing role: USR_INFORMATION_TECHNOLOGY_NETWORK
Ignoring role.
java.lang.IllegalArgumentException: Invalid key .ds-log_win_microsoft-windows-security-auditing-000481; not present in keySuperSet
	at com.selectivem.collections.CompactMapGroupBuilder$MapBuilder.get(CompactMapGroupBuilder.java:139) ~[special-collections-complete-1.4.0.jar:?]
	at org.opensearch.security.privileges.ActionPrivileges$StatefulIndexPrivileges.<init>(ActionPrivileges.java:1029) [opensearch-security-2.19.1.0.jar:2.19.1.0]
	at org.opensearch.security.privileges.ActionPrivileges.updateStatefulIndexPrivileges(ActionPrivileges.java:247) [opensearch-security-2.19.1.0.jar:2.19.1.0]
	at org.opensearch.security.privileges.ActionPrivileges.updateClusterStateMetadata(ActionPrivileges.java:265) [opensearch-security-2.19.1.0.jar:2.19.1.0]
	at org.opensearch.security.privileges.ClusterStateMetadataDependentPrivileges.lambda$updateClusterStateMetadataAsync$0(ClusterStateMetadataDependentPrivileges.java:68) [opensearch-security-2.19.1.0.jar:2.19.1.0]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) [?:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) [opensearch-2.19.1.jar:2.19.1]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
[2025-03-06T08:03:20,603][ERROR][o.o.s.p.ActionPrivileges ] [osc01n02] Unexpected exception while processing role: USR_INFORMATION_TECHNOLOGY_SIS
Ignoring role.
java.lang.IllegalArgumentException: Invalid key .ds-log_win_microsoft-windows-security-auditing-000481; not present in keySuperSet
	at com.selectivem.collections.CompactMapGroupBuilder$MapBuilder.get(CompactMapGroupBuilder.java:139) ~[special-collections-complete-1.4.0.jar:?]
	at org.opensearch.security.privileges.ActionPrivileges$StatefulIndexPrivileges.<init>(ActionPrivileges.java:1029) [opensearch-security-2.19.1.0.jar:2.19.1.0]
	at org.opensearch.security.privileges.ActionPrivileges.updateStatefulIndexPrivileges(ActionPrivileges.java:247) [opensearch-security-2.19.1.0.jar:2.19.1.0]
	at org.opensearch.security.privileges.ActionPrivileges.updateClusterStateMetadata(ActionPrivileges.java:265) [opensearch-security-2.19.1.0.jar:2.19.1.0]
	at org.opensearch.security.privileges.ClusterStateMetadataDependentPrivileges.lambda$updateClusterStateMetadataAsync$0(ClusterStateMetadataDependentPrivileges.java:68) [opensearch-security-2.19.1.0.jar:2.19.1.0]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) [?:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955) [opensearch-2.19.1.jar:2.19.1]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

Eugene7 · March 6, 2025, 5:19pm

Hi @landon_lslc,

Which version of the operating system do you use as a host for the OpenSearch cluster? How did you install OpenSearch?

landon_lslc · March 6, 2025, 5:22pm

The nodes are all Debian 12 and installed via the apt repo.

landon_lslc · March 6, 2025, 5:44pm

Is it possible that this is being caused by overlapping permissions on indices?

If we take a look at one of the problem roles in Dashboards, we see that the permissions are effectively listed twice. Once for log_win* and once for log_win_evtcol*.

I’m removing all of these duplicates and will see if it improves this in any case.
It seems like potentially a bug though, since the users do retain access and it seems to function fine. Other than the numerous errors.

landon_lslc · March 6, 2025, 6:39pm

I’ve tried to clean up those roles but unfortunately it is still throwing errors every once and a while. Anyone have any other ideas?

Topic		Replies	Views
Security_Exception on .opendistro-alerting-config index Security	6	424	February 29, 2024
Unrecognized Security Exceptions Security configure	2	691	February 24, 2022
Permission errors on indices:data/read/search[can_match] Security	2	565	October 20, 2022
Failed to start Document-level-monitor: Inconsistency of field data structures across documents for field Security Analytics	4	684	May 28, 2023
Opensearch alerting plugin is throwing exception Filter by user backend roles is enabled by security disabled Alerting	7	821	June 8, 2023

Invalid Key and role issues

Related topics