Trouble with mappings, detectors, and alerts

tlacuache · February 10, 2025, 7:10pm

I’m finding some things that don’t seem to be working the way I expect them to and I’m hoping someone smarter than me can tell me why.

First, to keep track of what I’ve read on the forums and in the docs so far:

this response from 2023 from @orazaly1508
this response to that same thread by @pdz
this comment by @sbcd90 (actually, the entire thread) which eventually led me to create github issue, although I’m not sure 100% that it’s related. I only bring it up because @sbcd90 in his comment I linked says “Security Analytics Detectors are compatible only with Index Aliases, not Index Patterns” and @orazaly1508 says in her comment I linked says “Yes, mapping APIs support data streams and index patterns” which seems to be contradictory to me.
mappings API documentation

So here’s what I’m trying to do. I have some of my own index templates which are made up of composable components. In my index templates, I am specifying an alias like this (don’t get hung up on the _REPLACER strings in there, those get resolved before import).

So, as my indexes are created via data ingestion, they are getting assigned to their aliases automatically:

I mention this bit about the aliases because of the comment from @sbcd90 in that other thread I linked about how security analytics detectors only support aliases or data streams, which is why I’m using this alias.

I have this mapping that I want to create for my malcolm_network index alias for the network ruleset. I’m creating the mapping here and can verify via a GET API after my POST creating the mapping that it seems to be created correctly:

$ docker compose exec opensearch curl -fsSL -w '\n' -H'Content-Type: application/json' -XGET  "http://localhost:9200/_plugins/_security_analytics/mappings?index_name=malcolm_network"  | jq

{
  "malcolm_network": {
    "mappings": {
      "properties": {
        "zeek": {
          "properties": {
            "x509": {
              "properties": {
                "certificate": {
                  "properties": {
                    "serial": {
                      "type": "alias",
                      "path": "zeek.x509.certificate_serial"
                    }
                  }
                }
              }
            }
          }
        },
        "service": {
          "type": "alias",
          "path": "network.protocol"
        },
        "rejected": {
          "type": "alias",
          "path": "zeek.dns.rejected"
        },
        "id": {
          "properties": {
            "resp_p": {
              "type": "alias",
              "path": "destination.port"
            },
            "resp_h": {
              "type": "alias",
              "path": "destination.ip"
            },
            "orig_h": {
              "type": "alias",
              "path": "source.ip"
            }
          }
        },
        "Z-flag": {
          "type": "alias",
          "path": "zeek.dns.Z"
        }
      }
    }
  }
}

So far so good. So then I go into the UI to create a detector. I do the following:

Give it a name
Select the malcolm_network alias as the data source
Select Network as the log type

This is where I hit my first snag:

I’ve already created these mappings! Why do they show up as being unmapped here? The weird thing is, if I go in and manually populate the mappings here and then continue on, then make the exact same GET API call I posted above later on, I get the exact same thing I’ve already created. The point of me creating these mappings in the API, I thought, was so they’d be used here in this Field Mapping section.

Continuing on, I uncheck the threat intel feeds checkbox because I’m not getting into that yet, then click Next.

Next, I removed the trigger, because I’m not getting into that yet either. So I click to create the detector and it says it’s attempting to create it.

A few minutes later, I look in my logs and see these errors. First these three lines:

[2025-02-10T19:05:50,739][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [netflow.firewall_event]];
[2025-02-10T19:05:50,739][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [netflow.firewall_event]];
[2025-02-10T19:05:50,739][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [blocked-flag]];

That makes sense to me, because I didn’t have mappings for those three fields, so I ignore those.

Then this:

Security Analytics - IndexService - createAliases: StatusCodeError: [illegal_argument_exception] The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead.
    at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (node:events:529:35)
    at IncomingMessage.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1400:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21) {
  status: 400,
  displayName: 'BadRequest',
  path: '/_aliases',
  query: {},
  body: {
    error: {
      root_cause: [Array],
      type: 'illegal_argument_exception',
      reason: 'The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead.'
    },
    status: 400
  },
  statusCode: 400,
  response: '{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead."}],"type":"illegal_argument_exception","reason":"The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead."},"status":400}',
  toString: [Function (anonymous)],
  toJSON: [Function (anonymous)]
}

What stands out to me here is this message:

The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead.

Um… the UI itself, and the documentation, and about everything else I’ve read says that I’m supposed to use index aliases. Why am I getting this error?

For what it’s worth, even after getting this error I can go into the threat detectors list and see my detector:

So I guess it’s working?

TL;DR, though:

Why don’t the mappings I created via the API show up as pre-populated in the UI when I go to create the detector.
Why do I see a “The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead.” error message when everything the (very limited) documentation and what I’ve read in the forums here indicates I’m supposed to use aliases?

Thanks!

-SG

kris · February 13, 2025, 4:24pm

@cwperks @darshit - would you have ideas to share with @tlacuache ? thanks

tlacuache · February 13, 2025, 5:10pm

I did just update to v2.19.0 yesterday, I’ll see if the behavior is the same as what I was seeing before.

tlacuache · February 13, 2025, 5:43pm

Okay, with v2.19.0 I don’t see the message about “The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead.” any more, so maybe that’s something that was improved in the latest version?

When I did go to create a detector for Windows events (e.g., like those coming from winlogbeat) I went through the process and then it said “Attempting to create the detector.” It sat there for several minutes, then I saw in the logs:

dashboards-1  | POST http://opensearch:9200/_plugins/_security_analytics/detectors => Socket timeout
dashboards-1  | Security Analytics - DetectorsService - createDetector: StatusCodeError: Request Timeout after 180000ms
dashboards-1  |     at /usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:397:9
dashboards-1  |     at Timeout.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:429:7)
dashboards-1  |     at listOnTimeout (node:internal/timers:569:17)
dashboards-1  |     at processTimers (node:internal/timers:512:7) {
dashboards-1  |   status: undefined,
dashboards-1  |   displayName: 'RequestTimeout',
dashboards-1  |   body: undefined
dashboards-1  | }

and then various errors from opensearch, here’s an example of a few:

I’m not using threat intel yet so I guess this is something I could ignore (?):

opensearch-1  | [2025-02-13T17:22:38,573][ERROR][o.o.s.t.s.SATIFSourceConfigService] [opensearch] Failed to fetch all threat intel source configs for search request [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[.opensearch-sap--job], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], routing='null', preference='_primary_first', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":9999,"query":{"bool":{"must":[{"bool":{"should":[{"match":{"_id":{"query":"alienvault_reputation_ip_database","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"filter":[{"bool":{"should":[{"bool":{"must":[{"exists":{"field":"source_config","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"seq_no_primary_term":true}, cancelAfterTimeInterval=null, pipeline=null, phaseTook=false}]

No idea about this one:

opensearch-1  | org.opensearch.index.IndexNotFoundException: no such index [.opensearch-sap--job]
opensearch-1  | 	at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:1070) ~[opensearch-2.19.0.jar:2.19.0]
opensearch-1  | 	at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:1007) ~[opensearch-2.19.0.jar:2.19.0]
opensearch-1  | 	at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:963) ~[opensearch-2.19.0.jar:2.19.0]
opensearch-1  | 	at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:255) ~[opensearch-2.19.0.jar:2.19.0]

I thought all of my fields showed as mapped, but here’s these too:

opensearch-1  | [2025-02-13T17:34:07,857][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [OriginalFileName]];
opensearch-1  | [2025-02-13T17:34:07,857][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [OriginalFileName]];
opensearch-1  | [2025-02-13T17:34:07,857][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [OriginalFileName]];
opensearch-1  | [2025-02-13T17:34:07,857][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [OriginalFileName]];
opensearch-1  | [2025-02-13T17:34:57,729][ERROR][o.o.a.c.l.LockService    ] [opensearch] Lock is null. Nothing to release.
opensearch-1  | [2025-02-13T17:34:57,741][ERROR][o.o.a.c.l.LockService    ] [opensearch] Lock is null. Nothing to release.
opensearch-1  | [2025-02-13T17:35:57,729][ERROR][o.o.a.c.l.LockService    ] [opensearch] Lock is null. Nothing to release.
opensearch-1  | [2025-02-13T17:35:57,736][ERROR][o.o.a.c.l.LockService    ] [opensearch] Lock is null. Nothing to release.
opensearch-1  | org.opensearch.index.IndexNotFoundException: no such index [.opensearch-sap--job]
opensearch-1  | 	at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:1070) ~[opensearch-2.19.0.jar:2.19.0]
opensearch-1  | 	at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:1007) ~[opensearch-2.19.0.jar:2.19.0]
opensearch-1  | 	at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:963) ~[opensearch-2.19.0.jar:2.19.0]
opensearch-1  | 	at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:255) ~[opensearch-2.19.0.jar:2.19.0]

I don’t know if any of those matter, but it’s sure noisy.

After a few minutes, though, I refreshed the page and see this:

Going into the detector I see:

So it shows as active, I guess, although there’s not “Created at” time which I don’t know if that’s a big deal.

Then, after a few minutes, I start seeing these messages:

opensearch-1  | [2025-02-13T17:38:17,169][ERROR][o.o.c.a.u.AlertingException] [opensearch] Alerting error: [.opendistro-alerting-config/5deJUJptR4GmvsxeG4jQVQ][[.opendistro-alerting-config][0]] VersionConflictEngineException[[CRtWAJUBTUf2G4w70lPF-metadata-0xtQAJUBTUf2G4w7VFJE-metadata]: version conflict, required seqNo [11], primary term [1]. current document has seqNo [17] and primary term [1]]
opensearch-1  | [2025-02-13T17:38:17,201][ERROR][o.o.a.DocumentLevelMonitorRunner] [opensearch] Failed running Document-level-monitor winlogtest
opensearch-1  | org.opensearch.commons.alerting.util.AlertingException: [CRtWAJUBTUf2G4w70lPF-metadata-0xtQAJUBTUf2G4w7VFJE-metadata]: version conflict, required seqNo [11], primary term [1]. current document has seqNo [17] and primary term [1]
opensearch-1  | 	at org.opensearch.commons.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70) ~[common-utils-2.19.0.0.jar:?]
opensearch-1  | 	at org.opensearch.alerting.MonitorMetadataService.upsertMetadata(MonitorMetadataService.kt:129) ~[opensearch-alerting-2.19.0.0.jar:2.19.0.0]
opensearch-1  | 	at org.opensearch.alerting.MonitorMetadataService$upsertMetadata$1.invokeSuspend(MonitorMetadataService.kt) ~[opensearch-alerting-2.19.0.0.jar:2.19.0.0]
opensearch-1  | 	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
opensearch-1  | 	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:285) [kotlinx-coroutines-core-1.1.1.jar:?]
opensearch-1  | 	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
opensearch-1  | 	at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
opensearch-1  | 	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
opensearch-1  | Caused by: java.lang.Exception: org.opensearch.index.engine.VersionConflictEngineException: [CRtWAJUBTUf2G4w70lPF-metadata-0xtQAJUBTUf2G4w7VFJE-metadata]: version conflict, required seqNo [11], primary term [1]. current document has seqNo [17] and primary term [1]
opensearch-1  | 	... 8 more
opensearch-1  | [2025-02-13T17:38:57,728][ERROR][o.o.a.c.l.LockService    ] [opensearch] Lock is null. Nothing to release.

I navigated to the Security Alerts dashboard and see:

I’m not sure what NoOp trigger is, but it’s certainly not a name I came up with. Clickin on the alert gives me:

Every minute or now so I get one of these bogus empty alerts.

I also navigated back to the threat detectors page, and have this now:

So the one I made got created twice? Maybe after the timeout? At least clicking on either of them now shows that they have an created/updated time. But I don’t know why there’s two.

I’m not throwing shade on anyone, but it seems to me like this feature is … a little rough around the edges? I don’t feel like I’m doing anything outside of what the normal workflow would be. But I feel like there’s a lot that’s not working now, at least in my situation.

EDIT: I’m going to attach a giant output/error log file from opensearch, this all happens in the 30 minutes or so after I start the detectors.

lots and lots of errors (somewhat curated for character limits)

opensearch-1  | [2025-02-13T17:16:01,958][ERROR][o.o.t.ExecuteResultResponseRecorder] [opensearch] Fail to confirm rcf update
opensearch-1  | org.opensearch.timeseries.common.exception.TimeSeriesException: fail to get config
opensearch-1  | [2025-02-13T17:16:01,964][ERROR][o.o.t.t.TaskManager      ] [opensearch] Failed to update realtime task for config uBtOAJUBTUf2G4w7c1IP
opensearch-1  | org.opensearch.timeseries.common.exception.ResourceNotFoundException: can't find latest task
opensearch-1  | [2025-02-13T17:16:01,966][ERROR][o.o.t.ExecuteResultResponseRecorder] [opensearch] Can't find latest realtime task of config uBtOAJUBTUf2G4w7c1IP
opensearch-1  | [2025-02-13T17:16:59,075][ERROR][o.o.s.t.TransportIndexDetectorAction] [opensearch] PERF_DEBUG_SAP: Fetching alias path pairs to construct rule_field_names
opensearch-1  | [2025-02-13T17:18:57,991][ERROR][o.o.s.t.TransportIndexDetectorAction] [opensearch] PERF_DEBUG_SAP: Fetching alias path pairs to construct rule_field_names
opensearch-1  | [2025-02-13T17:22:38,573][ERROR][o.o.s.t.s.SATIFSourceConfigService] [opensearch] Failed to fetch all threat intel source configs for search request [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[.opensearch-sap--job], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], routing='null', preference='_primary_first', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":9999,"query":{"bool":{"must":[{"bool":{"should":[{"match":{"_id":{"query":"alienvault_reputation_ip_database","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"filter":[{"bool":{"should":[{"bool":{"must":[{"exists":{"field":"source_config","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"seq_no_primary_term":true}, cancelAfterTimeInterval=null, pipeline=null, phaseTook=false}]
opensearch-1  | org.opensearch.index.IndexNotFoundException: no such index [.opensearch-sap--job]
opensearch-1  | [2025-02-13T17:22:38,578][ERROR][o.o.s.t.s.SATIFSourceConfigManagementService] [opensearch] Failed to fetch all threat intel source configs for search request [{"size":9999,"query":{"bool":{"must":[{"bool":{"should":[{"match":{"_id":{"query":"alienvault_reputation_ip_database","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"filter":[{"bool":{"should":[{"bool":{"must":[{"exists":{"field":"source_config","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"seq_no_primary_term":true}]
opensearch-1  | org.opensearch.index.IndexNotFoundException: no such index [.opensearch-sap--job]
opensearch-1  | [2025-02-13T17:24:03,677][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [ommandLine]];
opensearch-1  | [2025-02-13T17:26:02,571][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [ommandLine]];
opensearch-1  | [2025-02-13T17:32:07,611][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [ommandLine]];
opensearch-1  | [2025-02-13T17:34:07,856][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [ommandLine]];
opensearch-1  | [2025-02-13T17:38:15,401][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [ommandLine]];
opensearch-1  | [2025-02-13T17:38:17,169][ERROR][o.o.c.a.u.AlertingException] [opensearch] Alerting error: [.opendistro-alerting-config/5deJUJptR4GmvsxeG4jQVQ][[.opendistro-alerting-config][0]] VersionConflictEngineException[[CRtWAJUBTUf2G4w70lPF-metadata-0xtQAJUBTUf2G4w7VFJE-metadata]: version conflict, required seqNo [11], primary term [1]. current document has seqNo [17] and primary term [1]]
opensearch-1  | [2025-02-13T17:38:17,201][ERROR][o.o.a.DocumentLevelMonitorRunner] [opensearch] Failed running Document-level-monitor winlogtest
opensearch-1  | org.opensearch.commons.alerting.util.AlertingException: [CRtWAJUBTUf2G4w70lPF-metadata-0xtQAJUBTUf2G4w7VFJE-metadata]: version conflict, required seqNo [11], primary term [1]. current document has seqNo [17] and primary term [1]
opensearch-1  | Caused by: java.lang.Exception: org.opensearch.index.engine.VersionConflictEngineException: [CRtWAJUBTUf2G4w70lPF-metadata-0xtQAJUBTUf2G4w7VFJE-metadata]: version conflict, required seqNo [11], primary term [1]. current document has seqNo [17] and primary term [1]
opensearch-1  | 	... 8 more
opensearch-1  | [2025-02-13T17:40:23,026][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [ommandLine]];
opensearch-1  | [2025-02-13T17:40:24,237][ERROR][o.o.c.a.u.AlertingException] [opensearch] Alerting error: [.opendistro-alerting-config/5deJUJptR4GmvsxeG4jQVQ][[.opendistro-alerting-config][0]] VersionConflictEngineException[[RBtYAJUBTUf2G4w7oVNk-metadata-4BtSAJUBTUf2G4w7JFLF-metadata]: version conflict, required seqNo [16], primary term [1]. current document has seqNo [19] and primary term [1]]
opensearch-1  | [2025-02-13T17:40:24,253][ERROR][o.o.a.DocumentLevelMonitorRunner] [opensearch] Failed running Document-level-monitor winlogtest
opensearch-1  | org.opensearch.commons.alerting.util.AlertingException: [RBtYAJUBTUf2G4w7oVNk-metadata-4BtSAJUBTUf2G4w7JFLF-metadata]: version conflict, required seqNo [16], primary term [1]. current document has seqNo [19] and primary term [1]
opensearch-1  | Caused by: java.lang.Exception: org.opensearch.index.engine.VersionConflictEngineException: [RBtYAJUBTUf2G4w7oVNk-metadata-4BtSAJUBTUf2G4w7JFLF-metadata]: version conflict, required seqNo [16], primary term [1]. current document has seqNo [19] and primary term [1]
opensearch-1  | 	... 8 more
opensearch-1  | [2025-02-13T17:44:29,530][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [ommandLine]];
opensearch-1  | [2025-02-13T17:44:30,714][WARN ][o.o.w.QueryGroupTask     ] [opensearch] QueryGroup _id can't be null, It should be set before accessing it. This is abnormal behaviour 
opensearch-1  | [2025-02-13T17:44:32,814][ERROR][o.o.s.t.TransportIndexDetectorAction] [opensearch] PERF_DEBUG_SAP: Fetching alias path pairs to construct rule_field_names
opensearch-1  | [2025-02-13T17:44:34,655][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [answer]];
opensearch-1  | [2025-02-13T17:44:34,656][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [dns.question.registered_domain]];
opensearch-1  | [2025-02-13T17:45:33,901][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [answer]];
opensearch-1  | [2025-02-13T17:45:33,901][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [dns.question.registered_domain]];
opensearch-1  | [2025-02-13T17:46:16,364][WARN ][o.o.w.QueryGroupTask     ] [opensearch] QueryGroup _id can't be null, It should be set before accessing it. This is abnormal behaviour 
opensearch-1  | [2025-02-13T17:46:16,526][ERROR][o.o.s.t.TransportDeleteDetectorAction] [opensearch] Error deleting unused templates: groupSize must be greater than 0 but was 0
opensearch-1  | [2025-02-13T17:46:28,790][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [ommandLine]];
opensearch-1  | [2025-02-13T17:46:29,753][WARN ][o.o.w.QueryGroupTask     ] [opensearch] QueryGroup _id can't be null, It should be set before accessing it. This is abnormal behaviour 
opensearch-1  | [2025-02-13T17:47:59,619][ERROR][o.o.s.t.TransportIndexDetectorAction] [opensearch] PERF_DEBUG_SAP: Fetching alias path pairs to construct rule_field_names
opensearch-1  | [2025-02-13T17:48:01,422][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [answer]];
opensearch-1  | [2025-02-13T17:48:01,422][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [dns.question.registered_domain]];
opensearch-1  | [2025-02-13T17:49:00,536][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [answer]];
opensearch-1  | [2025-02-13T17:49:00,537][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [dns.question.registered_domain]];
opensearch-1  | [2025-02-13T17:50:00,655][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [answer]];
opensearch-1  | [2025-02-13T17:50:00,655][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [dns.question.registered_domain]];
opensearch-1  | [2025-02-13T17:50:25,450][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [ommandLine]];
opensearch-1  | [2025-02-13T17:50:26,701][ERROR][o.o.c.a.u.AlertingException] [opensearch] Alerting error: [.opendistro-alerting-config/5deJUJptR4GmvsxeG4jQVQ][[.opendistro-alerting-config][0]] VersionConflictEngineException[[CRtWAJUBTUf2G4w70lPF-metadata-0xtQAJUBTUf2G4w7VFJE-metadata]: version conflict, required seqNo [17], primary term [1]. current document has seqNo [23] and primary term [1]]
opensearch-1  | [2025-02-13T17:50:26,718][ERROR][o.o.a.DocumentLevelMonitorRunner] [opensearch] Failed running Document-level-monitor winlogtest
opensearch-1  | org.opensearch.commons.alerting.util.AlertingException: [CRtWAJUBTUf2G4w70lPF-metadata-0xtQAJUBTUf2G4w7VFJE-metadata]: version conflict, required seqNo [17], primary term [1]. current document has seqNo [23] and primary term [1]
opensearch-1  | Caused by: java.lang.Exception: org.opensearch.index.engine.VersionConflictEngineException: [CRtWAJUBTUf2G4w70lPF-metadata-0xtQAJUBTUf2G4w7VFJE-metadata]: version conflict, required seqNo [17], primary term [1]. current document has seqNo [23] and primary term [1]
opensearch-1  | 	... 8 more
opensearch-1  | [2025-02-13T17:51:00,861][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [answer]];
opensearch-1  | [2025-02-13T17:51:00,862][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [dns.question.registered_domain]];
opensearch-1  | [2025-02-13T17:52:04,488][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [answer]];
opensearch-1  | [2025-02-13T17:52:04,488][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [dns.question.registered_domain]];
opensearch-1  | [2025-02-13T17:52:37,803][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [ommandLine]];
opensearch-1  | [2025-02-13T17:53:00,491][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [answer]];
opensearch-1  | [2025-02-13T17:53:00,491][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [dns.question.registered_domain]];
opensearch-1  | [2025-02-13T17:53:11,621][ERROR][o.o.a.t.TransportDocLevelMonitorFanOutAction] [opensearch] 4BtSAJUBTUf2G4w7JFLF Failed to run fan_out on node XQiS5yRzQ9a71qYv-WdqCw. for Monitor Type doc_level_monitor ExecutionId RBtYAJUBTUf2G4w7oVNk_2025-02-13T17:44:57.743472233_10f09fc9-182b-4e1b-aa48-e2fbb5ba1544
opensearch-1  | java.lang.IllegalStateException: Monitor 4BtSAJUBTUf2G4w7JFLF: Failed to run percolate search for sourceIndex [malcolm_beats_winlog_25w06, malcolm_beats_initial, malcolm_beats_winlog_22w06, malcolm_beats_winlog_23w43, malcolm_beats_zeek_25w06, malcolm_beats_winlog_23w45, malcolm_beats_suricata_25w06, malcolm_beats_nginx_25w06] and queryIndex [.opensearch-sap-windows-detectors-queries-optimized-46d5cf6d-2ec1-494f-8b58-2dc4c949531e-000001] for 13247 document(s)
opensearch-1  | Caused by: org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
opensearch-1  | Caused by: org.opensearch.OpenSearchException$3: Query contains too many nested clauses; maxClauseCount is set to 8192
opensearch-1  | Caused by: org.apache.lucene.search.IndexSearcher$TooManyNestedClauses: Query contains too many nested clauses; maxClauseCount is set to 8192
opensearch-1  | [2025-02-13T17:53:11,624][ERROR][o.o.c.a.u.AlertingException] [opensearch] Alerting error: java.lang.IllegalStateException: Monitor 4BtSAJUBTUf2G4w7JFLF: Failed to run percolate search for sourceIndex [malcolm_beats_winlog_25w06, malcolm_beats_initial, malcolm_beats_winlog_22w06, malcolm_beats_winlog_23w43, malcolm_beats_zeek_25w06, malcolm_beats_winlog_23w45, malcolm_beats_suricata_25w06, malcolm_beats_nginx_25w06] and queryIndex [.opensearch-sap-windows-detectors-queries-optimized-46d5cf6d-2ec1-494f-8b58-2dc4c949531e-000001] for 13247 document(s)
opensearch-1  | [2025-02-13T17:53:11,625][ERROR][o.o.a.DocumentLevelMonitorRunner] [opensearch] Fan out failed in node XQiS5yRzQ9a71qYv-WdqCw
opensearch-1  | org.opensearch.transport.RemoteTransportException: [opensearch][172.17.0.5:9300][cluster:admin/opensearch/alerting/monitor/doclevel/fanout]
opensearch-1  | Caused by: org.opensearch.commons.alerting.util.AlertingException: Monitor 4BtSAJUBTUf2G4w7JFLF: Failed to run percolate search for sourceIndex [malcolm_beats_winlog_25w06, malcolm_beats_initial, malcolm_beats_winlog_22w06, malcolm_beats_winlog_23w43, malcolm_beats_zeek_25w06, malcolm_beats_winlog_23w45, malcolm_beats_suricata_25w06, malcolm_beats_nginx_25w06] and queryIndex [.opensearch-sap-windows-detectors-queries-optimized-46d5cf6d-2ec1-494f-8b58-2dc4c949531e-000001] for 13247 document(s)
opensearch-1  | Caused by: java.lang.Exception: java.lang.IllegalStateException: Monitor 4BtSAJUBTUf2G4w7JFLF: Failed to run percolate search for sourceIndex [malcolm_beats_winlog_25w06, malcolm_beats_initial, malcolm_beats_winlog_22w06, malcolm_beats_winlog_23w43, malcolm_beats_zeek_25w06, malcolm_beats_winlog_23w45, malcolm_beats_suricata_25w06, malcolm_beats_nginx_25w06] and queryIndex [.opensearch-sap-windows-detectors-queries-optimized-46d5cf6d-2ec1-494f-8b58-2dc4c949531e-000001] for 13247 document(s)
opensearch-1  | [2025-02-13T17:53:42,767][ERROR][o.o.a.t.TransportDocLevelMonitorFanOutAction] [opensearch] 4BtSAJUBTUf2G4w7JFLF Failed to run fan_out on node XQiS5yRzQ9a71qYv-WdqCw. for Monitor Type doc_level_monitor ExecutionId RBtYAJUBTUf2G4w7oVNk_2025-02-13T17:44:57.743472233_10f09fc9-182b-4e1b-aa48-e2fbb5ba1544
opensearch-1  | java.lang.IllegalStateException: Monitor 4BtSAJUBTUf2G4w7JFLF: Failed to run percolate search for sourceIndex [malcolm_beats_winlog_25w06, malcolm_beats_initial, malcolm_beats_winlog_22w06, malcolm_beats_winlog_23w43, malcolm_beats_zeek_25w06, malcolm_beats_winlog_23w45, malcolm_beats_suricata_25w06, malcolm_beats_nginx_25w06] and queryIndex [.opensearch-sap-windows-detectors-queries-optimized-46d5cf6d-2ec1-494f-8b58-2dc4c949531e-000001] for 14225 document(s)
opensearch-1  | Caused by: org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
opensearch-1  | Caused by: org.opensearch.OpenSearchException$3: Query contains too many nested clauses; maxClauseCount is set to 8192
opensearch-1  | Caused by: org.apache.lucene.search.IndexSearcher$TooManyNestedClauses: Query contains too many nested clauses; maxClauseCount is set to 8192
opensearch-1  | [2025-02-13T17:53:42,770][ERROR][o.o.c.a.u.AlertingException] [opensearch] Alerting error: java.lang.IllegalStateException: Monitor 4BtSAJUBTUf2G4w7JFLF: Failed to run percolate search for sourceIndex [malcolm_beats_winlog_25w06, malcolm_beats_initial, malcolm_beats_winlog_22w06, malcolm_beats_winlog_23w43, malcolm_beats_zeek_25w06, malcolm_beats_winlog_23w45, malcolm_beats_suricata_25w06, malcolm_beats_nginx_25w06] and queryIndex [.opensearch-sap-windows-detectors-queries-optimized-46d5cf6d-2ec1-494f-8b58-2dc4c949531e-000001] for 14225 document(s)
opensearch-1  | [2025-02-13T17:53:42,770][ERROR][o.o.a.DocumentLevelMonitorRunner] [opensearch] Fan out failed in node XQiS5yRzQ9a71qYv-WdqCw
opensearch-1  | org.opensearch.transport.RemoteTransportException: [opensearch][172.17.0.5:9300][cluster:admin/opensearch/alerting/monitor/doclevel/fanout]
opensearch-1  | Caused by: org.opensearch.commons.alerting.util.AlertingException: Monitor 4BtSAJUBTUf2G4w7JFLF: Failed to run percolate search for sourceIndex [malcolm_beats_winlog_25w06, malcolm_beats_initial, malcolm_beats_winlog_22w06, malcolm_beats_winlog_23w43, malcolm_beats_zeek_25w06, malcolm_beats_winlog_23w45, malcolm_beats_suricata_25w06, malcolm_beats_nginx_25w06] and queryIndex [.opensearch-sap-windows-detectors-queries-optimized-46d5cf6d-2ec1-494f-8b58-2dc4c949531e-000001] for 14225 document(s)
opensearch-1  | Caused by: java.lang.Exception: java.lang.IllegalStateException: Monitor 4BtSAJUBTUf2G4w7JFLF: Failed to run percolate search for sourceIndex [malcolm_beats_winlog_25w06, malcolm_beats_initial, malcolm_beats_winlog_22w06, malcolm_beats_winlog_23w43, malcolm_beats_zeek_25w06, malcolm_beats_winlog_23w45, malcolm_beats_suricata_25w06, malcolm_beats_nginx_25w06] and queryIndex [.opensearch-sap-windows-detectors-queries-optimized-46d5cf6d-2ec1-494f-8b58-2dc4c949531e-000001] for 14225 document(s)
opensearch-1  | [2025-02-13T17:54:00,787][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [answer]];
opensearch-1  | [2025-02-13T17:54:00,787][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [dns.question.registered_domain]];
opensearch-1  | [2025-02-13T17:54:02,384][WARN ][o.o.n.a.PluginBaseAction ] [opensearch] notifications:IllegalArgumentException:
opensearch-1  | org.opensearch.action.ActionRequestValidationException: Validation Failed: 1: id is missing;
opensearch-1  | [2025-02-13T17:54:02,393][WARN ][o.o.n.a.PluginBaseAction ] [opensearch] notifications:IllegalArgumentException:
opensearch-1  | org.opensearch.action.ActionRequestValidationException: Validation Failed: 1: id is missing;
opensearch-1  | [2025-02-13T17:54:02,398][WARN ][o.o.n.a.PluginBaseAction ] [opensearch] notifications:IllegalArgumentException:
opensearch-1  | org.opensearch.action.ActionRequestValidationException: Validation Failed: 1: id is missing;
opensearch-1  | [2025-02-13T17:54:02,410][ERROR][o.o.s.t.TransportCorrelateFindingAction] [opensearch] Exception occurred while processing correlations for monitor id 7xtsAJUBTUf2G4w7t1NR and finding id 260d24de-667d-44c1-8c91-2df8acc60d02
opensearch-1  | org.opensearch.ResourceAlreadyExistsException: index [.opensearch-sap-correlation-history-2025.02.13-1/dpQ8Ih0TQ72IM74nXPhVAQ] already exists
opensearch-1  | [2025-02-13T17:54:02,411][WARN ][o.o.n.a.PluginBaseAction ] [opensearch] notifications:IllegalArgumentException:
opensearch-1  | org.opensearch.action.ActionRequestValidationException: Validation Failed: 1: id is missing;
opensearch-1  | [2025-02-13T17:54:02,412][ERROR][o.o.s.t.TransportCorrelateFindingAction] [opensearch] Exception occurred while processing correlations for monitor id 7xtsAJUBTUf2G4w7t1NR and finding id 304b8f9a-bdb0-4fdc-a26d-28f09648395e
opensearch-1  | org.opensearch.ResourceAlreadyExistsException: index [.opensearch-sap-correlation-history-2025.02.13-1/dpQ8Ih0TQ72IM74nXPhVAQ] already exists
opensearch-1  | [2025-02-13T17:54:02,412][ERROR][o.o.s.u.SecurityAnalyticsException] [opensearch] Security Analytics error:
opensearch-1  | org.opensearch.ResourceAlreadyExistsException: index [.opensearch-sap-correlation-history-2025.02.13-1/dpQ8Ih0TQ72IM74nXPhVAQ] already exists
opensearch-1  | [2025-02-13T17:54:02,413][ERROR][o.o.s.u.SecurityAnalyticsException] [opensearch] Security Analytics error:
opensearch-1  | org.opensearch.ResourceAlreadyExistsException: index [.opensearch-sap-correlation-history-2025.02.13-1/dpQ8Ih0TQ72IM74nXPhVAQ] already exists
opensearch-1  | [2025-02-13T17:54:02,413][ERROR][o.o.s.t.TransportCorrelateFindingAction] [opensearch] Exception occurred while processing correlations for monitor id 7xtsAJUBTUf2G4w7t1NR and finding id 01ece641-961a-4369-87d0-c3abdfeaa267
opensearch-1  | org.opensearch.ResourceAlreadyExistsException: index [.opensearch-sap-correlation-history-2025.02.13-1/dpQ8Ih0TQ72IM74nXPhVAQ] already exists
opensearch-1  | [2025-02-13T17:54:02,416][ERROR][o.o.s.u.SecurityAnalyticsException] [opensearch] Security Analytics error:
opensearch-1  | org.opensearch.ResourceAlreadyExistsException: index [.opensearch-sap-correlation-history-2025.02.13-1/dpQ8Ih0TQ72IM74nXPhVAQ] already exists
opensearch-1  | [2025-02-13T17:54:02,592][ERROR][o.o.s.c.JoinEngine       ] [opensearch] [CORRELATIONS] Exception encountered while searching correlation rule index for finding id 59382e77-be47-421f-ad4a-76369e6cc69c
opensearch-1  | org.opensearch.index.IndexNotFoundException: no such index [.opensearch-sap-correlation-rules-config]
opensearch-1  | [2025-02-13T17:54:10,417][ERROR][o.o.a.t.TransportDocLevelMonitorFanOutAction] [opensearch] 4BtSAJUBTUf2G4w7JFLF Failed to run fan_out on node XQiS5yRzQ9a71qYv-WdqCw. for Monitor Type doc_level_monitor ExecutionId RBtYAJUBTUf2G4w7oVNk_2025-02-13T17:44:57.743472233_10f09fc9-182b-4e1b-aa48-e2fbb5ba1544
opensearch-1  | java.lang.IllegalStateException: Monitor 4BtSAJUBTUf2G4w7JFLF: Failed to run percolate search for sourceIndex [malcolm_beats_winlog_25w06, malcolm_beats_initial, malcolm_beats_winlog_22w06, malcolm_beats_winlog_23w43, malcolm_beats_zeek_25w06, malcolm_beats_winlog_23w45, malcolm_beats_suricata_25w06, malcolm_beats_nginx_25w06] and queryIndex [.opensearch-sap-windows-detectors-queries-optimized-46d5cf6d-2ec1-494f-8b58-2dc4c949531e-000001] for 19834 document(s)
opensearch-1  | Caused by: org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
opensearch-1  | Caused by: org.opensearch.OpenSearchException$3: Query contains too many nested clauses; maxClauseCount is set to 8192
opensearch-1  | Caused by: org.apache.lucene.search.IndexSearcher$TooManyNestedClauses: Query contains too many nested clauses; maxClauseCount is set to 8192
opensearch-1  | [2025-02-13T17:54:10,419][ERROR][o.o.c.a.u.AlertingException] [opensearch] Alerting error: java.lang.IllegalStateException: Monitor 4BtSAJUBTUf2G4w7JFLF: Failed to run percolate search for sourceIndex [malcolm_beats_winlog_25w06, malcolm_beats_initial, malcolm_beats_winlog_22w06, malcolm_beats_winlog_23w43, malcolm_beats_zeek_25w06, malcolm_beats_winlog_23w45, malcolm_beats_suricata_25w06, malcolm_beats_nginx_25w06] and queryIndex [.opensearch-sap-windows-detectors-queries-optimized-46d5cf6d-2ec1-494f-8b58-2dc4c949531e-000001] for 19834 document(s)
opensearch-1  | [2025-02-13T17:54:10,419][ERROR][o.o.a.DocumentLevelMonitorRunner] [opensearch] Fan out failed in node XQiS5yRzQ9a71qYv-WdqCw
opensearch-1  | org.opensearch.transport.RemoteTransportException: [opensearch][172.17.0.5:9300][cluster:admin/opensearch/alerting/monitor/doclevel/fanout]
opensearch-1  | Caused by: org.opensearch.commons.alerting.util.AlertingException: Monitor 4BtSAJUBTUf2G4w7JFLF: Failed to run percolate search for sourceIndex [malcolm_beats_winlog_25w06, malcolm_beats_initial, malcolm_beats_winlog_22w06, malcolm_beats_winlog_23w43, malcolm_beats_zeek_25w06, malcolm_beats_winlog_23w45, malcolm_beats_suricata_25w06, malcolm_beats_nginx_25w06] and queryIndex [.opensearch-sap-windows-detectors-queries-optimized-46d5cf6d-2ec1-494f-8b58-2dc4c949531e-000001] for 19834 document(s)
opensearch-1  | Caused by: java.lang.Exception: java.lang.IllegalStateException: Monitor 4BtSAJUBTUf2G4w7JFLF: Failed to run percolate search for sourceIndex [malcolm_beats_winlog_25w06, malcolm_beats_initial, malcolm_beats_winlog_22w06, malcolm_beats_winlog_23w43, malcolm_beats_zeek_25w06, malcolm_beats_winlog_23w45, malcolm_beats_suricata_25w06, malcolm_beats_nginx_25w06] and queryIndex [.opensearch-sap-windows-detectors-queries-optimized-46d5cf6d-2ec1-494f-8b58-2dc4c949531e-000001] for 19834 document(s)
opensearch-1  | [2025-02-13T17:54:11,787][ERROR][o.o.c.a.u.AlertingException] [opensearch] Alerting error: [.opendistro-alerting-config/5deJUJptR4GmvsxeG4jQVQ][[.opendistro-alerting-config][0]] VersionConflictEngineException[[RBtYAJUBTUf2G4w7oVNk-metadata-4BtSAJUBTUf2G4w7JFLF-metadata]: version conflict, required seqNo [19], primary term [1]. current document has seqNo [38] and primary term [1]]
opensearch-1  | [2025-02-13T17:54:11,812][ERROR][o.o.a.DocumentLevelMonitorRunner] [opensearch] Failed running Document-level-monitor winlogtest
opensearch-1  | org.opensearch.commons.alerting.util.AlertingException: [RBtYAJUBTUf2G4w7oVNk-metadata-4BtSAJUBTUf2G4w7JFLF-metadata]: version conflict, required seqNo [19], primary term [1]. current document has seqNo [38] and primary term [1]
opensearch-1  | Caused by: java.lang.Exception: org.opensearch.index.engine.VersionConflictEngineException: [RBtYAJUBTUf2G4w7oVNk-metadata-4BtSAJUBTUf2G4w7JFLF-metadata]: version conflict, required seqNo [19], primary term [1]. current document has seqNo [38] and primary term [1]
opensearch-1  | 	... 8 more
opensearch-1  | [2025-02-13T17:55:00,700][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [answer]];
opensearch-1  | [2025-02-13T17:55:00,700][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [dns.question.registered_domain]];

tlacuache · February 13, 2025, 5:45pm

Another good news, though: the problem I mentioned about the mappings not showing up autmatically in my first command is no longer an issue either for me in v2.19.0.

tlacuache · February 13, 2025, 5:54pm

And here’s another thing I don’t understand. Here are my mappings for the DNS log rules:

But then I see these errors in the logs:

opensearch-1  | [2025-02-13T17:53:00,491][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [answer]];
opensearch-1  | [2025-02-13T17:53:00,491][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [dns.question.registered_domain]];

No field mapping can be found for the field with name answer? I don’t even see that as a list of possible fields in the field mappings (see the screenshot).

tlacuache · February 13, 2025, 9:38pm

lol, I spoke too soon above, this still does happen with v2.19.0. It must just not happen with the windows event rules I was doing, but it does happen with the Network ruleset:

dashboards-1         | Security Analytics - IndexService - createAliases: StatusCodeError: [illegal_argument_exception] The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead.
dashboards-1         |     at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
dashboards-1         |     at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
dashboards-1         |     at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
dashboards-1         |     at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
dashboards-1         |     at IncomingMessage.emit (node:events:529:35)
dashboards-1         |     at IncomingMessage.emit (node:domain:489:12)
dashboards-1         |     at endReadableNT (node:internal/streams/readable:1400:12)
dashboards-1         |     at processTicksAndRejections (node:internal/process/task_queues:82:21) {
dashboards-1         |   status: 400,
dashboards-1         |   displayName: 'BadRequest',
dashboards-1         |   path: '/_aliases',
dashboards-1         |   query: {},
dashboards-1         |   body: {
dashboards-1         |     error: {
dashboards-1         |       root_cause: [Array],
dashboards-1         |       type: 'illegal_argument_exception',
dashboards-1         |       reason: 'The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead.'
dashboards-1         |     },
dashboards-1         |     status: 400
dashboards-1         |   },
dashboards-1         |   statusCode: 400,
dashboards-1         |   response: '{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead."}],"type":"illegal_argument_exception","reason":"The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead."},"status":400}',
dashboards-1         |   toString: [Function (anonymous)],
dashboards-1         |   toJSON: [Function (anonymous)]
dashboards-1         | }

tlacuache · February 13, 2025, 9:40pm

I’m just going to start logging bugs on github for all of this stuff.

As both slack and here seem to be dead, maybe I’ll get more eyes on it there.

cwperks · February 15, 2025, 10:34pm

Sorry for not responding earlier. I’m not an expert in the Security Analytics plugin, but I think I know where the error message is coming from and posted on the Github issue: [BUG] The provided expression matches an alias, specify the corresponding concrete indices instead. · Issue #1477 · opensearch-project/security-analytics · GitHub

Thank you for providing the steps to reproduce!

KateWinslet · February 27, 2025, 5:41pm

tlacuache:

I’m finding some things that don’t seem to be working the way I expect them to and I’m hoping someone smarter than me can tell me why.

First, to keep track of what I’ve read on the forums and in the docs so far:

this response from 2023 from @orazaly1508

this response to that same thread by @pdz

this comment by @sbcd90 (actually, the entire thread) which eventually led me to create github issue, although I’m not sure 100% that it’s related. I only bring it up because @sbcd90 in his comment I linked says “Security Analytics Detectors are compatible only with Index Aliases, not Index Patterns” and @orazaly1508 says in her comment I linked says “Yes, mapping APIs support data streams and index patterns” which seems to be contradictory to me.

mappings API documentation

So here’s what I’m trying to do. I have some of my own index templates which are made up of composable components. In my index templates, I am specifying an alias like this (don’t get hung up on the _REPLACER strings in there, those get resolved before import).

So, as my indexes are created via data ingestion, they are getting assigned to their aliases automatically:

image504×530 32.2 KB

I mention this bit about the aliases because of the comment from @sbcd90 in that other thread I linked about how security analytics detectors only support aliases or data streams, which is why I’m using this alias.

I have this mapping that I want to create for my malcolm_network index alias for the network ruleset. I’m creating the mapping here and can verify via a GET API after my POST creating the mapping that it seems to be created correctly:
$ docker compose exec opensearch curl -fsSL -w '\n' -H'Content-Type: application/json' -XGET  "http://localhost:9200/_plugins/_security_analytics/mappings?index_name=malcolm_network"  | jq
{
  "malcolm_network": {
    "mappings": {
      "properties": {
        "zeek": {
          "properties": {
            "x509": {
              "properties": {
                "certificate": {
                  "properties": {
                    "serial": {
                      "type": "alias",
                      "path": "zeek.x509.certificate_serial"
                    }
                  }
                }
              }
            }
          }
        },
        "service": {
          "type": "alias",
          "path": "network.protocol"
        },
        "rejected": {
          "type": "alias",
          "path": "zeek.dns.rejected"
        },
        "id": {
          "properties": {
            "resp_p": {
              "type": "alias",
              "path": "destination.port"
            },
            "resp_h": {
              "type": "alias",
              "path": "destination.ip"
            },
            "orig_h": {
              "type": "alias",
              "path": "source.ip"
            }
          }
        },
        "Z-flag": {
          "type": "alias",
          "path": "zeek.dns.Z"
        }
      }
    }
  }
}
So far so good. So then I go into the UI to create a detector. I do the following:

Give it a name

Select the malcolm_network alias as the data source

Select Network as the log type

This is where I hit my first snag:

image1031×1411 102 KB

I’ve already created these mappings! Why do they show up as being unmapped here? The weird thing is, if I go in and manually populate the mappings here and then continue on, then make the exact same GET API call I posted above later on, I get the exact same thing I’ve already created. The point of me creating these mappings in the API, I thought, was so they’d be used here in this Field Mapping section.

Continuing on, I uncheck the threat intel feeds checkbox because I’m not getting into that yet, then click Next.

Next, I removed the trigger, because I’m not getting into that yet either. So I click to create the detector and it says it’s attempting to create it.

A few minutes later, I look in my logs and see these errors. First these three lines:
[2025-02-10T19:05:50,739][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [netflow.firewall_event]];
[2025-02-10T19:05:50,739][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [netflow.firewall_event]];
[2025-02-10T19:05:50,739][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [blocked-flag]];
That makes sense to me, because I didn’t have mappings for those three fields, so I ignore those.

Then this:
Security Analytics - IndexService - createAliases: StatusCodeError: [illegal_argument_exception] The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead.
    at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (node:events:529:35)
    at IncomingMessage.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1400:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21) {
  status: 400,
  displayName: 'BadRequest',
  path: '/_aliases',
  query: {},
  body: {
    error: {
      root_cause: [Array],
      type: 'illegal_argument_exception',
      reason: 'The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead.'
    },
    status: 400
  },
  statusCode: 400,
  response: '{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead."}],"type":"illegal_argument_exception","reason":"The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead."},"status":400}',
  toString: [Function (anonymous)],
  toJSON: [Function (anonymous)]
}
What stands out to me here is this message:

The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead.

Um… the UI itself, and the documentation, and about everything else I’ve read says that I’m supposed to use index aliases. Why am I getting this error?

For what it’s worth, even after getting this error I can go into the threat detectors list and see my detector:

image1086×512 36.3 KB

Check Lesco Bill
So I guess it’s working?

TL;DR, though:

Why don’t the mappings I created via the API show up as pre-populated in the UI when I go to create the detector.

Why do I see a “The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead.” error message when everything the (very limited) documentation and what I’ve read in the forums here indicates I’m supposed to use aliases?

Thanks!

The UI may not immediately reflect API-defined mappings due to caching or timing issues. Regarding the alias error, Security Analytics likely requires concrete indices instead of aliases for detectors. Try using the actual index names instead of the alias (malcolm_network) when creating the detector to avoid the “matches an alias” error.

tlacuache · February 27, 2025, 6:10pm

Respectfully, that’s not what the documentation says, and I’ve read multiple things by multiple people (in comments on these forums) and in what seems to me to be contradictory documentation. For example:

In the Data source section, select one or more sources for the log data. Use an asterisk (*) to indicate a wildcard pattern. When selecting multiple data sources, their logs must be of the same type. It is good practice to create separate detectors for different log types. Security Analytics also supports the following:

Aliases: When configuring an alias as the data source, it must be attached to a Write index alias. When using an alias, ensure that your documents are ingested through the alias and not through the index for which the alias was created.

Data streams: A set of time-series data stored across multiple indexes but called using a single named resource.

There it indicates index wildcards are supported, which some of the comments I linked at the beginning of this thread indicate are not supported.

However, one thing I hadn’t noticed in the documentation:

Aliases: When configuring an alias as the data source, it must be attached to a Write index alias. When using an alias, ensure that your documents are ingested through the alias and not through the index for which the alias was created.

Excuse my ignorance, but what is a write index alias and how do I define it? I’m assuming it’s different than what I’m doing here, which maybe is my problem?

As far as “Try using the actual index names” instead of an alias, I feel like this totally defeats the point of security analytics detectors, if I have to create them for each index. My data is time-series data (typically one index per day) and I want the detectors to run against all the applicable data as it’s written.

tlacuache · March 17, 2025, 5:05pm

Bumping so it doesn’t go stale.

tallyoh · March 18, 2025, 9:16pm

Thanks for posting. I’m having similar issues. I appreciate you taking the time to document. I’m interested to see if anyone has any ideas…

tlacuache · March 19, 2025, 2:12am

I’m glad to hear I’m not the only one! I saw the issues I logged on github have at least been moved into triage so hopefully they’ll get looked at. I’ve been doing this long enough to know that 90% of the time the problem is me, which is why I try to be thorough in documenting what I’m doing: so people can point out what my wrong assumptions are. But in the 10% of cases where the problem is actually not me that’s helpful anyway.

Topic		Replies	Views
Best practices for daily-roated-indexes Security Analytics	2	33	February 26, 2025
Mapping API / index pattern Security Analytics	9	987	June 4, 2023
Security Analytics error when using Datastreams Security Analytics	7	901	September 29, 2024
Detector fields mapping issues for OpenSearch v2.5 Security Analytics	4	827	January 5, 2024
Tenant Indices Don't Work As Expected - Prevents Reporting Security troubleshoot , configure	3	741	April 4, 2024

Trouble with mappings, detectors, and alerts

Related topics