I’m finding some things that don’t seem to be working the way I expect them to and I’m hoping someone smarter than me can tell me why.
First, to keep track of what I’ve read on the forums and in the docs so far:
- this response from 2023 from @orazaly1508
- this response to that same thread by @pdz
- this comment by @sbcd90 (actually, the entire thread) which eventually led me to create github issue, although I’m not sure 100% that it’s related. I only bring it up because @sbcd90 in his comment I linked says “Security Analytics Detectors are compatible only with Index Aliases, not Index Patterns” and @orazaly1508 says in her comment I linked says “Yes, mapping APIs support data streams and index patterns” which seems to be contradictory to me.
- mappings API documentation
So here’s what I’m trying to do. I have some of my own index templates which are made up of composable components. In my index templates, I am specifying an alias like this (don’t get hung up on the _REPLACER strings in there, those get resolved before import).
So, as my indexes are created via data ingestion, they are getting assigned to their aliases automatically:
I mention this bit about the aliases because of the comment from @sbcd90 in that other thread I linked about how security analytics detectors only support aliases or data streams, which is why I’m using this alias.
I have this mapping that I want to create for my malcolm_network index alias for the network ruleset. I’m creating the mapping here and can verify via a GET API after my POST creating the mapping that it seems to be created correctly:
$ docker compose exec opensearch curl -fsSL -w '\n' -H'Content-Type: application/json' -XGET "http://localhost:9200/_plugins/_security_analytics/mappings?index_name=malcolm_network" | jq
{
"malcolm_network": {
"mappings": {
"properties": {
"zeek": {
"properties": {
"x509": {
"properties": {
"certificate": {
"properties": {
"serial": {
"type": "alias",
"path": "zeek.x509.certificate_serial"
}
}
}
}
}
}
},
"service": {
"type": "alias",
"path": "network.protocol"
},
"rejected": {
"type": "alias",
"path": "zeek.dns.rejected"
},
"id": {
"properties": {
"resp_p": {
"type": "alias",
"path": "destination.port"
},
"resp_h": {
"type": "alias",
"path": "destination.ip"
},
"orig_h": {
"type": "alias",
"path": "source.ip"
}
}
},
"Z-flag": {
"type": "alias",
"path": "zeek.dns.Z"
}
}
}
}
}
So far so good. So then I go into the UI to create a detector. I do the following:
- Give it a name
- Select the
malcolm_networkalias as the data source - Select
Networkas the log type
This is where I hit my first snag:
I’ve already created these mappings! Why do they show up as being unmapped here? The weird thing is, if I go in and manually populate the mappings here and then continue on, then make the exact same GET API call I posted above later on, I get the exact same thing I’ve already created. The point of me creating these mappings in the API, I thought, was so they’d be used here in this Field Mapping section.
Continuing on, I uncheck the threat intel feeds checkbox because I’m not getting into that yet, then click Next.
Next, I removed the trigger, because I’m not getting into that yet either. So I click to create the detector and it says it’s attempting to create it.
A few minutes later, I look in my logs and see these errors. First these three lines:
[2025-02-10T19:05:50,739][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [netflow.firewall_event]];
[2025-02-10T19:05:50,739][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [netflow.firewall_event]];
[2025-02-10T19:05:50,739][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [blocked-flag]];
That makes sense to me, because I didn’t have mappings for those three fields, so I ignore those.
Then this:
Security Analytics - IndexService - createAliases: StatusCodeError: [illegal_argument_exception] The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead.
at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
at IncomingMessage.emit (node:events:529:35)
at IncomingMessage.emit (node:domain:489:12)
at endReadableNT (node:internal/streams/readable:1400:12)
at processTicksAndRejections (node:internal/process/task_queues:82:21) {
status: 400,
displayName: 'BadRequest',
path: '/_aliases',
query: {},
body: {
error: {
root_cause: [Array],
type: 'illegal_argument_exception',
reason: 'The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead.'
},
status: 400
},
statusCode: 400,
response: '{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead."}],"type":"illegal_argument_exception","reason":"The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead."},"status":400}',
toString: [Function (anonymous)],
toJSON: [Function (anonymous)]
}
What stands out to me here is this message:
The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead.
Um… the UI itself, and the documentation, and about everything else I’ve read says that I’m supposed to use index aliases. Why am I getting this error?
For what it’s worth, even after getting this error I can go into the threat detectors list and see my detector:
So I guess it’s working?
TL;DR, though:
- Why don’t the mappings I created via the API show up as pre-populated in the UI when I go to create the detector.
- Why do I see a “The provided expression [malcolm_network] matches an alias, specify the corresponding concrete indices instead.” error message when everything the (very limited) documentation and what I’ve read in the forums here indicates I’m supposed to use aliases?
Thanks!
-SG