Something broke with index templates in 2.5.0

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch-2.5.0-1.x86_64

Describe the issue:

After the update to 2.5.0, winlogbeat index templates are not getting applied to new indexes.

Configuration:

I have an index template:

{
  "winlogbeat-7.12.1" : {
    "order" : 1,
    "index_patterns" : [
      "winlogbeat-7.12.1-*"
    ],
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "max_docvalue_fields_search" : "200",
        "query" : {
          "default_field" : [
            "message",
            "tags",
            "agent.ephemeral_id",
...

That seems to no longer be applied.

Relevant Logs or Screenshots:

New index settings:

# curl -X GET "https://XXX:9200/winlogbeat-7.12.1-2023.01.25/_settings?pretty" 
{
  "winlogbeat-7.12.1-2023.01.25" : {
    "settings" : {
      "index" : {
        "creation_date" : "1674604801637",
        "number_of_shards" : "1",
        "number_of_replicas" : "1",
        "uuid" : "sXqVcTjUTqWfiugG-4uBoA",
        "version" : {
          "created" : "136267827"
        },
        "provided_name" : "winlogbeat-7.12.1-2023.01.25"
      }
    }
  }
}

Previous index setting:

# curl -X GET "https://XXX:9200/winlogbeat-7.12.1-2023.01.24/_settings?pretty" 
{
  "winlogbeat-7.12.1-2023.01.24" : {
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "provided_name" : "winlogbeat-7.12.1-2023.01.24",
        "max_docvalue_fields_search" : "200",
        "query" : {
          "default_field" : [
            "message",
            "tags",
            "agent.ephemeral_id",
...

The issue seems to be that I have overlapping index template patterns:

winlogbeat-7.12.1                                         [winlogbeat-7.12.1-*] 1
.opensearch-sap-alias-mappings-index-template-winlogbeat- [winlogbeat-*] 0                [.opensearch-sap-alias-mappings-component-winlogbeat-]

and now:

[2023-01-24T09:15:29,837][WARN ][o.o.c.m.MetadataIndexTemplateService] [os-sea01.nwra.com] index template [.opensearch-sap-alias-mappings-index-template-winlogbeat-] has index patterns [winlogbeat-*] matching patterns from existing older templates [winlogbeat-7.12.1] with patterns (winlogbeat-7.12.1 => [winlogbeat-7.12.1-*]); this template [.opensearch-sap-alias-mappings-index-template-winlogbeat-] will take precedence during new index creation

which seems to be new behavior.

If I try to delete that template though I get a 404 error:

# curl -X DELETE "https://XXX:9200/_template/.opensearch-sap-alias-mappings-index-template-winlogbeat-"
{"error":{"root_cause":[{"type":"index_template_missing_exception","reason":"index_template [.opensearch-sap-alias-mappings-index-template-winlogbeat-] missing"}],"type":"index_template_missing_exception","reason":"index_template [.opensearch-sap-alias-mappings-index-template-winlogbeat-] missing"},"status":404}

Ah, I had the call wrong, this worked:

DELETE /_index_template/.opensearch-sap-alias-mappings-index-template-winlogbeat-

I have no idea where the .opensearch-sap… index template came from. I also have:

.opensearch-sap-detectors-queries-index-template          [.opensearch-sap-s3-detectors-queries*, .opensearch-sap-others_compliance-detectors-queries*, .opensearch-sap-others_application-detectors-queries*, .opensearch-sap-dns-detectors-queries*, .opensearch-sap-others_cloud-detectors-queries*, .opensearch-sap-others_web-detectors-queries*, .opensearch-sap-windows-detectors-queries*, .opensearch-sap-cloudtrail-detectors-queries*, .opensearch-sap-others_macos-detectors-queries*, .opensearch-sap-ad_ldap-detectors-queries*, .opensearch-sap-test_windows-detectors-queries*, .opensearch-sap-network-detectors-queries*, .opensearch-sap-apache_access-detectors-queries*, .opensearch-sap-linux-detectors-queries*, .opensearch-sap-others_apt-detectors-queries*, .opensearch-sap-others_proxy-detectors-queries*] 0                

Ah, I know now - this came from my attempt to create a security analytics detector with an index pattern of winlogbeat-*. That was an unexpected effect.

@opoplawski starting from 2.5 when detector is created with index pattern, alias or datastream, index template with alias mappings (as component template) will be created, to maintain user-selected field aliases in future rollovers/additions of new indices in pattern/alias/datastream.

@pdz can you elaborate on this please?

I am trying to find some documentation on this process because creating detectors is failing for me.
Probably due to using datastreams and overly complex WinlogBeat 8.5 index template.

Alex

can you paste errors here please?

This is from Opensearch-Dashboards (sorry my container is still called Kibana as it’s shorter :slight_smile: ):

kibana  | Security Analytics - DetectorsService - getDetector: StatusCodeError: [security_analytics_exception] Failed applying mappings to index
kibana  |     at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
kibana  |     at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
kibana  |     at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
kibana  |     at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
kibana  |     at IncomingMessage.emit (events.js:412:35)
kibana  |     at IncomingMessage.emit (domain.js:475:12)
kibana  |     at endReadableNT (internal/streams/readable.js:1333:12)
kibana  |     at processTicksAndRejections (internal/process/task_queues.js:82:21) {
kibana  |   status: 500,
kibana  |   displayName: 'InternalServerError',
kibana  |   path: '/_plugins/_security_analytics/mappings',
kibana  |   query: {},
kibana  |   body: {
kibana  |     error: {
kibana  |       root_cause: [Array],
kibana  |       type: 'security_analytics_exception',
kibana  |       reason: 'Failed applying mappings to index',
kibana  |       caused_by: [Object]
kibana  |     },
kibana  |     status: 500
kibana  |   },
kibana  |   statusCode: 500,
kibana  |   response: '{"error":{"root_cause":[{"type":"security_analytics_exception","reason":"Failed applying mappings to index"}],"type":"security_analytics_exception","reason":"Failed applying mappings to index","caused_by":{"type":"
illegal_argument_exception","reason":"Limit of total fields [2000] has been exceeded"}},"status":500}',
kibana  |   toString: [Function (anonymous)],
kibana  |   toJSON: [Function (anonymous)]
kibana  | }
kibana  | {"type":"response","@timestamp":"2023-01-28T20:20:20Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/_plugins/_security_analytics/mappings","method":"post","headers":{"host":"logging","user-agen
t":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0","accept":"*/*","accept-language":"de,en-US;q=0.7,en;q=0.3","accept-encoding":"gzip, deflate, br","referer":"https://logging/app/opensearch
_security_analytics_dashboards","content-type":"application/json","osd-version":"2.5.0","content-length":"243","origin":"https://logging","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","te":"
trailers","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-for":"172.16.1.199","connection":"close"},"remoteAddress":"172.18.0.1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/10
9.0","referer":"https://logging/app/opensearch_security_analytics_dashboards"},"res":{"statusCode":200,"responseTime":830,"contentLength":9},"message":"POST /_plugins/_security_analytics/mappings 200 830ms - 9.0B"}

This is from Opensearch:

[2023-01-28T20:20:20,799][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000018/Bwk7woWaRgSmBaiETS4M4g]
[2023-01-28T20:20:20,840][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000002/2dS3N0UJRs2xu2WMIdBAVg]
[2023-01-28T20:20:20,873][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000015/P3McMsnzTLerXNSMrwGVlw]
[2023-01-28T20:20:20,912][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000006/d1-jtlPJS8yjZIR-6YfHug]
[2023-01-28T20:20:20,943][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000001/xEHdG6DYR9ukD5kqZSMxmg]
[2023-01-28T20:20:20,976][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000010/ncZT3zNpROWpnIy81ouV3Q]
[2023-01-28T20:20:21,010][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000011/3LVjz60KSKyqzWTZDLhdCw]
[2023-01-28T20:20:21,046][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000005/wNIp48dyTOCeiYIeQORemA]
[2023-01-28T20:20:21,076][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000007/gZH7-DsIRveiCdOBaZwT9w]
[2023-01-28T20:20:21,108][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000009/vWPMsT55Svi2VQtwLdrXrA]
[2023-01-28T20:20:21,139][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000013/Z5uOmashQ7W20fCxvooRlg]
[2023-01-28T20:20:21,171][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000019/Kj3d1VFWRTK7E1fP6pxh-A]
[2023-01-28T20:20:21,206][INFO ][o.o.c.m.MetadataMappingService] [opensearch] [.ds-winlogbeat-000019/Kj3d1VFWRTK7E1fP6pxh-A] update_mapping [_doc]
[2023-01-28T20:20:21,208][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000003/B96TiycDR3aWyPFx2RMsqw]
[2023-01-28T20:20:21,248][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000008/5FdbCqgBSL2mCQUO8-a83A]
[2023-01-28T20:20:21,285][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000012/i4U1ehmnT6qLQCPX_Yv6Ew]
[2023-01-28T20:20:21,320][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000014/K3SAMkqmRNKTtLlv1zkRnw]
[2023-01-28T20:20:21,356][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000017/hXlM3qUTR--JZYXisWvFvg]
[2023-01-28T20:20:21,391][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000004/Hc3xJXweQza5nVSL0UX1uQ]
[2023-01-28T20:20:21,422][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000016/pzyx_bi4QtC5kZieqxJDMQ]
[2023-01-28T20:20:21,503][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch] Detected cluster change event for destination migration
[2023-01-28T20:20:21,503][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch] Reset destination migration process.
[2023-01-28T20:20:21,504][WARN ][r.suppressed             ] [opensearch] path: /_plugins/_security_analytics/mappings, params: {}
org.opensearch.securityanalytics.util.SecurityAnalyticsException: Failed applying mappings to index
        at org.opensearch.securityanalytics.mapper.MapperService$2.onFailure(MapperService.java:128) [opensearch-security-analytics-2.5.0.0.jar:2.5.0.0]
        at org.opensearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:78) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.securityanalytics.mapper.MapperService$3.onResponse(MapperService.java:215) [opensearch-security-analytics-2.5.0.0.jar:2.5.0.0]
        at org.opensearch.securityanalytics.mapper.MapperService$3.onResponse(MapperService.java:207) [opensearch-security-analytics-2.5.0.0.jar:2.5.0.0]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:113) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:107) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.action.support.RetryableAction$RetryingListener.onResponse(RetryableAction.java:181) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.action.ActionListener$2.onResponse(ActionListener.java:106) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.action.admin.indices.mapping.put.TransportPutMappingAction$1.onResponse(TransportPutMappingAction.java:180) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.action.admin.indices.mapping.put.TransportPutMappingAction$1.onResponse(TransportPutMappingAction.java:176) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.metadata.MetadataMappingService$1.onAllNodesAcked(MetadataMappingService.java:375) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.MasterService$SafeAckedClusterStateTaskListener.onAllNodesAcked(MasterService.java:717) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.MasterService$AckCountDownListener.finish(MasterService.java:852) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.MasterService$AckCountDownListener.onNodeAck(MasterService.java:843) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.MasterService$DelegatingAckListener.onNodeAck(MasterService.java:765) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.coordination.Coordinator$CoordinatorPublication$4$1.onSuccess(Coordinator.java:1709) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.ClusterApplierService$SafeClusterApplyListener.onSuccess(ClusterApplierService.java:657) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:494) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:186) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) [opensearch-2.5.0.jar:2.5.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
Caused by: java.lang.IllegalArgumentException: Limit of total fields [2000] has been exceeded
        at org.opensearch.index.mapper.MappingLookup.checkFieldLimit(MappingLookup.java:192) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.index.mapper.MappingLookup.checkLimits(MappingLookup.java:184) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.index.mapper.DocumentMapper.validate(DocumentMapper.java:329) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:491) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:447) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.index.mapper.MapperService.merge(MapperService.java:419) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.metadata.MetadataMappingService$PutMappingExecutor.applyRequest(MetadataMappingService.java:300) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.metadata.MetadataMappingService$PutMappingExecutor.execute(MetadataMappingService.java:244) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:867) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:190) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:228) ~[opensearch-2.5.0.jar:2.5.0]
        ... 6 more

So I guess by using the current WinlogBeat ECS template, I get close to the 1000 fields limit of the index. And some change by the Security Analytics module is putting that mark even a lot higher - I already adjusted the index template to be 2000.

What’s bothersome about this is that Security Analytics is now setup in some way but I don’t yet have a detector I can query via API. Some jobs are already at work behind the scenes though:

[2023-01-28T20:26:29,571][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.opensearch-sap-ad_ldap-alerts/vV5Ch9CKR1eis7Spm72EwQ]
[2023-01-28T20:26:29,577][INFO ][o.o.a.a.AlertIndices     ] [opensearch] Index mapping of .opensearch-sap-ad_ldap-alerts is updated
[2023-01-28T20:26:29,578][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.opensearch-sap-ad_ldap-alerts-history-2023.01.27-1/-9_iBWstRway72uKM4HjtQ]
[2023-01-28T20:26:29,584][INFO ][o.o.a.a.AlertIndices     ] [opensearch] Index mapping of .opensearch-sap-ad_ldap-alerts-history-2023.01.27-1 is updated
[2023-01-28T20:26:29,594][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.opensearch-sap-ad_ldap-findings-2023.01.27-1/UJuKxD7DQgWt5fj4SbRqrA]
[2023-01-28T20:26:29,596][INFO ][o.o.a.a.AlertIndices     ] [opensearch] Index mapping of .opensearch-sap-ad_ldap-findings-2023.01.27-1 is updated
[2023-01-28T20:26:29,610][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.opensearch-sap-ad_ldap-detectors-queries-000001/pmYVYK5PTpSGKBic4sLTtA]
[2023-01-28T20:26:29,615][ERROR][o.o.a.u.AlertingException] [opensearch] Alerting error: java.lang.IllegalArgumentException: analyzer [rule_analyzer] has not been configured in mappings
[2023-01-28T20:26:29,616][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch] uncaught exception in thread [DefaultDispatcher-worker-2]
org.opensearch.alerting.util.AlertingException: analyzer [rule_analyzer] has not been configured in mappings
        at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70) ~[opensearch-alerting-2.5.0.0.jar:2.5.0.0]
        at org.opensearch.alerting.util.DocLevelMonitorQueries.updateQueryIndexMappings(DocLevelMonitorQueries.kt:359) ~[opensearch-alerting-2.5.0.0.jar:2.5.0.0]
        at org.opensearch.alerting.util.DocLevelMonitorQueries.access$updateQueryIndexMappings(DocLevelMonitorQueries.kt:41) ~[opensearch-alerting-2.5.0.0.jar:2.5.0.0]
        at org.opensearch.alerting.util.DocLevelMonitorQueries$updateQueryIndexMappings$1.invokeSuspend(DocLevelMonitorQueries.kt) ~[opensearch-alerting-2.5.0.0.jar:2.5.0.0]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
        at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:285) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
Caused by: java.lang.Exception: java.lang.IllegalArgumentException: analyzer [rule_analyzer] has not been configured in mappings
        ... 9 more
uncaught exception in thread [DefaultDispatcher-worker-2]
AlertingException[analyzer [rule_analyzer] has not been configured in mappings]; nested: Exception[java.lang.IllegalArgumentException: analyzer [rule_analyzer] has not been configured in mappings];
        at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70)
        at org.opensearch.alerting.util.DocLevelMonitorQueries.updateQueryIndexMappings(DocLevelMonitorQueries.kt:359)
        at org.opensearch.alerting.util.DocLevelMonitorQueries.access$updateQueryIndexMappings(DocLevelMonitorQueries.kt:41)
        at org.opensearch.alerting.util.DocLevelMonitorQueries$updateQueryIndexMappings$1.invokeSuspend(DocLevelMonitorQueries.kt)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:285)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742)
Caused by: java.lang.Exception: java.lang.IllegalArgumentException: analyzer [rule_analyzer] has not been configured in mappings
        ... 9 more

When trying this first, the .opensearch-sap-ad_ldap-detectors-queries index (and others) was rolling over every minute, which pretty soon brought my system to the 1000 shards limit :frowning:

This was the error I received before: Security Analytics error when using Datastreams

I’d be glad to provide any further information - this really rocks and I would love to start using it.

Alex

Unfortunately, there is a bug when source_index has more mappings then default max number of mappings(1000). Until this is fixed in next version, you can resolve it by adding following index template:

PUT _index_template/opensearch-sap-query-index-temp
{
  "index_patterns": [
    ".opensearch-sap-s3-detectors-queries*",
    ".opensearch-sap-others_compliance-detectors-queries*",
    ".opensearch-sap-others_application-detectors-queries*",
    ".opensearch-sap-dns-detectors-queries*",
    ".opensearch-sap-others_cloud-detectors-queries*",
    ".opensearch-sap-others_web-detectors-queries*",
    ".opensearch-sap-windows-detectors-queries*",
    ".opensearch-sap-cloudtrail-detectors-queries*",
    ".opensearch-sap-others_macos-detectors-queries*",
    ".opensearch-sap-ad_ldap-detectors-queries*",
    ".opensearch-sap-network-detectors-queries*",
    ".opensearch-sap-apache_access-detectors-queries*",
    ".opensearch-sap-linux-detectors-queries*",
    ".opensearch-sap-others_apt-detectors-queries*",
    ".opensearch-sap-others_proxy-detectors-queries*"
  ],
  "template": {
    "settings": {
      "index" : {
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "analysis": {
          "analyzer": {
            "rule_analyzer": {
              "char_filter": [
                "rule_ws_filter"
              ],
              "tokenizer": "keyword"
            }
          },
          "char_filter": {
            "rule_ws_filter": {
              "pattern": "(_ws_)",
              "type": "pattern_replace",
              "replacement": " "
            }
          }
        },
        "hidden": "true"
      }
    },
    "mappings": {},
    "aliases": {}
  },
  "priority": 500
}
1 Like

Thank you (also for providing the answer to my Datastream question)!

Now I get:

kibana  |   status: 400,
kibana  |   displayName: 'BadRequest',
kibana  |   path: '/_plugins/_security_analytics/mappings',
kibana  |   query: {},
kibana  |   body: {
kibana  |     error: {
kibana  |       root_cause: [Array],
kibana  |       type: 'illegal_argument_exception',
kibana  |       reason: 'composable template [winlogbeat] template after composition with component templates [.opensearch-sap-alias-mappings-component-winlogbeat] is invalid',
kibana  |       caused_by: [Object]
kibana  |     },

There is no further error in Opensearch :frowning:
When I google this error I only get a reference to some change in the timestamp field for datastreams in ElasticSearch 8: Component templates are considered invalid without timestamp_field · Issue #58956 · elastic/elasticsearch · GitHub

How is your datastream index template setup? Can please provide steps you used.

So much to learn - I didn’t realize that _template and _index_template are two different things :slight_smile:
I checked and there are no more legacy templates (although I might have used them before).

I export the winlogbeat index template like this:

./winlogbeat-8.5.3.exe export template | \
 sed -E 's/winlogbeat-[^"]*/winlogbeat/g;s/flattened/keyword/g;s/constant_keyword/keyword/g;s/match_only_text/text/g;s/wildcard/keyword/g' | \
 jq 'del( .template.settings.index.lifecycle ) | .template.settings.index.default_pipeline |= "winlogbeat-routing"' \
 > winlogbeat_index-template.json

curl -X POST http://localhost:9200/_index_template/winlogbeat -H 'content-type: application/json' -d @winlogbeat_index-template.json

POSTing this to _index_template/winlogbeat I get something like
GET _index_template/winlogbeat

{
  "index_templates": [
    {
      "name": "winlogbeat",
      "index_template": {
        "index_patterns": [
          "winlogbeat"
        ],
        "template": {
          "settings": {
            "index": {
              "mapping": {
                "total_fields": {
                  "limit": "10000"
                }
              },
              "refresh_interval": "5s",
              "number_of_shards": "1",
              "max_docvalue_fields_search": "200",
              "query": {...},
              "default_pipeline": "winlogbeat-routing",
              "number_of_replicas": "0"
            }
          },
          "mappings": {...}
        },
        "composed_of": [],
        "priority": 150,
        "data_stream": {
          "timestamp_field": {
            "name": "@timestamp"
          }
        }
      }
    }
  ]
}

Index pattern and ingestion pipelines are exported from winlogbeat.exe as well.
There is a manual ISM policy rolling over any datastream index daily.

Even if I add the composed index reference like this, the error remains the same:

  "composed_of": [ ".opensearch-sap-alias-mappings-component-winlogbeat"]

Thanks again for all your help!

When creating a detector, SAP should find existing template for your datastream and insert that alias mappings component template into composed_of array.

Can you please do:

GET _component_template/.opensearch-sap-alias-mappings-component-winlogbeat

and paste results here?

If you don’t have any important data on SA plugin, I would recommend following:

  1. delete all “.opensearch-sap*” indices
  2. Apply this template:
PUT _index_template/opensearch-sap-query-index-temp
{
  "index_patterns": [
    ".opensearch-sap-s3-detectors-queries*",
    ".opensearch-sap-others_compliance-detectors-queries*",
    ".opensearch-sap-others_application-detectors-queries*",
    ".opensearch-sap-dns-detectors-queries*",
    ".opensearch-sap-others_cloud-detectors-queries*",
    ".opensearch-sap-others_web-detectors-queries*",
    ".opensearch-sap-windows-detectors-queries*",
    ".opensearch-sap-cloudtrail-detectors-queries*",
    ".opensearch-sap-others_macos-detectors-queries*",
    ".opensearch-sap-ad_ldap-detectors-queries*",
    ".opensearch-sap-network-detectors-queries*",
    ".opensearch-sap-apache_access-detectors-queries*",
    ".opensearch-sap-linux-detectors-queries*",
    ".opensearch-sap-others_apt-detectors-queries*",
    ".opensearch-sap-others_proxy-detectors-queries*"
  ],
  "template": {
    "settings": {
      "index" : {
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "analysis": {
          "analyzer": {
            "rule_analyzer": {
              "char_filter": [
                "rule_ws_filter"
              ],
              "tokenizer": "keyword"
            }
          },
          "char_filter": {
            "rule_ws_filter": {
              "pattern": "(_ws_)",
              "type": "pattern_replace",
              "replacement": " "
            }
          }
        },
        "hidden": "true"
      }
    },
    "mappings": {},
    "aliases": {}
  },
  "priority": 500
}
  1. try creating detector
{
  "component_templates": [
    {
      "name": ".opensearch-sap-alias-mappings-component-winlogbeat",
      "component_template": {
        "template": {
          "mappings": {
            "properties": {
              "winlog.event_data.AuthenticationPackageName": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-SubjectLogonId": {
                "path": "winlog.event_data.SubjectLogonId",
                "type": "alias"
              },
              "winlog.event_data.LogonId": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "server.user.hash": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-Description": {
                "path": "winlog.event_data.Description",
                "type": "alias"
              },
              "winlog-event_data-param1": {
                "path": "winlog.event_data.param1",
                "type": "alias"
              },
              "winlog.event_data.IpAddress": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-param2": {
                "path": "winlog.event_data.param2",
                "type": "alias"
              },
              "winlog-user-type": {
                "path": "winlog.user.type",
                "type": "alias"
              },
              "winlog.provider_name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.event_data.Signed": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "host.hostname": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.computer_name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-TargetUserName": {
                "path": "winlog.event_data.TargetUserName",
                "type": "alias"
              },
              "winlog.event_data.KeyLength": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "windows-message": {
                "path": "message",
                "type": "alias"
              },
              "winlog-event_data-Signed": {
                "path": "winlog.event_data.Signed",
                "type": "alias"
              },
              "winlog-computerObject-name": {
                "path": "winlog.computerObject.name",
                "type": "alias"
              },
              "winlog.keywords": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.event_data.DeviceName": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-ScriptBlockText": {
                "path": "winlog.event_data.ScriptBlockText",
                "type": "alias"
              },
              "winlog-event_data-DeviceName": {
                "path": "winlog.event_data.DeviceName",
                "type": "alias"
              },
              "winlog-event_data-ProcessId": {
                "path": "winlog.event_data.ProcessId",
                "type": "alias"
              },
              "winlog.event_data.OriginalFileName": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.event_data.Detail": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-LogonId": {
                "path": "winlog.event_data.LogonId",
                "type": "alias"
              },
              "winlog-event_data-AuthenticationPackageName": {
                "path": "winlog.event_data.AuthenticationPackageName",
                "type": "alias"
              },
              "winlog.event_data.Company": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_id": {
                "path": "winlog.event_id",
                "type": "alias"
              },
              "winlog.event_data.TargetUserName": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-OriginalFileName": {
                "path": "winlog.event_data.OriginalFileName",
                "type": "alias"
              },
              "winlog.event_data.TargetServerName": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.event_data.TargetUserSid": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-Product": {
                "path": "winlog.event_data.Product",
                "type": "alias"
              },
              "winlog.event_data.param1": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.event_data.param2": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-FileVersion": {
                "path": "winlog.event_data.FileVersion",
                "type": "alias"
              },
              "winlog-event_data-ServiceName": {
                "path": "winlog.event_data.ServiceName",
                "type": "alias"
              },
              "winlog-event_data-LogonProcessName": {
                "path": "winlog.event_data.LogonProcessName",
                "type": "alias"
              },
              "winlog-event_data-PrivilegeList": {
                "path": "winlog.event_data.PrivilegeList",
                "type": "alias"
              },
              "winlog.event_data.FileVersion": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-Status": {
                "path": "winlog.event_data.Status",
                "type": "alias"
              },
              "winlog-event_data-Detail": {
                "path": "winlog.event_data.Detail",
                "type": "alias"
              },
              "winlog-channel": {
                "path": "winlog.channel",
                "type": "alias"
              },
              "winlog.event_data.State": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.event_data.ServiceName": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-State": {
                "path": "winlog.event_data.State",
                "type": "alias"
              },
              "winlog.event_data.SubjectUserSid": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.event_data.Description": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.computerObject.name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.event_data.PrivilegeList": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-IpAddress": {
                "path": "winlog.event_data.IpAddress",
                "type": "alias"
              },
              "winlog-event_data-TargetLogonId": {
                "path": "winlog.event_data.TargetLogonId",
                "type": "alias"
              },
              "winlog-provider_name": {
                "path": "winlog.provider_name",
                "type": "alias"
              },
              "server-user-hash": {
                "path": "server.user.hash",
                "type": "alias"
              },
              "winlog-computer_name": {
                "path": "winlog.computer_name",
                "type": "alias"
              },
              "winlog-event_data-KeyLength": {
                "path": "winlog.event_data.KeyLength",
                "type": "alias"
              },
              "winlog-keywords": {
                "path": "winlog.keywords",
                "type": "alias"
              },
              "winlog.user.type": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.event_data.ScriptBlockText": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.event_data.Status": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.event_data.SubjectLogonId": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.event_data.TargetLogonId": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "host-hostname": {
                "path": "host.hostname",
                "type": "alias"
              },
              "winlog-task": {
                "path": "winlog.task",
                "type": "alias"
              },
              "winlog-event_data-TargetUserSid": {
                "path": "winlog.event_data.TargetUserSid",
                "type": "alias"
              },
              "winlog.event_data.ProcessId": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "timestamp": {
                "path": "@timestamp",
                "type": "alias"
              },
              "winlog-event_data-ProcessPath": {
                "path": "winlog.event_data.ProcessPath",
                "type": "alias"
              },
              "winlog.task": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-user-name": {
                "path": "winlog.user.name",
                "type": "alias"
              },
              "winlog.event_data.ProcessPath": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-SubjectUserName": {
                "path": "winlog.event_data.SubjectUserName",
                "type": "alias"
              },
              "winlog.event_data.SubjectUserName": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.event_data.IntegrityLevel": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-LogonType": {
                "path": "winlog.event_data.LogonType",
                "type": "alias"
              },
              "winlog-event_data-SubjectUserSid": {
                "path": "winlog.event_data.SubjectUserSid",
                "type": "alias"
              },
              "winlog.user.name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.event_id": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.event_data.LogonProcessName": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "message": {
                "type": "text"
              },
              "winlog.event_data.Workstation": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "@timestamp": {
                "type": "date"
              },
              "winlog-event_data-IntegrityLevel": {
                "path": "winlog.event_data.IntegrityLevel",
                "type": "alias"
              },
              "winlog.event_data.LogonType": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.channel": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-SubjectDomainName": {
                "path": "winlog.event_data.SubjectDomainName",
                "type": "alias"
              },
              "winlog.event_data.Path": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-Path": {
                "path": "winlog.event_data.Path",
                "type": "alias"
              },
              "winlog.event_data.SubjectDomainName": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-TargetServerName": {
                "path": "winlog.event_data.TargetServerName",
                "type": "alias"
              },
              "winlog-event_data-ProcessName": {
                "path": "winlog.event_data.ProcessName",
                "type": "alias"
              },
              "winlog.event_data.Product": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog.event_data.ProcessName": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "winlog-event_data-Company": {
                "path": "winlog.event_data.Company",
                "type": "alias"
              },
              "winlog-event_data-Workstation": {
                "path": "winlog.event_data.Workstation",
                "type": "alias"
              }
            }
          }
        },
        "version": 0
      }
    }
  ]
}

I already performed steps 1.-3. before - just tried again now with the same error as before.
While the component_template with the alias mapping is created, it does not get added to the winlogbeat index template. As I posted above, it does not help either if I added it manually to the composed_of array :frowning:

Yea there is an issue with handling that template field inside index_template.

Can you use concrete indices, until this is fixed? With concrete indices use index pattern: winlog*

1 Like

:smiley: I will have to…

Thank you for all the work! I am looking forward to having this work with data_streams.

If there is anything I can do to find a fix for this, please let me know!

I just tried that on a production workload.

  • Started from scratch (fresh cluster)
  • created opensearch-sap-query-index-temp
  • modified winlogbeat index-template (not containing data_stream references) - this worked on my test setup
  • switched to index-ingestion in FluentD

I now have a winlogbeat-YYYY-MM-DD index and I am using winlogbeat* as pattern for the detector.

Unfortunately the error still turns up:

  path: '/_plugins/_security_analytics/mappings',
  query: {},
  body: {
    error: {
      root_cause: [Array],
      type: 'illegal_argument_exception',
      reason: 'composable template [winlogbeat] template after composition with component templates [.opensearch-sap-alias-mappings-component-winlogbeat] is invalid',
      caused_by: [Object]
    },
    status: 400
  },

Oops, sorry I missed this message.

You have to create index template with component template (put it in composed_of array). There is issue with template field in index templates in 2.5

My winlogbeat index template looks something like this now:

PUT _index_template/winlogbeat
{
        "index_patterns": [
          "winlogbeat*"
        ],
        "template": {
          "settings": {
            "index": {
              "mapping": {
                "total_fields": {
                  "limit": "10000"
                }
              },
              "refresh_interval": "5s",
              "number_of_shards": "1",
              "max_docvalue_fields_search": "200",
              "query": {
                "default_field": [...]
              },
              "default_pipeline": "winlogbeat-routing",
              "analysis": {
                "analyzer": {
                  "winlogbeat_powershell_script_analyzer": {
                    "pattern": """[\W&&[^-]]+""",
                    "type": "pattern"
                  }
                }
              },
              "number_of_replicas": "0"
            }
          },
          "mappings": { ... }
        },
        "composed_of": [ ".opensearch-sap-alias-mappings-component-winlogbeat" ],
        "priority": 150
}

But the error remains the same :frowning:

I really appreciate all your work - if you tell me to shut up and wait for 2.6, I’ll do so :slight_smile:
Otherwise if there is something I can do please let me know!

You’re using “template” field in index template. What you can do instead is create Component Template, then add it to composed_of array of index template. Take a look at this example: link

I moved the winlogbeat index_template to a component_template, switched back to data_streams and tried again. I’ve been able to create a detector now, but I don’t know if it’s working until there’s an alert :smiley:

GET _index_template/winlogbeat
{
  "index_templates": [
    {
      "name": "winlogbeat",
      "index_template": {
        "index_patterns": [
          "winlogbeat*"
        ],
        "composed_of": [
          ".opensearch-sap-alias-mappings-component-winlogbeat",
          "winlogbeat"
        ],
        "priority": 150,
        "data_stream": {
          "timestamp_field": {
            "name": "@timestamp"
          }
        }
      }
    }
  ]
}

I am starting to realize now the issue lies with the mapping.
What happens if the first component_template defines

              "winlog.event_data.LogonId": {
                "ignore_above": 1024,
                "type": "keyword"
              }

and the second

              "winlog-event_data-LogonId": {
                "path": "winlog.event_data.LogonId",
                "type": "alias"
              }

How is the detector mapping generated anyways?

Detector is making percolate queries from parsed sigma rules. Field names used in these queries could be original field names used in Sigma rule, or if defined in mapping yml file, could be converted/mapped to new value. In your example you would need winlog-event_data-LogonId field in your log index for rule which uses this field to work. That can be either alias or “normal” field. If your log index doesn’t have field named like this, you will get a chance to create an alias field with rule field name which points to some other field in index.(during detector create/update flow)

1 Like