Something broke with index templates in 2.5.0

This is from Opensearch-Dashboards (sorry my container is still called Kibana as it’s shorter :slight_smile: ):

kibana  | Security Analytics - DetectorsService - getDetector: StatusCodeError: [security_analytics_exception] Failed applying mappings to index
kibana  |     at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
kibana  |     at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
kibana  |     at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
kibana  |     at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
kibana  |     at IncomingMessage.emit (events.js:412:35)
kibana  |     at IncomingMessage.emit (domain.js:475:12)
kibana  |     at endReadableNT (internal/streams/readable.js:1333:12)
kibana  |     at processTicksAndRejections (internal/process/task_queues.js:82:21) {
kibana  |   status: 500,
kibana  |   displayName: 'InternalServerError',
kibana  |   path: '/_plugins/_security_analytics/mappings',
kibana  |   query: {},
kibana  |   body: {
kibana  |     error: {
kibana  |       root_cause: [Array],
kibana  |       type: 'security_analytics_exception',
kibana  |       reason: 'Failed applying mappings to index',
kibana  |       caused_by: [Object]
kibana  |     },
kibana  |     status: 500
kibana  |   },
kibana  |   statusCode: 500,
kibana  |   response: '{"error":{"root_cause":[{"type":"security_analytics_exception","reason":"Failed applying mappings to index"}],"type":"security_analytics_exception","reason":"Failed applying mappings to index","caused_by":{"type":"
illegal_argument_exception","reason":"Limit of total fields [2000] has been exceeded"}},"status":500}',
kibana  |   toString: [Function (anonymous)],
kibana  |   toJSON: [Function (anonymous)]
kibana  | }
kibana  | {"type":"response","@timestamp":"2023-01-28T20:20:20Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/_plugins/_security_analytics/mappings","method":"post","headers":{"host":"logging","user-agen
t":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0","accept":"*/*","accept-language":"de,en-US;q=0.7,en;q=0.3","accept-encoding":"gzip, deflate, br","referer":"https://logging/app/opensearch
_security_analytics_dashboards","content-type":"application/json","osd-version":"2.5.0","content-length":"243","origin":"https://logging","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","te":"
trailers","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-for":"172.16.1.199","connection":"close"},"remoteAddress":"172.18.0.1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/10
9.0","referer":"https://logging/app/opensearch_security_analytics_dashboards"},"res":{"statusCode":200,"responseTime":830,"contentLength":9},"message":"POST /_plugins/_security_analytics/mappings 200 830ms - 9.0B"}

This is from Opensearch:

[2023-01-28T20:20:20,799][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000018/Bwk7woWaRgSmBaiETS4M4g]
[2023-01-28T20:20:20,840][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000002/2dS3N0UJRs2xu2WMIdBAVg]
[2023-01-28T20:20:20,873][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000015/P3McMsnzTLerXNSMrwGVlw]
[2023-01-28T20:20:20,912][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000006/d1-jtlPJS8yjZIR-6YfHug]
[2023-01-28T20:20:20,943][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000001/xEHdG6DYR9ukD5kqZSMxmg]
[2023-01-28T20:20:20,976][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000010/ncZT3zNpROWpnIy81ouV3Q]
[2023-01-28T20:20:21,010][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000011/3LVjz60KSKyqzWTZDLhdCw]
[2023-01-28T20:20:21,046][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000005/wNIp48dyTOCeiYIeQORemA]
[2023-01-28T20:20:21,076][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000007/gZH7-DsIRveiCdOBaZwT9w]
[2023-01-28T20:20:21,108][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000009/vWPMsT55Svi2VQtwLdrXrA]
[2023-01-28T20:20:21,139][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000013/Z5uOmashQ7W20fCxvooRlg]
[2023-01-28T20:20:21,171][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000019/Kj3d1VFWRTK7E1fP6pxh-A]
[2023-01-28T20:20:21,206][INFO ][o.o.c.m.MetadataMappingService] [opensearch] [.ds-winlogbeat-000019/Kj3d1VFWRTK7E1fP6pxh-A] update_mapping [_doc]
[2023-01-28T20:20:21,208][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000003/B96TiycDR3aWyPFx2RMsqw]
[2023-01-28T20:20:21,248][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000008/5FdbCqgBSL2mCQUO8-a83A]
[2023-01-28T20:20:21,285][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000012/i4U1ehmnT6qLQCPX_Yv6Ew]
[2023-01-28T20:20:21,320][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000014/K3SAMkqmRNKTtLlv1zkRnw]
[2023-01-28T20:20:21,356][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000017/hXlM3qUTR--JZYXisWvFvg]
[2023-01-28T20:20:21,391][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000004/Hc3xJXweQza5nVSL0UX1uQ]
[2023-01-28T20:20:21,422][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000016/pzyx_bi4QtC5kZieqxJDMQ]
[2023-01-28T20:20:21,503][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch] Detected cluster change event for destination migration
[2023-01-28T20:20:21,503][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch] Reset destination migration process.
[2023-01-28T20:20:21,504][WARN ][r.suppressed             ] [opensearch] path: /_plugins/_security_analytics/mappings, params: {}
org.opensearch.securityanalytics.util.SecurityAnalyticsException: Failed applying mappings to index
        at org.opensearch.securityanalytics.mapper.MapperService$2.onFailure(MapperService.java:128) [opensearch-security-analytics-2.5.0.0.jar:2.5.0.0]
        at org.opensearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:78) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.securityanalytics.mapper.MapperService$3.onResponse(MapperService.java:215) [opensearch-security-analytics-2.5.0.0.jar:2.5.0.0]
        at org.opensearch.securityanalytics.mapper.MapperService$3.onResponse(MapperService.java:207) [opensearch-security-analytics-2.5.0.0.jar:2.5.0.0]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:113) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:107) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.action.support.RetryableAction$RetryingListener.onResponse(RetryableAction.java:181) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.action.ActionListener$2.onResponse(ActionListener.java:106) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.action.admin.indices.mapping.put.TransportPutMappingAction$1.onResponse(TransportPutMappingAction.java:180) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.action.admin.indices.mapping.put.TransportPutMappingAction$1.onResponse(TransportPutMappingAction.java:176) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.metadata.MetadataMappingService$1.onAllNodesAcked(MetadataMappingService.java:375) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.MasterService$SafeAckedClusterStateTaskListener.onAllNodesAcked(MasterService.java:717) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.MasterService$AckCountDownListener.finish(MasterService.java:852) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.MasterService$AckCountDownListener.onNodeAck(MasterService.java:843) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.MasterService$DelegatingAckListener.onNodeAck(MasterService.java:765) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.coordination.Coordinator$CoordinatorPublication$4$1.onSuccess(Coordinator.java:1709) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.ClusterApplierService$SafeClusterApplyListener.onSuccess(ClusterApplierService.java:657) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:494) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:186) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) [opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) [opensearch-2.5.0.jar:2.5.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
Caused by: java.lang.IllegalArgumentException: Limit of total fields [2000] has been exceeded
        at org.opensearch.index.mapper.MappingLookup.checkFieldLimit(MappingLookup.java:192) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.index.mapper.MappingLookup.checkLimits(MappingLookup.java:184) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.index.mapper.DocumentMapper.validate(DocumentMapper.java:329) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:491) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:447) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.index.mapper.MapperService.merge(MapperService.java:419) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.metadata.MetadataMappingService$PutMappingExecutor.applyRequest(MetadataMappingService.java:300) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.metadata.MetadataMappingService$PutMappingExecutor.execute(MetadataMappingService.java:244) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:867) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:190) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:228) ~[opensearch-2.5.0.jar:2.5.0]
        ... 6 more

So I guess by using the current WinlogBeat ECS template, I get close to the 1000 fields limit of the index. And some change by the Security Analytics module is putting that mark even a lot higher - I already adjusted the index template to be 2000.

What’s bothersome about this is that Security Analytics is now setup in some way but I don’t yet have a detector I can query via API. Some jobs are already at work behind the scenes though:

[2023-01-28T20:26:29,571][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.opensearch-sap-ad_ldap-alerts/vV5Ch9CKR1eis7Spm72EwQ]
[2023-01-28T20:26:29,577][INFO ][o.o.a.a.AlertIndices     ] [opensearch] Index mapping of .opensearch-sap-ad_ldap-alerts is updated
[2023-01-28T20:26:29,578][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.opensearch-sap-ad_ldap-alerts-history-2023.01.27-1/-9_iBWstRway72uKM4HjtQ]
[2023-01-28T20:26:29,584][INFO ][o.o.a.a.AlertIndices     ] [opensearch] Index mapping of .opensearch-sap-ad_ldap-alerts-history-2023.01.27-1 is updated
[2023-01-28T20:26:29,594][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.opensearch-sap-ad_ldap-findings-2023.01.27-1/UJuKxD7DQgWt5fj4SbRqrA]
[2023-01-28T20:26:29,596][INFO ][o.o.a.a.AlertIndices     ] [opensearch] Index mapping of .opensearch-sap-ad_ldap-findings-2023.01.27-1 is updated
[2023-01-28T20:26:29,610][INFO ][o.o.p.PluginsService     ] [opensearch] PluginService:onIndexModule index:[.opensearch-sap-ad_ldap-detectors-queries-000001/pmYVYK5PTpSGKBic4sLTtA]
[2023-01-28T20:26:29,615][ERROR][o.o.a.u.AlertingException] [opensearch] Alerting error: java.lang.IllegalArgumentException: analyzer [rule_analyzer] has not been configured in mappings
[2023-01-28T20:26:29,616][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch] uncaught exception in thread [DefaultDispatcher-worker-2]
org.opensearch.alerting.util.AlertingException: analyzer [rule_analyzer] has not been configured in mappings
        at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70) ~[opensearch-alerting-2.5.0.0.jar:2.5.0.0]
        at org.opensearch.alerting.util.DocLevelMonitorQueries.updateQueryIndexMappings(DocLevelMonitorQueries.kt:359) ~[opensearch-alerting-2.5.0.0.jar:2.5.0.0]
        at org.opensearch.alerting.util.DocLevelMonitorQueries.access$updateQueryIndexMappings(DocLevelMonitorQueries.kt:41) ~[opensearch-alerting-2.5.0.0.jar:2.5.0.0]
        at org.opensearch.alerting.util.DocLevelMonitorQueries$updateQueryIndexMappings$1.invokeSuspend(DocLevelMonitorQueries.kt) ~[opensearch-alerting-2.5.0.0.jar:2.5.0.0]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
        at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:285) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
Caused by: java.lang.Exception: java.lang.IllegalArgumentException: analyzer [rule_analyzer] has not been configured in mappings
        ... 9 more
uncaught exception in thread [DefaultDispatcher-worker-2]
AlertingException[analyzer [rule_analyzer] has not been configured in mappings]; nested: Exception[java.lang.IllegalArgumentException: analyzer [rule_analyzer] has not been configured in mappings];
        at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70)
        at org.opensearch.alerting.util.DocLevelMonitorQueries.updateQueryIndexMappings(DocLevelMonitorQueries.kt:359)
        at org.opensearch.alerting.util.DocLevelMonitorQueries.access$updateQueryIndexMappings(DocLevelMonitorQueries.kt:41)
        at org.opensearch.alerting.util.DocLevelMonitorQueries$updateQueryIndexMappings$1.invokeSuspend(DocLevelMonitorQueries.kt)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:285)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742)
Caused by: java.lang.Exception: java.lang.IllegalArgumentException: analyzer [rule_analyzer] has not been configured in mappings
        ... 9 more

When trying this first, the .opensearch-sap-ad_ldap-detectors-queries index (and others) was rolling over every minute, which pretty soon brought my system to the 1000 shards limit :frowning:

This was the error I received before: Security Analytics error when using Datastreams

I’d be glad to provide any further information - this really rocks and I would love to start using it.

Alex