This is from Opensearch-Dashboards (sorry my container is still called Kibana as it’s shorter ):
kibana | Security Analytics - DetectorsService - getDetector: StatusCodeError: [security_analytics_exception] Failed applying mappings to index
kibana | at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
kibana | at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
kibana | at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
kibana | at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
kibana | at IncomingMessage.emit (events.js:412:35)
kibana | at IncomingMessage.emit (domain.js:475:12)
kibana | at endReadableNT (internal/streams/readable.js:1333:12)
kibana | at processTicksAndRejections (internal/process/task_queues.js:82:21) {
kibana | status: 500,
kibana | displayName: 'InternalServerError',
kibana | path: '/_plugins/_security_analytics/mappings',
kibana | query: {},
kibana | body: {
kibana | error: {
kibana | root_cause: [Array],
kibana | type: 'security_analytics_exception',
kibana | reason: 'Failed applying mappings to index',
kibana | caused_by: [Object]
kibana | },
kibana | status: 500
kibana | },
kibana | statusCode: 500,
kibana | response: '{"error":{"root_cause":[{"type":"security_analytics_exception","reason":"Failed applying mappings to index"}],"type":"security_analytics_exception","reason":"Failed applying mappings to index","caused_by":{"type":"
illegal_argument_exception","reason":"Limit of total fields [2000] has been exceeded"}},"status":500}',
kibana | toString: [Function (anonymous)],
kibana | toJSON: [Function (anonymous)]
kibana | }
kibana | {"type":"response","@timestamp":"2023-01-28T20:20:20Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/_plugins/_security_analytics/mappings","method":"post","headers":{"host":"logging","user-agen
t":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0","accept":"*/*","accept-language":"de,en-US;q=0.7,en;q=0.3","accept-encoding":"gzip, deflate, br","referer":"https://logging/app/opensearch
_security_analytics_dashboards","content-type":"application/json","osd-version":"2.5.0","content-length":"243","origin":"https://logging","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","te":"
trailers","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-for":"172.16.1.199","connection":"close"},"remoteAddress":"172.18.0.1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/10
9.0","referer":"https://logging/app/opensearch_security_analytics_dashboards"},"res":{"statusCode":200,"responseTime":830,"contentLength":9},"message":"POST /_plugins/_security_analytics/mappings 200 830ms - 9.0B"}
This is from Opensearch:
[2023-01-28T20:20:20,799][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000018/Bwk7woWaRgSmBaiETS4M4g]
[2023-01-28T20:20:20,840][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000002/2dS3N0UJRs2xu2WMIdBAVg]
[2023-01-28T20:20:20,873][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000015/P3McMsnzTLerXNSMrwGVlw]
[2023-01-28T20:20:20,912][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000006/d1-jtlPJS8yjZIR-6YfHug]
[2023-01-28T20:20:20,943][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000001/xEHdG6DYR9ukD5kqZSMxmg]
[2023-01-28T20:20:20,976][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000010/ncZT3zNpROWpnIy81ouV3Q]
[2023-01-28T20:20:21,010][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000011/3LVjz60KSKyqzWTZDLhdCw]
[2023-01-28T20:20:21,046][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000005/wNIp48dyTOCeiYIeQORemA]
[2023-01-28T20:20:21,076][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000007/gZH7-DsIRveiCdOBaZwT9w]
[2023-01-28T20:20:21,108][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000009/vWPMsT55Svi2VQtwLdrXrA]
[2023-01-28T20:20:21,139][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000013/Z5uOmashQ7W20fCxvooRlg]
[2023-01-28T20:20:21,171][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000019/Kj3d1VFWRTK7E1fP6pxh-A]
[2023-01-28T20:20:21,206][INFO ][o.o.c.m.MetadataMappingService] [opensearch] [.ds-winlogbeat-000019/Kj3d1VFWRTK7E1fP6pxh-A] update_mapping [_doc]
[2023-01-28T20:20:21,208][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000003/B96TiycDR3aWyPFx2RMsqw]
[2023-01-28T20:20:21,248][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000008/5FdbCqgBSL2mCQUO8-a83A]
[2023-01-28T20:20:21,285][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000012/i4U1ehmnT6qLQCPX_Yv6Ew]
[2023-01-28T20:20:21,320][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000014/K3SAMkqmRNKTtLlv1zkRnw]
[2023-01-28T20:20:21,356][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000017/hXlM3qUTR--JZYXisWvFvg]
[2023-01-28T20:20:21,391][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000004/Hc3xJXweQza5nVSL0UX1uQ]
[2023-01-28T20:20:21,422][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.ds-winlogbeat-000016/pzyx_bi4QtC5kZieqxJDMQ]
[2023-01-28T20:20:21,503][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch] Detected cluster change event for destination migration
[2023-01-28T20:20:21,503][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opensearch] Reset destination migration process.
[2023-01-28T20:20:21,504][WARN ][r.suppressed ] [opensearch] path: /_plugins/_security_analytics/mappings, params: {}
org.opensearch.securityanalytics.util.SecurityAnalyticsException: Failed applying mappings to index
at org.opensearch.securityanalytics.mapper.MapperService$2.onFailure(MapperService.java:128) [opensearch-security-analytics-2.5.0.0.jar:2.5.0.0]
at org.opensearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:78) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.securityanalytics.mapper.MapperService$3.onResponse(MapperService.java:215) [opensearch-security-analytics-2.5.0.0.jar:2.5.0.0]
at org.opensearch.securityanalytics.mapper.MapperService$3.onResponse(MapperService.java:207) [opensearch-security-analytics-2.5.0.0.jar:2.5.0.0]
at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:113) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:107) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.support.RetryableAction$RetryingListener.onResponse(RetryableAction.java:181) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.ActionListener$2.onResponse(ActionListener.java:106) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.admin.indices.mapping.put.TransportPutMappingAction$1.onResponse(TransportPutMappingAction.java:180) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.action.admin.indices.mapping.put.TransportPutMappingAction$1.onResponse(TransportPutMappingAction.java:176) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.metadata.MetadataMappingService$1.onAllNodesAcked(MetadataMappingService.java:375) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.service.MasterService$SafeAckedClusterStateTaskListener.onAllNodesAcked(MasterService.java:717) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.service.MasterService$AckCountDownListener.finish(MasterService.java:852) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.service.MasterService$AckCountDownListener.onNodeAck(MasterService.java:843) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.service.MasterService$DelegatingAckListener.onNodeAck(MasterService.java:765) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.coordination.Coordinator$CoordinatorPublication$4$1.onSuccess(Coordinator.java:1709) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.service.ClusterApplierService$SafeClusterApplyListener.onSuccess(ClusterApplierService.java:657) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:494) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:186) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) [opensearch-2.5.0.jar:2.5.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) [opensearch-2.5.0.jar:2.5.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
Caused by: java.lang.IllegalArgumentException: Limit of total fields [2000] has been exceeded
at org.opensearch.index.mapper.MappingLookup.checkFieldLimit(MappingLookup.java:192) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.index.mapper.MappingLookup.checkLimits(MappingLookup.java:184) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.index.mapper.DocumentMapper.validate(DocumentMapper.java:329) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:491) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:447) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.index.mapper.MapperService.merge(MapperService.java:419) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.metadata.MetadataMappingService$PutMappingExecutor.applyRequest(MetadataMappingService.java:300) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.metadata.MetadataMappingService$PutMappingExecutor.execute(MetadataMappingService.java:244) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:867) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:190) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:228) ~[opensearch-2.5.0.jar:2.5.0]
... 6 more
So I guess by using the current WinlogBeat ECS template, I get close to the 1000 fields limit of the index. And some change by the Security Analytics module is putting that mark even a lot higher - I already adjusted the index template to be 2000.
What’s bothersome about this is that Security Analytics is now setup in some way but I don’t yet have a detector I can query via API. Some jobs are already at work behind the scenes though:
[2023-01-28T20:26:29,571][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.opensearch-sap-ad_ldap-alerts/vV5Ch9CKR1eis7Spm72EwQ]
[2023-01-28T20:26:29,577][INFO ][o.o.a.a.AlertIndices ] [opensearch] Index mapping of .opensearch-sap-ad_ldap-alerts is updated
[2023-01-28T20:26:29,578][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.opensearch-sap-ad_ldap-alerts-history-2023.01.27-1/-9_iBWstRway72uKM4HjtQ]
[2023-01-28T20:26:29,584][INFO ][o.o.a.a.AlertIndices ] [opensearch] Index mapping of .opensearch-sap-ad_ldap-alerts-history-2023.01.27-1 is updated
[2023-01-28T20:26:29,594][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.opensearch-sap-ad_ldap-findings-2023.01.27-1/UJuKxD7DQgWt5fj4SbRqrA]
[2023-01-28T20:26:29,596][INFO ][o.o.a.a.AlertIndices ] [opensearch] Index mapping of .opensearch-sap-ad_ldap-findings-2023.01.27-1 is updated
[2023-01-28T20:26:29,610][INFO ][o.o.p.PluginsService ] [opensearch] PluginService:onIndexModule index:[.opensearch-sap-ad_ldap-detectors-queries-000001/pmYVYK5PTpSGKBic4sLTtA]
[2023-01-28T20:26:29,615][ERROR][o.o.a.u.AlertingException] [opensearch] Alerting error: java.lang.IllegalArgumentException: analyzer [rule_analyzer] has not been configured in mappings
[2023-01-28T20:26:29,616][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch] uncaught exception in thread [DefaultDispatcher-worker-2]
org.opensearch.alerting.util.AlertingException: analyzer [rule_analyzer] has not been configured in mappings
at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70) ~[opensearch-alerting-2.5.0.0.jar:2.5.0.0]
at org.opensearch.alerting.util.DocLevelMonitorQueries.updateQueryIndexMappings(DocLevelMonitorQueries.kt:359) ~[opensearch-alerting-2.5.0.0.jar:2.5.0.0]
at org.opensearch.alerting.util.DocLevelMonitorQueries.access$updateQueryIndexMappings(DocLevelMonitorQueries.kt:41) ~[opensearch-alerting-2.5.0.0.jar:2.5.0.0]
at org.opensearch.alerting.util.DocLevelMonitorQueries$updateQueryIndexMappings$1.invokeSuspend(DocLevelMonitorQueries.kt) ~[opensearch-alerting-2.5.0.0.jar:2.5.0.0]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:285) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
Caused by: java.lang.Exception: java.lang.IllegalArgumentException: analyzer [rule_analyzer] has not been configured in mappings
... 9 more
uncaught exception in thread [DefaultDispatcher-worker-2]
AlertingException[analyzer [rule_analyzer] has not been configured in mappings]; nested: Exception[java.lang.IllegalArgumentException: analyzer [rule_analyzer] has not been configured in mappings];
at org.opensearch.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70)
at org.opensearch.alerting.util.DocLevelMonitorQueries.updateQueryIndexMappings(DocLevelMonitorQueries.kt:359)
at org.opensearch.alerting.util.DocLevelMonitorQueries.access$updateQueryIndexMappings(DocLevelMonitorQueries.kt:41)
at org.opensearch.alerting.util.DocLevelMonitorQueries$updateQueryIndexMappings$1.invokeSuspend(DocLevelMonitorQueries.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:285)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594)
at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742)
Caused by: java.lang.Exception: java.lang.IllegalArgumentException: analyzer [rule_analyzer] has not been configured in mappings
... 9 more
When trying this first, the .opensearch-sap-ad_ldap-detectors-queries index (and others) was rolling over every minute, which pretty soon brought my system to the 1000 shards limit
This was the error I received before: Security Analytics error when using Datastreams
I’d be glad to provide any further information - this really rocks and I would love to start using it.
Alex