Greetings guys,
i’d like to ask for advice…
Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch 2.18.0
dashboard 2.18.0
logstash-oss-with-opensearch-output-plugin:8.9.0
Describe the issue*:
- i have to stop detector for linux logs (which was working fine …) for testing purposes where i have to test detector for kubernetes logs.
- after few weeks i start it again and now i am getting no alerts/findings…
also there is another issue which is not in version 2.16.0
when i set index pattern and then alias from pattern, it includes only current or old indices and when new day will come and new index as well, it is not part of this alias.
as you can see missing indicies from today and yestarday.
i dont know if these issues has relation…
Configuration:
- opensearch cluster is runnig as helm chart within 3 nodes RKE2 cluster.
- linux audit logs is collecting via auditbeat 8.13.1 and sending to logstash which is forwarding to opensearch.
Relevant Logs or Screenshots:
in opensearch pods i can see such log msgs:
this is collected logs from all 3 pods which seems to be relevant to me:
[2024-12-03T22:03:30,286][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-0] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [process.exe]];
[2024-12-03T22:03:30,286][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-0] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [process.command_line]];
[2024-12-03T22:03:30,288][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-0] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [TargetFilename]];
[2024-12-03T22:03:30,288][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-0] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [system.auth.user]];
[2024-12-03T22:03:30,288][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-0] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [type]];
[2024-12-04T02:14:40,559][INFO ][o.o.j.s.JobSweeper ] [opensearch-cluster-master-0] Running full sweep
[2024-12-03T22:01:17,986][ERROR][o.o.s.t.TransportIndexDetectorAction] [opensearch-cluster-master-0] PERF_DEBUG_SAP: Fetching alias path pairs to construct rule_field_names
[2024-12-04T09:40:01,250][WARN ][o.o.s.b.SearchBackpressureService] [opensearch-cluster-master-0] [monitor_only mode] cancelling task [213915] due to high resource consumption [heap usage exceeded [575.3mb >= 283.1mb]]
[2024-12-04T20:33:04,439][ERROR][o.o.c.a.u.AlertingException] [opensearch-cluster-master-2] Alerting error: java.lang.IllegalStateException: Monitor JC-KjpMBcP4vYYW2wLjv: Failed to run percolate search for sourceIndex [k3s-maas-2024.12.03] and queryIndex [.opensearch-sap-linux-detectors-queries-optimized-7ea4c8a2-274d-4d12-b520-ec7879829bc7-000001] for 21028 document(s)
[2024-12-04T20:40:22,498][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-1] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [capabilities]];
[2024-12-04T20:40:22,499][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-1] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [hostPath]];
[2024-12-04T20:40:22,499][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-1] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [responseStatus.code]];
[2024-12-04T20:40:22,577][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-1] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [capabilities]];
[2024-12-04T20:40:22,577][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-1] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [hostPath]];
[2024-12-04T20:40:22,577][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-1] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [responseStatus.code]];
[2024-12-04T20:40:22,757][INFO ][o.o.a.a.AlertIndices ] [opensearch-cluster-master-1] Index mapping of .opensearch-sap-kubernetes-alerts is updated
[2024-12-04T20:40:22,766][INFO ][o.o.a.a.AlertIndices ] [opensearch-cluster-master-1] Index mapping of .opensearch-sap-kubernetes-alerts-history-2024.11.25-1 is updated
[2024-12-04T20:40:22,775][INFO ][o.o.a.a.AlertIndices ] [opensearch-cluster-master-1] Index mapping of .opensearch-sap-kubernetes-alerts is updated
[2024-12-04T20:41:22,405][INFO ][o.o.a.t.TransportExecuteWorkflowAction] [opensearch-cluster-master-1] Executing workflow from API - id: ZMv2i5MB3dFpDdtbMCPz, periodStart: 2024-12-04T20:40:22.388Z, periodEnd: 2024-12-04T20:41:22.388Z, dryrun: false
[2024-12-04T20:41:22,416][INFO ][o.o.a.a.AlertIndices ] [opensearch-cluster-master-1] Index mapping of .opensearch-sap-kubernetes-alerts is updated
[2024-12-04T20:41:22,422][INFO ][o.o.a.a.AlertIndices ] [opensearch-cluster-master-1] Index mapping of .opensearch-sap-kubernetes-alerts-history-2024.11.25-1 is updated
[2024-12-04T20:41:22,426][INFO ][o.o.a.a.AlertIndices ] [opensearch-cluster-master-1] Index mapping of .opensearch-sap-kubernetes-findings-2024.12.03-000007 is updated
[2024-12-04T20:41:22,496][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-1] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [capabilities]];
[2024-12-04T20:41:22,496][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-1] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [hostPath]];
[2024-12-04T20:41:22,496][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-1] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [responseStatus.code]];
[2024-12-04T20:41:22,610][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-1] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [capabilities]];
[2024-12-04T20:41:22,610][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-1] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [hostPath]];
[2024-12-04T20:41:22,610][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-1] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [responseStatus.code]];
[2024-12-04T20:41:25,197][INFO ][o.o.a.a.AlertIndices ] [opensearch-cluster-master-1] Index mapping of .opensearch-sap-kubernetes-alerts is updated
[2024-12-04T20:41:25,205][INFO ][o.o.a.a.AlertIndices ] [opensearch-cluster-master-1] Index mapping of .opensearch-sap-kubernetes-alerts-history-2024.11.25-1 is updated
[2024-12-04T20:41:25,217][INFO ][o.o.a.a.AlertIndices ] [opensearch-cluster-master-1] Index mapping of .opensearch-sap-kubernetes-alerts is updated
[2024-12-04T20:41:34,152][INFO ][o.o.s.t.SecureTransportAction] [opensearch-cluster-master-1] User and roles string from thread context: admin|admin|all_access|EdgAIR
[2024-12-04T20:41:34,156][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-master-1] Indexing [23] customLogTypes
[2024-12-04T20:41:34,188][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-master-1] Loaded [23] customLogType docs successfully!
[2024-12-04T20:41:34,839][INFO ][o.o.s.t.SecureTransportAction] [opensearch-cluster-master-1] User and roles string from thread context: admin|admin|all_access|EdgAIR
[2024-12-04T20:41:35,441][INFO ][o.o.s.t.SecureTransportAction] [opensearch-cluster-master-1] User and roles string from thread context: admin|admin|all_access|EdgAIR
[2024-12-04T20:41:35,441][INFO ][o.o.s.t.SecureTransportAction] [opensearch-cluster-master-1] User and roles string from thread context: admin|admin|all_access|EdgAIR
[2024-12-04T20:41:35,622][INFO ][o.o.s.t.SecureTransportAction] [opensearch-cluster-master-1] User and roles string from thread context: admin|admin|all_access|EdgAIR
[2024-12-04T20:41:35,622][INFO ][o.o.s.t.SecureTransportAction] [opensearch-cluster-master-1] User and roles string from thread context: admin|admin|all_access|EdgAIR
[2024-12-04T20:41:35,868][INFO ][o.o.s.t.SecureTransportAction] [opensearch-cluster-master-1] User and roles string from thread context: admin|admin|all_access|EdgAIR
[2024-12-04T20:41:38,450][INFO ][o.o.s.t.SecureTransportAction] [opensearch-cluster-master-1] User and roles string from thread context: admin|admin|all_access|EdgAIR
[2024-12-04T20:41:39,789][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-master-1] Indexing [23] customLogTypes
[2024-12-04T20:41:39,820][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-master-1] Loaded [23] customLogType docs successfully!
[2024-12-04T20:41:42,035][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-master-1] Indexing [23] customLogTypes
[2024-12-04T20:41:42,066][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-master-1] Loaded [23] customLogType docs successfully!
[2024-12-04T20:41:42,074][INFO ][o.o.s.t.SecureTransportAction] [opensearch-cluster-master-1] User and roles string from thread context: admin|admin|all_access|EdgAIR
[2024-12-04T20:41:42,241][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-master-1] Indexing [23] customLogTypes
[2024-12-04T20:41:42,272][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-master-1] Loaded [23] customLogType docs successfully!
[2024-12-04T20:41:42,518][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-master-1] Indexing [23] customLogTypes
[2024-12-04T20:41:42,547][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-master-1] Loaded [23] customLogType docs successfully!
[2024-12-04T20:41:42,737][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-master-1] Indexing [23] customLogTypes
[2024-12-04T20:41:42,766][INFO ][o.o.s.l.LogTypeService ] [opensearch-cluster-master-1] Loaded [23] customLogType docs successfully!
[2024-12-04T20:42:34,937][ERROR][o.o.a.DocumentLevelMonitorRunner] [opensearch-cluster-master-2] Failed running Document-level-monitor k3s-siem
[2024-12-04T20:42:34,913][ERROR][o.o.a.DocumentLevelMonitorRunner] [opensearch-cluster-master-2] Fan out failed in node bw9uT_CFTvmhEuTHR6V8jw
org.opensearch.transport.RemoteTransportException: [opensearch-cluster-master-2][10.42.0.226:9300][cluster:admin/opensearch/alerting/monitor/doclevel/fanout]
Caused by: org.opensearch.commons.alerting.util.AlertingException: Monitor F4hkk5MBKTlbnrV7XP6J: Failed to run percolate search for sourceIndex [k3s-maas-2024.12.04] and queryIndex [.opensearch-sap-linux-detectors-queries-optimized-44ffe117-0281-4d3f-9c75-1cb44316ffcf-000001] for 780 document(s)
at org.opensearch.commons.alerting.util.AlertingException$Companion.wrap(AlertingException.kt:70) ~[common-utils-2.18.0.0.jar:?]
at org.opensearch.alerting.transport.TransportDocLevelMonitorFanOutAction.executeMonitor(TransportDocLevelMonitorFanOutAction.kt:347) [opensearch-alerting-2.18.0.0.jar:2.18.0.0]
Caused by: org.apache.lucene.search.IndexSearcher$TooManyNestedClauses: Query contains too many nested clauses; maxClauseCount is set to 1024
[2024-12-04T20:42:31,633][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [ParentCommandLine]];
[2024-12-04T20:42:31,633][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [ParentCommandLine]];
[2024-12-04T20:42:31,633][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [ParentCommandLine]];
[2024-12-04T20:42:31,633][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [process.real_user.id_k3s-maas-2024.12.04_F4hkk5MBKTlbnrV7XP6J]];
[2024-12-04T20:42:31,633][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [process.real_user.id_k3s-maas-2024.12.04_F4hkk5MBKTlbnrV7XP6J]];
[2024-12-04T20:42:31,633][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [rsa.web.remote_domain]];
[2024-12-04T20:42:31,633][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [rsa.web.remote_domain]];
[2024-12-04T20:42:31,633][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [DestinationIp]];
[2024-12-04T20:42:31,633][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [TargetFilename]];
[2024-12-04T20:42:31,633][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [TargetFilename]];
[2024-12-04T20:42:31,634][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [TargetFilename]];
[2024-12-04T20:42:31,634][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [TargetFilename]];
[2024-12-04T20:42:31,634][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [TargetFilename]];
[2024-12-04T20:42:31,634][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [TargetFilename]];
[2024-12-04T20:42:31,634][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [TargetFilename]];
[2024-12-04T20:42:31,634][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [type]];
[2024-12-04T20:42:31,634][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [type]];
[2024-12-04T20:42:31,634][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [type]];
[2024-12-04T20:42:31,634][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [type]];
[2024-12-04T20:42:31,634][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [type]];
[2024-12-04T20:41:24,992][ERROR][o.o.s.c.JoinEngine ] [opensearch-cluster-master-2] [CORRELATIONS] Exception encountered while searching correlation rule index for finding id e92eb043-7397-4777-9d14-95245becee41
org.opensearch.index.IndexNotFoundException: no such index [.opensearch-sap-correlation-rules-config]
at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:1070) ~[opensearch-2.18.0.jar:2.18.0]
[2024-12-04T20:42:31,634][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [type]];
after i made manuall mapping like few weeks ago which solved my issue with detector:
i think, then, there has left only :
[2024-12-04T20:42:31,634][ERROR][o.o.a.u.DocLevelMonitorQueries] [opensearch-cluster-master-2] MapperParsingException[failed to parse]; nested: QueryShardException[No field mapping can be found for the field with name [type]]; but not sure…
i tried remove _type via lgostash.conf with:
remove_field => ["_type"]
no changes…
Thanks for any hint
L