I’m running on AWS Gov Cloud, with a detector running every 1 minute. OpenSearch version = 2.7, Service software version = OpenSearch_2_7_R20230706-P1 (latest). We have created a cloudtrail detector using all 31 out-of-the-box rules for cloudtrail.
We have an ingestion lambda that takes S3 log files, maps their records to ECS (Elastic Common Schema), and sends them to the OpenSearch _bulk input API, using several pieces code taken from GitHub - aws-samples/siem-on-amazon-opensearch-service: A solution for collecting, correlating and visualizing multiple types of logs to help investigate security incidents.
We have an index template that defines all the fields used by all the rules, and have ensured our ingestion lambda maps our data to them appropriately.
I built a hand-crafted log file with 70 records, with sample data that should be detected by each/all of the 31 cloudtrail rules, and the ID of each record is a unique string such as “bill-security-analytics-test-14d”. I’ve verified that they all loaded correctly and the data is in all the fields that should be inspected by the Security Analytics cloudtrail rules. However, the detector only generated 20 findings. I re-ingested the same file (without changing anything), and the detector generated another 20 findings (some overlap with the previous findings, some are records that weren’t found by the initial run). I waited a day and ingested it several more times, and 2 to 15 findings were generated each time (some overlapping, some new records). I cut my input file back to 15 records, and then to 7 records, and even tried changing the timestamp and record IDs of the 7 records, and it’s still inconsistent.
Any suggestions? (I posted this yesterday afternoon with lots of details and records, but the forum spam filter quarantined my post. So I’m trying again today with a much smaller post.)