Security Analytics detector not consistently generating Findings
Is there some sort of limit regarding how many findings a Security Analytics detector or rule will report in a given run?
In order to test our log ingestion mapping and Security Analytics configuration, I’ve hand-crafted a CloudTrail log file that has 70 records, and all but 3 of them should trigger one of the out-of-the-box CloudTrail rules. But when I ingest the log file, I get only 10 to 20 findings. Re-ingesting the same file into the same index (which replaces the existing records because the have hand-crafted unique IDs) will generate another set of findings (maybe 5 to 20 of them), and there is some overlap between the sets of findings, but it’s not consistent. Changing the eventTime and eventId (which is mapped to the opensearch @id) doesn’t make any difference… it still shows a random set of Findings from the file that’s ingested.
I’m running on AWS Gov Cloud, with the detector running every 1 minute. OpenSearch version = 2.7, Service software version = OpenSearch_2_7_R20230706-P1 (latest). We have created a cloudtrail detector using all 31 out-of-the-box rules for cloudtrail.
We have mapped 12 fields in our detector:
aws-cloudtrail-user_identity-session_context-session_issuer-type userIdentity.sessionContext.sessionIssuer.type aws-cloudtrail-user_identity-arn userIdentity.arn aws-cloudtrail-response_elements-text responseElements.text aws-cloudtrail-response_elements-publicly_accessible responseElements.publiclyAccessible aws-cloudtrail-response_elements-pending_modified_values-master_user_password responseElements.pendingModifiedValues.masterUserPassword aws-cloudtrail-request_parameters-container_definitions-command requestParameters.containerDefinitions.command aws-cloudtrail-request_parameters-attribute requestParameters.attribute aws-cloudtrail-event_type eventType aws-cloudtrail-event_source eventSource aws-cloudtrail-event_name eventName aws-cloudtrail-error_message errorMessage aws-cloudtrail-error_code errorCode
(I don’t understand why we have to map these, it seems that cloudtrail_logtype.json is forcing the fields to be renamed, but that’s a separate issue I haven’t addressed yet.)
(We also made our ingestion create a requestParameters.attribute which holds the literal text “userData” when it detects the userData field contains data. Hacky, but we had to do this to make rule “AWS EC2 Download Userdata” trigger. Again, not what this error report is about.)
Here’s an example of a record that should trigger the AWS Route 53 Domain Transfer Lock Disabled. Sometimes it does, sometimes it doesn’t.
{
"eventVersion": "bill security analytics test - 9/8/2023",
"eventTest": "Test Rule: AWS Route 53 Domain Transfer Lock Disabled",
"eventTime": "2023-09-06T01:10:01Z",
"eventSource": "route53.amazonaws.com",
"eventName": "DisableDomainTransferLock",
"awsRegion": "us-gov-west-1",
"requestParameters": {
"someParameter": "250",
"text": "{\"someParameter\": \"250\"}"
},
"requestID": "bill-10",
"eventID": "bill-security-analytics-test-10",
"@message": "{\"eventVersion\": \"bill security analytics test - 9/8/2023\", \"eventTest\": \"Test Rule: AWS Route 53 Domain Transfer Lock Disabled\", \"eventTime\": \"2023-09-06T01:10:01Z\", \"eventSource\": \"route53.amazonaws.com\", \"eventName\": \"DisableDomainTransferLock\", \"awsRegion\": \"us-gov-west-1\", \"requestParameters\": {\"someParameter\": \"250\"}, \"responseElements\": {}, \"requestID\": \"bill-10\", \"eventID\": \"bill-security-analytics-test-10\"}",
"@timestamp": "2023-09-06T01:10:01+00:00",
"@log_type": "cloudtrail",
"@log_s3bucket": "tlz-cloudtrail-central-33xxxx",
"@log_s3key": "AWSLogs/o-r1ji8l6vag/12xxxx/CloudTrail/us-gov-west-1/2023/09/06/12xxxx_CloudTrail_us-gov-west-1_20230906T0035Z_billTest.json",
"event": {
"module": "route53.amazonaws.com",
"ingested": "2023-09-12T12:58:21.083095-04:00",
"action": "DisableDomainTransferLock",
"kind": "event",
"category": [
"iam"
],
"outcome": "success"
},
"@id": "bill-security-analytics-test-10",
"ecs": {
"version": "1.10.0"
},
"cloud": {
"provider": "aws",
"region": "us-gov-west-1",
"account": {
"id": "12xxxx"
}
},
"rule": {
"name": "DisableDomainTransferLock"
},
"@s3etag": "cloudtrail-detector-15-20",
"@log_line": "4/70"
}
Here’s what I call rule 14 in my hand-crafted ID: “AWS ElastiCache Security Group Modified or Deleted”
I created 5 records (14a, 14b, 14c, 14d, and 14e) to try and trigger all 5 eventNames.
selection: eventSource: elasticache.amazonaws.com eventName: - DeleteCacheSecurityGroup - AuthorizeCacheSecurityGroupIngress - RevokeCacheSecurityGroupIngress - AuthorizeCacheSecurityGroupEgress - RevokeCacheSecurityGroupEgress condition: selection
Here’s what I call rule 15 in my handcrafted ID: AWS EKS Cluster Created or Deleted
I created 2 records (15a, 15b) to try and trigger these.
selection: eventSource: eks.amazonaws.com eventName: - CreateCluster - DeleteCluster condition: selection
Here’s some records that should all trigger AWS ElastiCache Security Group Modified or Deleted.
- I loaded them once, 14d triggered at 12:38pm
- Reloaded, and 14d, 14b, and 14e triggered at 12:59pm.
- 15a and 15b have not triggered yet in the 5 runs I did.
- I tried changing the eventTime and the eventID (which is mapped to the OpenSearch @id) assuming that the detector kept track of what it had already processed, but it is still inconsistent.
{
"Records": [
{
"eventVersion": "bill security analytics test - 9/8/2023",
"eventTest": "Test Rule: AWS ElastiCache Security Group Modified or Deleted - del cache",
"eventTime": "2023-09-06T01:14:01Z",
"eventSource": "elasticache.amazonaws.com",
"eventName": "DeleteCacheSecurityGroup",
"awsRegion": "us-gov-west-1",
"requestParameters": {
"someParameter": "250"
},
"responseElements": {
},
"requestID": "bill-14a",
"eventID": "bill-security-analytics-test-14a"
},
{
"eventVersion": "bill security analytics test - 9/8/2023",
"eventTest": "Test Rule: AWS ElastiCache Security Group Modified or Deleted - auth cache",
"eventTime": "2023-09-06T01:14:21Z",
"eventSource": "elasticache.amazonaws.com",
"eventName": "AuthorizeCacheSecurityGroupIngress",
"awsRegion": "us-gov-west-1",
"requestParameters": {
"someParameter": "250"
},
"responseElements": {
},
"requestID": "bill-14b",
"eventID": "bill-security-analytics-test-14b"
},
{
"eventVersion": "bill security analytics test - 9/8/2023",
"eventTest": "Test Rule: AWS ElastiCache Security Group Modified or Deleted - revoke cache",
"eventTime": "2023-09-06T01:14:31Z",
"eventSource": "elasticache.amazonaws.com",
"eventName": "RevokeCacheSecurityGroupIngress",
"awsRegion": "us-gov-west-1",
"requestParameters": {
"someParameter": "250"
},
"responseElements": {
},
"requestID": "bill-14c",
"eventID": "bill-security-analytics-test-14c"
},
{
"eventVersion": "bill security analytics test - 9/8/2023",
"eventTest": "Test Rule: AWS ElastiCache Security Group Modified or Deleted - auth egress cache",
"eventTime": "2023-09-06T01:14:41Z",
"eventSource": "elasticache.amazonaws.com",
"eventName": "AuthorizeCacheSecurityGroupEgress",
"awsRegion": "us-gov-west-1",
"requestParameters": {
"someParameter": "250"
},
"responseElements": {
},
"requestID": "bill-14d",
"eventID": "bill-security-analytics-test-14d"
},
{
"eventVersion": "bill security analytics test - 9/8/2023",
"eventTest": "Test Rule: AWS ElastiCache Security Group Modified or Deleted - revoke egress cache",
"eventTime": "2023-09-06T01:14:51Z",
"eventSource": "elasticache.amazonaws.com",
"eventName": "RevokeCacheSecurityGroupEgress",
"awsRegion": "us-gov-west-1",
"requestParameters": {
"someParameter": "250"
},
"responseElements": {
},
"requestID": "bill-14e",
"eventID": "bill-security-analytics-test-14e"
},
{
"eventVersion": "bill security analytics test - 9/8/2023",
"eventTest": "Test Rule: AWS EKS Cluster Created or Deleted - create",
"eventTime": "2023-09-06T01:15:01Z",
"eventSource": "eks.amazonaws.com",
"eventName": "CreateCluster",
"awsRegion": "us-gov-west-1",
"requestParameters": {
"someParameter": "250"
},
"responseElements": {
},
"requestID": "bill-15a",
"eventID": "bill-security-analytics-test-15a"
},
{
"eventVersion": "bill security analytics test - 9/8/2023",
"eventTest": "Test Rule: AWS EKS Cluster Created or Deleted - delete",
"eventTime": "2023-09-06T01:15:11Z",
"eventSource": "eks.amazonaws.com",
"eventName": "DeleteCluster",
"awsRegion": "us-gov-west-1",
"requestParameters": {
"someParameter": "250"
},
"responseElements": {
},
"requestID": "bill-15b",
"eventID": "bill-security-analytics-test-15b"
},
{
"eventVersion": "bill security analytics test - 9/8/2023",
"eventTest": "Test Rule: AWS Route 53 Domain Transfer Lock Disabled",
"eventTime": "2023-09-12T01:10:01Z",
"eventSource": "route53.amazonaws.com",
"eventName": "DisableDomainTransferLock",
"awsRegion": "us-gov-west-1",
"eventID": "bill-security-analytics-test-10-2"
}
]
}
Here’s what record 14d looks like in OpenSearch after being mapped by our ingestion lambda: (we built an custom ingestion mapping lambda, using pieces of code from GitHub - aws-samples/siem-on-amazon-opensearch-service: A solution for collecting, correlating and visualizing multiple types of logs to help investigate security incidents.)
{
"eventVersion": "bill security analytics test - 9/8/2023",
"eventTest": "Test Rule: AWS ElastiCache Security Group Modified or Deleted - auth egress cache",
"eventTime": "2023-09-06T01:14:41Z",
"eventSource": "elasticache.amazonaws.com",
"eventName": "AuthorizeCacheSecurityGroupEgress",
"awsRegion": "us-gov-west-1",
"requestParameters": {
"someParameter": "250",
"text": "{\"someParameter\": \"250\"}"
},
"requestID": "bill-14d",
"eventID": "bill-security-analytics-test-14d",
"@message": "{\"eventVersion\": \"bill security analytics test - 9/8/2023\", \"eventTest\": \"Test Rule: AWS ElastiCache Security Group Modified or Deleted - auth egress cache\", \"eventTime\": \"2023-09-06T01:14:41Z\", \"eventSource\": \"elasticache.amazonaws.com\", \"eventName\": \"AuthorizeCacheSecurityGroupEgress\", \"awsRegion\": \"us-gov-west-1\", \"requestParameters\": {\"someParameter\": \"250\"}, \"responseElements\": {}, \"requestID\": \"bill-14d\", \"eventID\": \"bill-security-analytics-test-14d\"}",
"@timestamp": "2023-09-06T01:14:41+00:00",
"@log_type": "cloudtrail",
"@log_s3bucket": "tlz-cloudtrail-central-33xxxx",
"@log_s3key": "AWSLogs/o-r1ji8l6vag/12xxxx/CloudTrail/us-gov-west-1/2023/09/06/12xxxx_CloudTrail_us-gov-west-1_20230906T0035Z_billTest-15-29.json",
"event": {
"module": "elasticache.amazonaws.com",
"ingested": "2023-09-12T12:58:21.084980-04:00",
"action": "AuthorizeCacheSecurityGroupEgress",
"kind": "event",
"category": [
"iam"
],
"outcome": "success"
},
"@id": "bill-security-analytics-test-14d",
"ecs": {
"version": "1.10.0"
},
"cloud": {
"provider": "aws",
"region": "us-gov-west-1",
"account": {
"id": "12xxxx"
}
},
"rule": {
"name": "AuthorizeCacheSecurityGroupEgress"
},
"@s3etag": "cloudtrail-detector-15-20",
"@log_line": "13/70"
}
Any ideas as to why the cloudtrail detector doesn’t detect all the records in my sample ingestion file, and what I can do about it?