Mapping API / index pattern

[romain-sp]

release v 2.6.0

If understand correctly, Sigma does not support nested field such as x.y.z so I need to create a mapping.

I have a datastream and index pattern named “ntd.hunt-dns”

If I create a detector on the UI, the auto mappings creation works seems to work fine

GET /_plugins/_security_analytics/mappings
{
   "index_name": "ntd.hunt-dns"
}

{
  "ntd.hunt-dns": {
    "mappings": {
      "properties": {
        "dns-question-name": {
          "type": "alias",
          "path": "dns.question.name"
        }
      }
    }
  }
}

But if try using the API, it does not work

PUT /_plugins/_security_analytics/mappings
{
   "index_name": "ntd.hunt-dns",
   "field": "dns-response-code",
   "alias": "dns.response.code"
}

{
  "error": {
    "root_cause": [
      {
        "type": "security_analytics_exception",
        "reason": "Could not find index [ntd.hunt-dns]"
      }
    ],
    "type": "security_analytics_exception",
    "reason": "Could not find index [ntd.hunt-dns]",
    "caused_by": {
      "type": "exception",
      "reason": "org.opensearch.OpenSearchStatusException: Could not find index [ntd.hunt-dns]"
    }
  },
  "status": 404
}

Does mapping support datastream / index pattern ?

[pdz]

@romain-sp

PUT /_plugins/_security_analytics/mappings

This API is actually unused and it probably should be removed. You should use POST /_plugins/_security_analytics/mappingsto apply mappings. It supports datastreams.

[orazaly1508]

Yes, mapping APIs support data streams and index patterns. However, the endpoint you are using to create mappings, /_plugins/_security_analytics/mappings, is specific to the OpenSearch Security Analytics plugin and may not work with data streams. Additionally, you should use the POST method instead of PUT.

Here’s an example of how you can create a mapping using the correct endpoint and method:

POST /_plugins/_security_analytics/mapping
{
  "index_name": "ntd.hunt-dns",
  "mappings": {
    "properties": {
      "dns-response-code": {
        "type": "alias",
        "path": "dns.response.code"
      }
    }
  }
}

This endpoint supports datastreams, so you can use it with datastream names as well.

[romain-sp]

The UI supports index patterns data source for detector

And it successfully created this auto mapping

GET /_plugins/_security_analytics/mappings
{
   "index_name": "ntd.hunt-dns"
}

{
  "ntd.hunt-dns": {
    "mappings": {
      "properties": {
        "dns-question-name": {
          "type": "alias",
          "path": "dns.question.name"
        }
      }
    }
  }
}

If I use the POST method without specifiying "rule_topic": "dns" I get this error

POST /_plugins/_security_analytics/mapping
{
  "index_name": "ntd.hunt-dns",
  "alias_mappings": {
    "properties": {
      "dns-response-code": {
        "type": "alias",
        "path": "dns.response.code"
      }
    }
  }
}

{
  "error": "no handler found for uri [/_plugins/_security_analytics/mapping] and method [POST]"
}

But it seems to be accepted with the "rule_topic": "dns"

POST /_plugins/_security_analytics/mappings
{
   "index_name": "ntd.hunt-dns",
   "rule_topic": "dns",
   "alias_mappings": {
        "properties": {
            "dns-response-code": {
            "type": "alias",
            "path": "dns.response.code"
          }
       }
   }
}

{
  "acknowledged": true
}

However, the mapping does not seem applied, as I still see the initial mapping created by the UI

GET /_plugins/_security_analytics/mappings
{
   "index_name": "ntd.hunt-dns"
}

{
  "ntd.hunt-dns": {
    "mappings": {
      "properties": {
        "dns-question-name": {
          "type": "alias",
          "path": "dns.question.name"
        }
      }
    }
  }
}

I also tried to use POST with "mappings" instead of "alias_mappings" and it is accepted as well but still has no effect

[romain-sp]

An other exemple with no auto mapping created by the UI

POST /_plugins/_security_analytics/mappings
{
   "index_name": "ntd.hunt-syslog",
   "rule_topic": "network",
   "alias_mappings": {
        "properties": {
            "source-ip": {
            "type": "alias",
            "path": "source.ip"
          }
       }
   }
}

{
  "acknowledged": true
}

GET /_plugins/_security_analytics/mappings
{
   "index_name": "ntd.hunt-syslog",
    "rule_topic": "network"
}

{
  "error": {
    "root_cause": [
      {
        "type": "security_analytics_exception",
        "reason": "No applied aliases found"
      }
    ],
    "type": "security_analytics_exception",
    "reason": "No applied aliases found",
    "caused_by": {
      "type": "exception",
      "reason": "org.opensearch.OpenSearchStatusException: No applied aliases found"
    }
  },
  "status": 404
}

[pdz]

@romain-sp

"source-ip"

“source-ip” is not mapped Sigma field hence you don’t get any applied aliases.

This is expected workflow using APIs:

Call GET /_plugins/_security_analytics/mappings/view to get automatically mapped fields(returned in properties field) and unmapped sigma fields(returned in unmapped_field_aliases field).

You will get unmapped Sigma fields from ALL rules of rule category you selected(for example windows). If you don’t want to select ALL rules in detector, you can fetch all rule fields for rules that you want enabled. Here is query example for fetching Sigma field names for 1 rule of network category with id 286b47ed-f6fe-40b3-b3a8-35129acd43bc:

POST /.opensearch-sap-pre-packaged-rules-config/_search
{
	"_source":[
		"rule.query_field_names"
	],
	"query":{
		"nested":{
			"path":"rule",
			"query":{
				"bool":{
					"must":[
						{
							"term":{
								"rule.category":{
									"value":"network"
								}
							}
						},
						{
							"terms":{
								"_id":[
									"286b47ed-f6fe-40b3-b3a8-35129acd43bc"
								]
							}
						}
					]
				}
			}
		}
	}
}

Now that you know which fields you need to map, you will pack them in a single request and call:

POST /_plugins/_security_analytics/mappings
{
   "index_name": "ntd.hunt-syslog",
   "rule_topic": "network",
   "alias_mappings": {
        "properties": {
            "some-rule-field": {
            "type": "alias",
            "path": "some-index-field"
          },
          ....
       }
   }
}

After this call you can call Create Detector API.

[romain-sp]

Thanks for the clarification.
So maybe I just don’t have to right approach for what I try to achieve…

I try to create Sigma rules for index patterns having custom nested fields (dns.response.code, source.ip …)

I try to create a new mapping because:

• I want to map my custom fields
• dot notation is not supported

Here is a rule I try to create
selection:
dns.response_code:
- NXDOMAIN
- SERVFAIL
condition: selection | count(dns.question.name) by source.ip > 10
timeframe: 2m

what is right approach for doing this ?

[romain-sp]

Is it possible to include my own OSMapping file?

[romain-sp]

Still trying to figure out how to create rules with nested fields.
Is this not supported ?