Doc-Level Detector: Custom Rule Field Mappings Not Persisting After Creation

[sunil_9944]

Hi Team,

I am using OpenSearch Security Analytics and facing an issue where field mappings appear during detector creation, but after saving the detector, the Field Mappings section becomes empty.

What I Did

1. Created a Doc-level detector using the Apache Access Log log type.

2. During creation, OpenSearch automatically suggested correct mappings, e.g.:

Detection rule field: http.request.url
Data source field: http.request.url

3. The “Mapped fields” section correctly showed entries.

4. After clicking Create Detector, I opened the detector → Field Mappings tab shows “No items found.”

My Log Fields (from my index)

Example log document:

client.ip: 192.168.1.100
event.created: 2025-11-18T20:37:00Z
host.hostname: test-nginx-server
http.request.method: GET
http.request.referrer: -
http.request.url: /index.php?id=1 UNION SELECT username,password FROM users
http.request.version: 1.1
http.response.body.bytes: 1234
http.response.status_code: 200
user_agent_original: Mozilla/5.0   <— Not ECS format
log_type: access

Observations

• The field mappings suggested during the creation step are NOT stored in the detector JSON.

• After saving the detector, field_mappings is empty.

• This happens even if only 1 field (e.g., http.request.url) requires mapping.

• It seems that OpenSearch requires full ECS compatibility for all rule-required fields, and if even one field (like user_agent_original) doesn’t match ECS (user_agent.original), then no mappings are persisted at all.

Expected Behavior

If I map one field manually, such as:

http.request.url → http.request.url

…then that mapping should persist, even if other optional ECS fields are missing.

Actual Behavior

• Mapping shows during creation.

• Mapping disappears after saving the detector (UI shows “No items found”).

• The detector JSON contains "field_mappings": [].

Questions

1. Is this the intended behavior for doc-level detectors?
   (All-or-nothing ECS validation)

2. Can OpenSearch allow partial mappings to persist even if other rule fields are missing?

3. Is there a recommended approach for logs that are ECS-like but not fully ECS compliant?

Environment

• OpenSearch 3.1.0

• Running inside Docker

• Log Type: Apache Access

• Index: nginx_logs-* (custom logs)

[Anthony]

@sunil_9944 I am not able to see predefined fields in Apache Access logs, and the documentation confirmed this too. It would seem that currently you cannot make the Field mappings UI show rows for apache_access via the APIs, because the log type intentionally has no field mappings shipped. Your detector and rule still work, the pre-packaged apache_access rules use the raw fields (verb, response, etc.) directly,

You can use the following API to see the list available, empty properties is the reason for empty mappings:

GET _plugins/_security_analytics/mappings/view
{
  "index_name": "apache_access_logs",
  "rule_topic": "apache_access"
}

{
  "properties": {},
  "unmapped_index_fields": [
    "verb",
    "response",
    "request",
    "referrer",
    "ident",
    "httpversion",
    "clientip",
    "bytes",
    "auth",
    "agent",
    "@timestamp"
  ]
}

You can compare this to window example:

GET _plugins/_security_analytics/mappings/view
{
  "index_name": "windows",
  "rule_topic": "windows"
}

{
  "properties": {
    "process.command_line": {
      "path": "CommandLine",
      "type": "alias"
    },
    "winlog.event_id": {
      "path": "EventID",
      "type": "alias"
    },
    "winlog.user.name": {
      "path": "User",
      "type": "alias"
    }
  },
  "unmapped_index_fields": [
    "@timestamp"
  ],
  "unmapped_field_aliases": [
    "winlog.event_data.AuthenticationPackageName",
    "winlog.event_data.ScriptBlockLogging",
    "winlog.event_data.SidHistory",
    "winlog.event_data.Properties",
...