# Security Analytics error when using Datastreams

**URL:** <https://forum.opensearch.org/t/security-analytics-error-when-using-datastreams/12304>\
**Category:** Security Analytics\
**Created:** [January 27, 2023, 8:22am UTC](https://forum.opensearch.org/t/security-analytics-error-when-using-datastreams/12304 "2023-01-27T08:22:26Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![adn77](https://avatars.discourse-cdn.com/v4/letter/a/dc4da7/32.png) [@adn77](https://forum.opensearch.org/u/adn77)\
**Post date:** [January 27, 2023, 8:22am UTC](https://forum.opensearch.org/t/security-analytics-error-when-using-datastreams/12304/1 "2023-01-27T08:22:27Z")

</div>

As Security Analytics is GA now I am wondering if it is already supposed to work with Datastreams?

There is very little documentation on what the detector setup actually does, e.g. which index templates are created or modified by the mappings API

This is the error I get when I try to create a detector:

```auto
Security Analytics - DetectorsService - getDetector: StatusCodeError: [security_analytics_exception] Can't upsert index template for concrete index!
    at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (events.js:412:35)
    at IncomingMessage.emit (domain.js:475:12)
    at endReadableNT (internal/streams/readable.js:1333:12)
    at processTicksAndRejections (internal/process/task_queues.js:82:21) {
  status: 500,
  displayName: 'InternalServerError',
  path: '/_plugins/_security_analytics/mappings',
  query: {},
  body: {
    error: {
      root_cause: [Array],
      type: 'security_analytics_exception',
      reason: "Can't upsert index template for concrete index!",
      caused_by: [Object]
    },
    status: 500
  },
  statusCode: 500,
  response: `{"error":{"root_cause":[{"type":"security_analytics_exception","reason":"Can't upsert index template for concrete index!"}],"type":"security_analytics_exception","reason":"Can't upsert index template for concrete index!","caused_by":{"type":"exception","reason":"java.lang.IllegalStateException: Can't
upsert index template for concrete index!"}},"status":500}`,
  toString: [Function (anonymous)],
  toJSON: [Function (anonymous)]
}
{"type":"response","@timestamp":"2023-01-26T11:53:07Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/_plugins/_security_analytics/mappings","method":"post","headers":{"host":"logging","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0","a
ccept":"*/*","accept-language":"de,en-US;q=0.7,en;q=0.3","accept-encoding":"gzip, deflate, br","referer":"https://logging/app/opensearch_security_analytics_dashboards","content-type":"application/json","osd-version":"2.5.0","content-length":"239","origin":"https://logging","sec-fetch-dest
":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","te":"trailers","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-for":"172.16.1.204","connection":"close"},"remoteAddress":"172.18.0.1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0","r
eferer":"https://logging/app/opensearch_security_analytics_dashboards"},"res":{"statusCode":200,"responseTime":201,"contentLength":9},"message":"POST /_plugins/_security_analytics/mappings 200 201ms - 9.0B"}

```

My datastream indexes look something like **.ds-winlogbeat-000003**. In Detector setup I chose **winlogbeat** \* as index pattern.

I can see, that the index template has been extended by the component template

```auto
    {
      "name": "winlogbeat",
      "index_template": {
        "index_patterns": [
          "winlogbeat*"
        ],
...
        "composed_of": [
          ".opensearch-sap-alias-mappings-component-winlogbeat"
        ]
      }
    },

```

The component template exists as well:

```auto
{
  "component_templates": [
    {
      "name": ".opensearch-sap-alias-mappings-component-winlogbeat",
      "component_template": {
        "template": {
          "mappings": {
            "properties": {}
          }
        },
        "version": 0
      }
    }
  ]
}

```

I briefly looked at the code but couldn’t really make sense of what’s happening 🙂

Alex

PS: I just noticed that the current state created some hundred objects **.opensearch-sap-ad\_ldap…** as a result of my (unsuccessful) detector-creation which prevented my indexes from rolling over as the maximum number of shards (1000) had been reached ☹

---

<div class="post-metadata">

**Author:** ![adn77](https://avatars.discourse-cdn.com/v4/letter/a/dc4da7/32.png) [@adn77](https://forum.opensearch.org/u/adn77)\
**Post date:** [January 28, 2023, 8:38pm UTC](https://forum.opensearch.org/t/security-analytics-error-when-using-datastreams/12304/2 "2023-01-28T20:38:34Z")

</div>

Sorry for cross-posting. Continuing here: [Something broke with index templates in 2.5.0 - #6 by adn77](https://forum.opensearch.org/t/something-broke-with-index-templates-in-2-5-0/12276/6)

---

<div class="post-metadata">

**Author:** ![pdz](https://avatars.discourse-cdn.com/v4/letter/p/51bf81/32.png) [@pdz](https://forum.opensearch.org/u/pdz)\
**Post date:** [January 30, 2023, 1:52pm UTC](https://forum.opensearch.org/t/security-analytics-error-when-using-datastreams/12304/3 "2023-01-30T13:52:27Z")

</div>

For datastreams you should pass actual datastream name as source\_index, instead of index pattern. (your datastream name is **winlogbeat**?)

---

<div class="post-metadata">

**Author:** ![adn77](https://avatars.discourse-cdn.com/v4/letter/a/dc4da7/32.png) [@adn77](https://forum.opensearch.org/u/adn77)\
**Post date:** [February 1, 2023, 10:02pm UTC](https://forum.opensearch.org/t/security-analytics-error-when-using-datastreams/12304/4 "2023-02-01T22:02:40Z")

</div>

I reconfigured my FluentD log ingestion to use indices instead of data\_streams and now the detector creation works.

How would I further debug the data\_stream compatibility?

Alex

---

<div class="post-metadata">

**Author:** ![kris](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/kris/32/2208_2.png) [@kris](https://forum.opensearch.org/u/kris)\
**Post date:** [February 1, 2023, 10:55pm UTC](https://forum.opensearch.org/t/security-analytics-error-when-using-datastreams/12304/5 "2023-02-01T22:55:04Z")

</div>

(moved to appropriate ‘Security Analytics’ plug category)

---

<div class="post-metadata">

**Author:** ![adn77](https://avatars.discourse-cdn.com/v4/letter/a/dc4da7/32.png) [@adn77](https://forum.opensearch.org/u/adn77)\
**Post date:** [February 28, 2023, 4:10pm UTC](https://forum.opensearch.org/t/security-analytics-error-when-using-datastreams/12304/6 "2023-02-28T16:10:32Z")

</div>

It actually works using datastreams as well. The index mapping turned out to be the problem.

---

<div class="post-metadata">

**Author:** ![kris](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/kris/32/2208_2.png) [@kris](https://forum.opensearch.org/u/kris)\
**Post date:** [February 28, 2023, 5:59pm UTC](https://forum.opensearch.org/t/security-analytics-error-when-using-datastreams/12304/7 "2023-02-28T17:59:04Z")

</div>

thanks for letting us know @adn77

---

<div class="post-metadata">

**Author:** ![Enyce](https://avatars.discourse-cdn.com/v4/letter/e/ecccb3/32.png) [@Enyce](https://forum.opensearch.org/u/Enyce)\
**Post date:** [September 29, 2024, 8:18pm UTC](https://forum.opensearch.org/t/security-analytics-error-when-using-datastreams/12304/8 "2024-09-29T20:18:09Z")

</div>

I have datastream organized by month, is it possible to apply the detector on a index pattern of datastream ? so it automatically apply to future datastream not created yet ?