Security Analytics error when using Datastreams

Plugins Security Analytics
1

As Security Analytics is GA now I am wondering if it is already supposed to work with Datastreams?

There is very little documentation on what the detector setup actually does, e.g. which index templates are created or modified by the mappings API

This is the error I get when I try to create a detector:

Security Analytics - DetectorsService - getDetector: StatusCodeError: [security_analytics_exception] Can't upsert index template for concrete index!
    at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (events.js:412:35)
    at IncomingMessage.emit (domain.js:475:12)
    at endReadableNT (internal/streams/readable.js:1333:12)
    at processTicksAndRejections (internal/process/task_queues.js:82:21) {
  status: 500,
  displayName: 'InternalServerError',
  path: '/_plugins/_security_analytics/mappings',
  query: {},
  body: {
    error: {
      root_cause: [Array],
      type: 'security_analytics_exception',
      reason: "Can't upsert index template for concrete index!",
      caused_by: [Object]
    },
    status: 500
  },
  statusCode: 500,
  response: `{"error":{"root_cause":[{"type":"security_analytics_exception","reason":"Can't upsert index template for concrete index!"}],"type":"security_analytics_exception","reason":"Can't upsert index template for concrete index!","caused_by":{"type":"exception","reason":"java.lang.IllegalStateException: Can't
upsert index template for concrete index!"}},"status":500}`,
  toString: [Function (anonymous)],
  toJSON: [Function (anonymous)]
}
{"type":"response","@timestamp":"2023-01-26T11:53:07Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/_plugins/_security_analytics/mappings","method":"post","headers":{"host":"logging","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0","a
ccept":"*/*","accept-language":"de,en-US;q=0.7,en;q=0.3","accept-encoding":"gzip, deflate, br","referer":"https://logging/app/opensearch_security_analytics_dashboards","content-type":"application/json","osd-version":"2.5.0","content-length":"239","origin":"https://logging","sec-fetch-dest
":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","te":"trailers","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-for":"172.16.1.204","connection":"close"},"remoteAddress":"172.18.0.1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0","r
eferer":"https://logging/app/opensearch_security_analytics_dashboards"},"res":{"statusCode":200,"responseTime":201,"contentLength":9},"message":"POST /_plugins/_security_analytics/mappings 200 201ms - 9.0B"}

My datastream indexes look something like .ds-winlogbeat-000003. In Detector setup I chose winlogbeat* as index pattern.

I can see, that the index template has been extended by the component template

    {
      "name": "winlogbeat",
      "index_template": {
        "index_patterns": [
          "winlogbeat*"
        ],
...
        "composed_of": [
          ".opensearch-sap-alias-mappings-component-winlogbeat"
        ]
      }
    },

The component template exists as well:

{
  "component_templates": [
    {
      "name": ".opensearch-sap-alias-mappings-component-winlogbeat",
      "component_template": {
        "template": {
          "mappings": {
            "properties": {}
          }
        },
        "version": 0
      }
    }
  ]
}

I briefly looked at the code but couldn’t really make sense of what’s happening :slight_smile:

Alex

PS: I just noticed that the current state created some hundred objects .opensearch-sap-ad_ldap… as a result of my (unsuccessful) detector-creation which prevented my indexes from rolling over as the maximum number of shards (1000) had been reached :frowning:

2

Sorry for cross-posting. Continuing here: Something broke with index templates in 2.5.0 - #6 by adn77

3

For datastreams you should pass actual datastream name as source_index, instead of index pattern. (your datastream name is winlogbeat?)

4

I reconfigured my FluentD log ingestion to use indices instead of data_streams and now the detector creation works.

How would I further debug the data_stream compatibility?

Alex

5

(moved to appropriate ‘Security Analytics’ plug category)

6

It actually works using datastreams as well. The index mapping turned out to be the problem.

1 Like
7

thanks for letting us know @adn77

8

I have datastream organized by month, is it possible to apply the detector on a index pattern of datastream ? so it automatically apply to future datastream not created yet ?