Security analytics - not able create detector

Good morning,

i’d like to ask,
i am facing issue within security analytics plugin, following errors are appearing immediately after click on security analytics from dashboard.

image
image

im trying under root and also ldap user which has mapped all_access ,
version of opensearch is. v 2.13.0

i tried add via gui role security_analytics_full_access according documentation:
reserved: true
cluster_permissions:

  • ‘cluster:admin/opensearch/securityanalytics/alerts/
  • 'cluster:admin/opensearch/securityanalytics/correlations/
  • ‘cluster:admin/opensearch/securityanalytics/detector/
  • 'cluster:admin/opensearch/securityanalytics/findings/
  • ‘cluster:admin/opensearch/securityanalytics/logtype/
  • 'cluster:admin/opensearch/securityanalytics/mapping/
  • ‘cluster:admin/opensearch/securityanalytics/rule/
    index_permissions:
  • index_patterns:
  • '
    allowed_actions:
  • ‘indices:admin/mapping/put’
  • ‘indices:admin/mappings/get’
    ( also tried via roles.yml in secret [ restart statefulset] somehow does not want to load…)

but i think is not necessary to add this role as i have mapped all_access…

some logs from opensearch-master pod

24-05-07T21:27:23.411971307+02:00 [2024-05-07T19:27:23,407][ERROR][o.o.s.u.SecurityAnalyticsException] [opensearch-cluster-master-1] Security Analytics error:
 [2024-05-07T19:30:10,817][INFO ][o.o.s.t.SecureTransportAction] [opensearch-cluster-master-1] User and roles string from thread context: a3663612| offline_access, uma_authorization],[default-roles-opensearch|all_access,security_analytics_full_access|Tenant1 
2024-05-07T21:30:10.825768530+02:00 [2024-05-07T19:30:10,825][ERROR][o.o.s.u.SecurityAnalyticsException] [opensearch-cluster-master-1] Security Analytics error: 2024-05-07T21:30:10.825792298+02:00 org.opensearch.transport.RemoteTransportException: [opensearch-cluster-master-2][10.42.0.185:9300][indices:admin/mapping/put] 
2024-05-07T21:30:10.825795687+02:00 Caused by: java.lang.IllegalArgumentException: Mapper for [name] conflicts with existing mapper:

e.g. create detector for anomaly detection works fine same as any other plugins except analytics …

log from dashboard pod:

Security Analytics - RulesService - getRules: StatusCodeError: [security_analytics_exception] [opensearch-cluster-master-2][10.42.0.185:9300][indices:admin/mapping/put]
2024-05-09T14:41:02.504010610+02:00     at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
2024-05-09T14:41:02.504017838+02:00     at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
2024-05-09T14:41:02.504023064+02:00     at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
2024-05-09T14:41:02.504028288+02:00     at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
2024-05-09T14:41:02.504034698+02:00     at IncomingMessage.emit (node:events:529:35)
2024-05-09T14:41:02.504039996+02:00     at IncomingMessage.emit (node:domain:489:12)
2024-05-09T14:41:02.504047983+02:00     at endReadableNT (node:internal/streams/readable:1400:12)
2024-05-09T14:41:02.504053509+02:00     at processTicksAndRejections (node:internal/process/task_queues:82:21) {
2024-05-09T14:41:02.504058996+02:00   status: 400,
2024-05-09T14:41:02.504064607+02:00   displayName: 'BadRequest',
2024-05-09T14:41:02.504069508+02:00   path: '/_plugins/_security_analytics/rules/_search?pre_packaged=true',
2024-05-09T14:41:02.504074652+02:00   query: {},
  body: {
2024-05-09T14:41:02.504084201+02:00     error: {
2024-05-09T14:41:02.504089134+02:00       root_cause: [Array],
2024-05-09T14:41:02.504094009+02:00       type: 'security_analytics_exception',
2024-05-09T14:41:02.504099234+02:00       reason: '[opensearch-cluster-master-2][10.42.0.185:9300][indices:admin/mapping/put]',
      caused_by: [Object]
2024-05-09T14:41:02.504108796+02:00     },
2024-05-09T14:41:02.504113767+02:00     status: 400
2024-05-09T14:41:02.504118757+02:00   },
  statusCode: 400,
  response: '{"error":{"root_cause":[{"type":"security_analytics_exception","reason":"[opensearch-cluster-master-2][10.42.0.185:9300][indices:admin/mapping/put]"}],"type":"security_analytics_exception","reason":"[opensearch-cluster-master-2][10.42.0.185:9300][indices:admin/mapping/put]","caused_by":{"type":"exception","reason":"org.opensearch.transport.RemoteTransportException: [opensearch-cluster-master-2][10.42.0.185:9300][indices:admin/mapping/put]"}},"status":400}',
2024-05-09T14:41:02.504135115+02:00   toString: [Function (anonymous)],
2024-05-09T14:41:02.504139979+02:00   toJSON: [Function (anonymous)]
2024-05-09T14:41:02.504145120+02:00 }
{"type":"response","@timestamp":"2024-05-09T12:41:02Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/_plugins/_security_analytics/rules/_search?prePackaged=true","method":"post","headers":{"host":"logs.com","x-request-id":"386c7472726da2210aebfb0fba60b0bf","x-real-ip":"10.0.254.13","x-forwarded-for":"10.0.254.13","x-forwarded-host":"logs.com","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","content-length":"86","sec-ch-ua":"\"Chromium\";v=\"124\", \"Google Chrome\";v=\"124\", \"Not-A.Brand\";v=\"99\"","dnt":"1","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36","osd-version":"2.13.0","content-type":"application/json","osd-xsrf":"osd-fetch","sec-ch-ua-platform":"\"macOS\"","accept":"*/*","origin":"https://logs.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://logs.com/app/opensearch_security_analytics_dashboards","accept-encoding":"gzip, deflate, br, zstd","accept-language":"sk-SK,sk;q=0.9,cs;q=0.8,en-US;q=0.7,en;q=0.6","priority":"u=1, i","securitytenant":"Tenant1"},"remoteAddress":"10.42.2.0","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36","referer":"https://logs.com/app/opensearch_security_analytics_dashboards"},"res":{"statusCode":200,"responseTime":66,"contentLength":9},"message":"POST /_plugins/_security_analytics/rules/_search?prePackaged=true 200 66ms - 9.0B"}

thank you for any hint
L

Hey @lejbl

Looks like two different issues.
The first one is permissions and the second one is mapping.

When you create the detector you may need to map out what fields are needed.

I seen someone posted about this on GitHub

As for permission issues it would help to show your config.yml , roles.yml & roles_mapping.yml files.

Good afternoon Gsmitt,

when i click on security analytics from menu i got :

i have nothing to choose

after click to create detector i have only choose the index or index pattern as a source, no logs type ,

on github it is my post :smiley:

here are the files:

config.yml
_meta:
  type: "config"
  config_version: 2
config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
              enabled: true
              internalProxies: '......'
              remoteIpHeader: 'x-forwarded-for'
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via local DB"
        http_enabled: true
        transport_enabled: true
        order: "1"
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern
      ldap:
        description: "Authenticate via LDAP or Active Directory"
        http_enabled: true
        transport_enabled: true
        order: "2"
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: ldap
          config:
            # enable ldaps
             enable_ssl: false
            # enable start tls, enable_ssl should be false
             enable_start_tls: false
            # send client certificate
             enable_ssl_client_auth: false
            # verify ldap hostname
             verify_hostnames: true
             hosts:
             - 10.0.0.14:389
             bind_dn: cn=admin,dc=adm
             password: rety65j8k
             userbase: 'ou=people,dc=adm'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
             usersearch: '(sAMAccountName={0})'
            # Use this attribute from the user as username (if not set then DN is used)
             username_attribute: uid
      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: "0"
        http_authenticator:
          type: openid
          challenge: false
          config:
             openid_connect_url: https://keycloak.com/realms/opensearch/.well-known/openid-configuration
             subject_key: preferred_username
             skip_users:
                  - fluentbit
                  - admin
             roles_key: roles
        authentication_backend:
          type: noop
roles.yml
_meta:
  type: "roles"
  config_version: 2
dashboard_read_only:
  reserved: true
security_rest_api_access:
  reserved: true
# Allows users to view monitors, destinations and alerts
alerting_read_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/alerting/alerts/get'
    - 'cluster:admin/opendistro/alerting/destination/get'
    - 'cluster:admin/opendistro/alerting/monitor/get'
    - 'cluster:admin/opendistro/alerting/monitor/search'
# Allows users to view and acknowledge alerts
alerting_ack_alerts:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/alerting/alerts/*'
# Allows users to use all alerting functionality
alerting_full_access:
  reserved: true
  cluster_permissions:
    - 'cluster_monitor'
    - 'cluster:admin/opendistro/alerting/*'
  index_permissions:
    - index_patterns:
        - '*'
      allowed_actions:
        - 'indices_monitor'
        - 'indices:admin/aliases/get'
        - 'indices:admin/mappings/get'
# Allow users to read Anomaly Detection detectors and results
anomaly_read_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/ad/detector/info'
    - 'cluster:admin/opendistro/ad/detector/search'
    - 'cluster:admin/opendistro/ad/detectors/get'
    - 'cluster:admin/opendistro/ad/result/search'
    - 'cluster:admin/opendistro/ad/tasks/search'
    - 'cluster:admin/opendistro/ad/detector/validate'
    - 'cluster:admin/opendistro/ad/result/topAnomalies'
# Allows users to use all Anomaly Detection functionality
anomaly_full_access:
  reserved: true
  cluster_permissions:
    - 'cluster_monitor'
    - 'cluster:admin/opendistro/ad/*'
  index_permissions:
    - index_patterns:
        - '*'
      allowed_actions:
        - 'indices_monitor'
        - 'indices:admin/aliases/get'
        - 'indices:admin/mappings/get'
# Allows users to read Notebooks
notebooks_read_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/notebooks/list'
    - 'cluster:admin/opendistro/notebooks/get'
# Allows users to all Notebooks functionality
notebooks_full_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/notebooks/create'
    - 'cluster:admin/opendistro/notebooks/update'
    - 'cluster:admin/opendistro/notebooks/delete'
    - 'cluster:admin/opendistro/notebooks/get'
    - 'cluster:admin/opendistro/notebooks/list'
# Allows users to read observability objects
observability_read_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opensearch/observability/get'
# Allows users to all Observability functionality
observability_full_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opensearch/observability/create'
    - 'cluster:admin/opensearch/observability/update'
    - 'cluster:admin/opensearch/observability/delete'
    - 'cluster:admin/opensearch/observability/get'
# Allows users to read and download Reports
reports_instances_read_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/reports/instance/list'
    - 'cluster:admin/opendistro/reports/instance/get'
    - 'cluster:admin/opendistro/reports/menu/download'
# Allows users to read and download Reports and Report-definitions
reports_read_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/reports/definition/get'
    - 'cluster:admin/opendistro/reports/definition/list'
    - 'cluster:admin/opendistro/reports/instance/list'
    - 'cluster:admin/opendistro/reports/instance/get'
    - 'cluster:admin/opendistro/reports/menu/download'
# Allows users to all Reports functionality
reports_full_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/reports/definition/create'
    - 'cluster:admin/opendistro/reports/definition/update'
    - 'cluster:admin/opendistro/reports/definition/on_demand'
    - 'cluster:admin/opendistro/reports/definition/delete'
    - 'cluster:admin/opendistro/reports/definition/get'
    - 'cluster:admin/opendistro/reports/definition/list'
    - 'cluster:admin/opendistro/reports/instance/list'
    - 'cluster:admin/opendistro/reports/instance/get'
    - 'cluster:admin/opendistro/reports/menu/download'
# Allows users to use all asynchronous-search functionality
asynchronous_search_full_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/asynchronous_search/*'
  index_permissions:
    - index_patterns:
        - '*'
      allowed_actions:
        - 'indices:data/read/search*'
# Allows users to read stored asynchronous-search results
asynchronous_search_read_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/asynchronous_search/get'
# Allows user to use all index_management actions - ism policies, rollups, transforms
index_management_full_access:
  reserved: true
  cluster_permissions:
    - "cluster:admin/opendistro/ism/*"
    - "cluster:admin/opendistro/rollup/*"
    - "cluster:admin/opendistro/transform/*"
  index_permissions:
    - index_patterns:
        - '*'
      allowed_actions:
        - 'indices:admin/opensearch/ism/*'
# Allows users to use all cross cluster replication functionality at leader cluster
cross_cluster_replication_leader_full_access:
  reserved: true
  index_permissions:
    - index_patterns:
        - '*'
      allowed_actions:
        - "indices:admin/plugins/replication/index/setup/validate"
        - "indices:data/read/plugins/replication/changes"
        - "indices:data/read/plugins/replication/file_chunk"
# Allows users to use all cross cluster replication functionality at follower cluster
cross_cluster_replication_follower_full_access:
  reserved: true
  cluster_permissions:
    - "cluster:admin/plugins/replication/autofollow/update"
  index_permissions:
    - index_patterns:
        - '*'
      allowed_actions:
        - "indices:admin/plugins/replication/index/setup/validate"
        - "indices:data/write/plugins/replication/changes"
        - "indices:admin/plugins/replication/index/start"
        - "indices:admin/plugins/replication/index/pause"
        - "indices:admin/plugins/replication/index/resume"
        - "indices:admin/plugins/replication/index/stop"
        - "indices:admin/plugins/replication/index/update"
        - "indices:admin/plugins/replication/index/status_check"
roles_mapping.yml
_meta:
  type: "rolesmapping"
  config_version: 2
all_access:
  reserved: false
  backend_roles:
  - "admin"
  - "fluentbit"
  users:
  - "a23423"
  - "a2345"
  - "a53524"
  - "a423245"
  - "a234524"
  description: "Maps admin to all_access"
readall:
  reserved: false
  backend_roles:
  - "readall"
  - "opensearch_readonly"
manage_snapshots:
  reserved: false
  backend_roles:
  - "snapshotrestore"
dashboard_server:
  reserved: true
  users:
  - "dashboarduser"

Thank You