Alerting / Monitors

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 3.8

Describe the issue: Members sharing at least one common backend role are not able to see the alerts or monitor definition. When I look at the monitor definition, I do not see the backend roles.

Configuration:

We have enabled this in our OS cluster.

plugins.alerting.filter_by_backend_roles: true

Relevant Logs or Screenshots: No errors or messages in the cluster or dashboard logs. The top level objects are.

"monitor": {
    "type": "monitor",
    "schema_version": 8,
    "name": "OS: Latency > 4000 per proxy",
    "monitor_type": "query_level_monitor",
    "enabled": true,
    "enabled_time": 1783587970198,
    "schedule": {},
    "inputs": [],
    "triggers": [],
    "last_update_time": 1787319302777,
    "data_sources": {},
    "delete_query_index_in_every_run": false,
    "should_create_single_alert_for_findings": false,
    "owner": "alerting"
  }
2

We have a scenario where users cannot be in the same Domain group (onshore/offshore), how do we make sure that an alert created by onshore team is visible to offshore domain group.

3

@muraliv Could you share your config.yml file? Please hide all sensitive information.

4

Hi @pablo

Greetings. Here is the sanitized version of the config.yml file.

---
_meta:
  type: "config"
  config_version: 2
config:
  dynamic:
    api_tokens:
      enabled: true
      max_duration_seconds: 71536000
      max_tokens: 100
    kibana:
      multitenancy_enabled: true
      private_tenant_enabled: true
      default_tenant: "Admins"
      server_username: "kibanaserver"
      index: ".kibana"
    do_not_fail_on_forbidden: false
    http:
      anonymous_auth_enabled: false
    authc:
      internal_auth:
        description: "Authenticate via internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: "basic"
          challenge: true
        authentication_backend:
          type: "intern"
      ldap_auth:
        description: "Active Directory authentication"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: "basic"
          challenge: false
        authentication_backend:
          type: "ldap"
          config:
            enable_ssl: true
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            pemtrustedcas_filepath: "certs/ldap-ca-bundle.pem"
            hosts:
            - "mydomain.com:636"
            follow_referrals: false
            bind_dn: "REDACTED"
            password: "REDACTED"
            userbase: "REDACTED"
            usersearch: "(sAMAccountName={0})"
            username_attribute: "sAMAccountName"
            connect_timeout: 5000
            response_timeout: 5000
            skip_users:
            - "admin"
            - "kibanaserver"
            - "kibanaro"
            - "logstash"
            - "readall"
            - "snapshotrestore"
    authz:
      ldap_authz:
        description: "LDAP group authorization"
        http_enabled: true
        transport_enabled: true
        authorization_backend:
          type: "ldap"
          config:
            enable_ssl: true
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            pemtrustedcas_filepath: "certs/ldap-ca-bundle.pem"
            hosts:
            - "mydomain.com:636"
            follow_referrals: false
            bind_dn: "REDACTED"
            password: "REDACTED"
            userbase: "REDACTED"
            usersearch: "(sAMAccountName={0})"
            rolebase: "OU=Groups,DC=win,DC=ad,DC=corp,DC=com"
            rolesearch: "(&(|(cn=SVC-elm-*)(cn=CISELM_*)(cn=ELM_*)(cn=elm_*))(member={0}))"
            userroleattribute: null
            userrolename: "disabled"
            rolename: "cn"
            resolve_nested_roles: false
            connect_timeout: 5000
            response_timeout: 5000
            skip_users:
            - "admin"
            - "kibanaserver"
            - "kibanaro"
            - "logstash"
            - "readall"
            - "snapshotrestore"
5

@muraliv I’ve tested alerting with LDAP and backed roles. I had no issues accessing monitors with different LDAP users that share one or more backend roles.
The API below doesn’t return backend roles because they’re stored in the .opendistro-alerting-config index.

GET _plugins/_alerting/monitors/_search

This index can be searched only with a superadmin certificate defined in plugins.security.authcz.admin_dn in opensearch.yml

curl --insecure --cert config/kirk.pem --key config/kirk-key.pem --cacert config/root-ca.pem https://localhost:9200/.opendistro-alerting-config/_search?pretty

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".opendistro-alerting-config",
        "_id" : "a7ebz6AB_2asX442wstt",
        "_score" : 1.0,
        "_source":{"monitor":{"type":"monitor","schema_version":0,"name":"monitor1-osuser1","monitor_type":"query_level_monitor","user":{"name":"osuser1","backend_roles":["os-hr","os-supervisor","kibanauser"],"roles":["kibana_user","alerting"],"user_requested_tenant":null,"user_requested_tenant_access":null,"custom_attribute_names":[]},"enabled":true,"enabled_time":1790189486510,"schedule":{"period":{"interval":1,"unit":"MINUTES"}},"inputs":[{"search":{"indices":["security*"],"query":{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"from":"{{period_end}}||-1h","to":"{{period_end}}","include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}}}}],"triggers":[],"last_update_time":1790189486510,"ui_metadata":{"schedule":{"cronExpression":"0 */1 * * *","period":{"unit":"MINUTES","interval":1},"timezone":null,"daily":0,"monthly":{"type":"day","day":1},"weekly":{"tue":false,"wed":false,"thur":false,"sat":false,"fri":false,"mon":false,"sun":false},"frequency":"interval"},"search":{"searchType":"graph","bucketValue":1,"timeField":"@timestamp","bucketUnitOfTime":"h","groupBy":[],"filters":[],"aggregations":[]},"monitor_type":"query_level_monitor"},"data_sources":{"query_index":".opensearch-alerting-queries","findings_index":".opensearch-alerting-finding-history-write","findings_index_pattern":"<.opensearch-alerting-finding-history-{now/d}-1>","alerts_index":".opendistro-alerting-alerts","alerts_history_index":".opendistro-alerting-alert-history-write","alerts_history_index_pattern":"<.opendistro-alerting-alert-history-{now/d}-1>","comments_index":".opensearch-alerting-comments-history-write","comments_index_pattern":"<.opensearch-alerting-comments-history-{now/d}-1>","query_index_mappings_by_type":{},"findings_enabled":false},"delete_query_index_in_every_run":false,"should_create_single_alert_for_findings":false,"owner":"alerting"}}

According to the documentation, the Alerting plugin has an additional option, plugins.alerting.filter_by_backend_roles_access_strategy, for filtering backend roles.
Did you set it in your cluster configuration?

6

Hi @pablo

Thank you for testing this at your end. I ran the query for a specific monitor_id and do not see the output like what you have. I have that setting set to “default” which is intersect.

{
  "_index" : ".opendistro-alerting-config",
  "_id" : "oXX4ap8BBk3d7Yfex3Kg-metadata",
  "_version" : 2,
  "_seq_no" : 6,
  "_primary_term" : 1,
  "found" : true,
  "_source":{"metadata":{"monitor_id":"oXX4ap8BBk3d7Yfex3Kg","last_action_execution_times":[]}}
}

I can see the details of the monitor in DEV tools though.

Thanks

Murali

7

@muraliv Did you run that query with the superadmin certificate as I did? Is that certificate listed in the plugins.security.authcz.admin_dn? Do you use custom or demo SSL certificates?
Only that certificate allows you to see that index content.

Could you share your _cluster/settings?

8

Hi @pablo

The problem is that the users do not share a common backend role (because of onshore vs offshore). We are looking at a workaround to make this work. I am not getting the output that includes the user info & all of the backend roles, as I am seeing in your output.

Yes, I used the admin certificate listed in the opensearch.yml file under plugins.security.authcz.admin_dn. We are using custom certificates and the cluster settings exceeding the character limit. What’s the best way to share this info ?

Thanks

Murali

```

9

Hi @pablo ,

Greetings. These are the results when I run the search against the .opendistro-alerting-config index.

{"_index":".opendistro-alerting-config","_id":"oXX4ap8BBk3d7Yfex3Kg-metadata","_version":2,"_seq_no":6,"_primary_term":1,"found":true,"_source":{"metadata":{"monitor_id":"oXX4ap8BBk3d7Yfex3Kg","last_action_execution_times":[]}}}

thanks

Murali

10

I’m a bit lost here. So do your users share the same backend role?
Did you create your monitors via the API or OpenSearch Dashboards?

Did you use LDAP or an internal user to create the monitor? If LDAP, could you share the authinfo or check if backend roles are returned?

curl --insecure -u <ldap_user>:<password> https://localhost:9200/_plugins/_security/authinfo?pretty

The interesting part is that you don’t get the monitor definition in that index; you get only metadata, and another document with _id: oXX4ap8BBk3d7Yfex3Kg should contain the definition of your monitor. As far as I’m aware, there is no setting that would split a monitor’s metadata and definition into different indices.

11

@muraliv I’ve found an alternative solution to your issue.

This is currently experimental but worth checking, as it could fit your needs.

12

Hi @pablo ,

We have users in two locations (onshore & offshore) and they cannot share a common backend role. One of the offshore users created the monitor using the UI. We only use LDAP for authentication. The user is offline at the moment, will share the auth info as soon as I can.

I can get the monitor details in DEV tools by running this command.

GET _plugins/_alerting/monitors/_search
{
  "query": {
    "match_all": {}
  }
}
13

Hi @pablo ,

I tried the workaround and it did not help.

Thanks

Murali

14

What can I do to find the root cause for this behavior ?

Thanks

Murali

15

Hi @pablo

I got the user details for the monitor in question. I had to include this in the query to display those details.

.opendistro-alerting-config/_doc/oXX4ap8BBk3d7Yfex3Kg?_source=monitor.user.*,monitor.name&pretty

"_source":{"monitor":{"name":"OS: Apigee np latency > 4000 per cluster","user":{"backend_roles":["elm_jtpv_prod_offshore_admin","elm_jeev_nonprod_offshore_admin","CISELM_D0MV_24824_PROD_AWS","elm_jeev_prod_offshore_admin","elm_jtpv_nonprod_offshore_admin","CISELM_D0MV_24824_DEV_AWS","elm_d0mv_prod_offshore_admin","elm_d0mv_nonprod_offshore_admin"],"custom_attribute_names":[],"roles":["own_index","elm_jeev_role","elm_d0mv_role","elm_jtpv_role"],"name":"PAMNANE","user_requested_tenant":null,"user_requested_tenant_access":null}}}
}
16

Hi @pablo

I tried to migrate from the legacy to the new option that you shared with me and I am getting this error. Tried the migration in DEV tools and also with admin certificate.

```
{“error”:{“root_cause”:[{“type”:“illegal_state_exception”,“reason”:“No allowed indices configured”}],“type”:“illegal_state_exception”,“reason”:“No allowed indices configured”},“status”:500}
```

17

Hi @pablo

We just upgraded our sandbox environment to 3.9 and it works without any issues. How can we make it work in 3.8 ?

Thanks

Murali

18

@muraliv do you mean that resource sharing worked for you in 3.9.0?

19

Hi @pablo

I did not get an error when I tried to migrate the monitor. When I searched that monitor, I am seeing this in the output (not available in 3.8).


          "all_shared_principals": [
            "user:venkmu9",
            "backend:elm_jozv_prod_onshore_admin",
            "backend:elm_hvuv_prod_onshore_admin"
          ]

I would like to know how can I make it work in 3.8. How do I specify the allowed indices in 3.8 ?

No allowed indices configured
20

@muraliv I think this could be working as it is no longer experimental in 3.9.0

image
image1025×454 72.6 KB

image
image1127×553 81 KB