Alert trigger not sending SNS message or test message

Plugins Alerting
1

Hello,

I have an alert that when triggered it does not send SNS alert and the “Submit Test Message” also does not send email alert.

Any ideas on why this is happening in Kibana?

Thanks,

Freddie2020

2

Is there not someone who has run into this issue or maybe have a clue to why possibly the SNS and email alerts are not going through even though I run a test message an the run test “true” but no alert?

Any help appreciated!

3

Hi Freddie,

If you’re using Amazon Elasticsearch Service, I suggest you check the settings from both Alerting side and SNS side.

  1. The SNS topic and IAM role are correct in the Alerting destination.
  2. The IAM role has got correct permission and trust relationship.
  3. The SNS topic has got validate subscription.

While this forum covers ODFE, and you may also request help through AWS support.

Tianli

4

Hi @ftianli ,

I have verified the SNS topic and IAM role are correct in the Alerting destination.
I have checked IAM role has got correct permissions and verified via documentation.
I have also checked that SNS has been subscribed and validated.

Is there anything else I can check or that could cause this issue.

Thanks,

FREDDIE2020

5

Hi @FREDDIE2020
I think you could check the alert information which stores the index.
The ongoing alerts are stored in .opendistro-alerting-alerts, the completed alerts are stored in .opendistro-alerting-alert-history-<date>.
And see if there are any error messages there.
For example:
To check an ongoing alert for a specific monitor,

GET .opendistro-alerting-alerts/_search
{
    "query": {
        "term": {
            "monitor_id": "XXX"
        }
    }
}
1 Like
6

Thanks @ftianli
Let me check that and get back to you.

FREDDIE2020

7

Hope can help you solve the issue. :+1:

1 Like
8

Hi @ftianli,

Here is the error I can see after executing the GET .opendisto-alerting-alerts/_search with monitor_id:

“state” : “ERROR”,
“error_message” : “Error running action:\nUser: arn:aws:sts::xxxxxxxxxxxx:assumed-role/cp-sts-grant-role/south-us-east-1-prod-012511484642 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::xxxxxxxxxxxx:role/core/tfsesprod-monitoring-admin (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: 57d4a32a-fb0b-4499-ac29-5f34ab21f464)”,

Any ideas on what this error means or what is needed to resolve this?

Thanks again @ftianli