Pager Duty Custom Webhook - Destination

Plugins Alerting
1

I am running AWS Elasticsearch and want to set up an Alert to Pager Duty. I have set up a service in PD, and can post an event to the service using;

curl -H “Content-type: application/json” -X POST https://events.pagerduty.com/generic/2010-04-15/create_event.json -d ‘{ “service_key”: “<service_key>”, “event_type”: “trigger”, “description”: “pd-test” }’

How to I configure the destination using custom webhook, query parameters and Header information from the Kibana UI? I’ve tried this format but it’s not working;

58
58619×1243 46.2 KB

I’ve set up a trigger and added the Custom Webhook as an Action;

25
251551×856 68.8 KB

I have read that the body of parameteres following -d should be entered in the ‘message’ of the alerting Action, however this also does not work. Please can you assist, thank you.

2

Hi @Swaps,

You mentioned you put the body in the “message” of the Action, but the images look like you put them in the query parameters instead?

Also make sure you follow the documentation for Authorization:
https://v2.developer.pagerduty.com/docs/authentication

If this still does not work, you can open your network request tab in your browser through the developer tools and use that “Send test message”. You should get a response that hopefully has a more clear reason for why it failed.

3

Hi @dbbaughe,

Thanks for the response - I did try putting the body in the message of the action in addition to trying the query parameters method above, but I did not screenshot this. It looks like the topic where I read that was more about how to configure a message so that it would appear in the alert, so I think my params need to be configured in the Destination section.

Am I right to enter the ‘query parameters’ as the body parameters;

36
36779×1257 51 KB

The Pager Duty service generated an ‘Integration key’, which I pass as a ‘service_key’ in the curl command, however I have not generated an API token. Are both required for the destination?

The network request tab responded with this when I used the ‘Send test message’ facility;

{“Send to Pager Duty ‘Elasticsearch Alerting’ Svc”:{“name”:“Send to Pager Duty ‘Elasticsearch Alerting’ Svc”,“output”:{},“throttled”:false,“error”:“java.io.IOException: Failed: HttpResponseProxy{HTTP/1.1 400 Bad Request [Server: nginx, Date: Tue, 09 Jul 2019 17:25:03 GMT, Content-Type: application/json; charset=utf-8, Connection: keep-alive] org.apache.http.client.entity.DecompressingEntity@3c518d8b}”}}}}}}

I’m not authorized to generate an API token but I will give that a go - I’ll update the thread.

Thank you!

4

Just looking at the Pager Duty docs again, and it looks like the API token authentication is used for the REST API v2 - whereas I am using the Events API v2;

The REST API provides a way for third parties to connect to a PagerDuty account and access or manipulate configuration data on that account. It is not for connecting your monitoring tools to send events to PagerDuty; for that, use the Events API.

There is a utility in PD where you can send a v2 event, and using params this works;

routing_key: <integration key
event_action: trigger
payload.summary: test
payload.source: test
payload.severity: info

You can pass the routing/integration key as a ‘X-Routing-Key’ header, but if it is not set it will use the routing_key param.

Can I confirm what the syntax is for the params when entering in the fields of the destination? Do we need double quotes or colons? I am still getting the ‘400 Bad Request’ so I’m wondering if it is a syntax issue.

Thank you.

5

Hi @dbbaughe,

To clarify, using the developer tools - network utility to 'Send test message, I get the following error;

“output”:{},“throttled”:false,“error”:"java.io.IOException: Failed: HttpResponseProxy{HTTP/1.1 400 Bad Request

I have read in another discussion that I need to enter the parameters ‘in (the) message body of action’, but I’m unsure where this is. I can only see actions as part of the trigger, (i.e. where to send if a trigger condition is met), and the message part here would be the message that is sent to the destination. For this reason it seems odd to be putting the service_key in the message, as this would then be exposed in the message every time an alert was raised.

Any assistance you could provide would be very much appreciated!

6

Hi @Swaps,

Sorry for the delay, I’ll take a look today and try to replicate the issue using a PagerDuty account.

Thanks,
Drew

7

Hi @Swaps,

I created a test PD account and was able to publish an alert in this way:

31%20AM
31%20AM747×923 34.6 KB

23%20AM
23%20AM1248×597 33.5 KB

One thing to note, I was getting a similar 400 error until I removed the single quotes I had around the action message body.

Let me know if this helps!

Drew

3 Likes
8

Hi @dbbaughe,

Thank you! That has worked! I really appreciate the screenshots and particular detail to the syntax, which is where I think I was tripping up.

In the meantime I had also tried the ‘Custom Event Transform’ integration in PD, and managed to get this working straight away. For this I had all my parameters in the part where the webhook is setup, and nothing in the message section.

Really appreciate all your help with this, thanks!

9

I just wanted to update on the message part - I wanted to construct a more meaningful message using the mustache templating, which calls various values from the query results;

{ “event_action”: “trigger”, “payload”: { “summary”: “Tripwire has detected {{ctx.results.0.hits.total}} file changes on{{#ctx.results.0.hits.hits}} {{_source.host.name}};
{{{_source.message}}} {{/ctx.results.0.hits.hits}}”, “source”: “Alerting ES Plugin”, “severity”: “info”}}

However can I check the syntax is correct? It appears to only alert when there are no hits using the ‘send test message’ button. When there are hits, only the ‘{{ctx.results.0.hits.total}}’ field is passed ok, those inside the ‘Section’ are ignored, despite the message preview box displaying what I expect to see.

Thanks,
Swapna

10

UPDATE;

Turns out I did not need the triple curly brackets. The message preview does not seem to be consistent with whatever is parsed to the PD alert - I needed curly brackets for the message preview to be correct, but double brackets for the PD message to be correct (with a ‘garbled’ message preview). This one works in the PD alert;

{ “event_action”: “trigger”, “payload”: { “source”: “Alerting ES Plugin”, “severity”: “info”, “summary”: “Tripwire has detected {{ctx.results.0.hits.total}} file changes;{{#ctx.results.0.hits.hits}}{{_source.beat.hostname}} {{_source.message}}{{/ctx.results.0.hits.hits}}” } }

Now I can send the alert using ‘test send message’ button. However, using the trigger, the alert goes into ‘ERROR’ state. In developer tools, the error code is;

HttpResponseProxy{HTTP/1.1 400 Bad Request [Server: nginx, Date: Thu, 25 Jul 2019 07:50:11 GMT, Content-Type: application/json; charset=utf-8, Connection: keep-alive]

I fear this is a syntax issue again… any ideas?

Thank you!

11

My bad - I had not updated the trigger. It appears to be working now.

1 Like
12

Have anyone tried adding additional details in

payload.custom_details