Mustache Template function / Removing trailing comma?

Plugins Alerting
1

What do we do when dealing with JSON object on webhooks? Current templating with aggregations basically makes it impossible due to trailing commas.

For example, using any kind of templating requires iterations over an object in the ES response object. However, every json iteration that will be seperated by a comma will always have a trailing comma like so:

{
  "obj1": "val1",
  "obj2": "val2",
  "obj3": "val3",
  "obj4": "val4", <- Sadface
}

The vast majority of JSON compliant webhooks will reject this / fail to decode the JSON object on the other end.

Any suggestions on how to get around this?

2

Hi @Dandy,

Could you give an example of where you’re getting the trailing comma from?

Thanks

3

Hello @dbbaughe

Sorry for the delay, but take any instance example where you’re iterating over an object. Lets say we have a webhook that is triggered when events are received. The webhook, as most do, expects JSON. Our ES response are aggrgated like so:

"aggs": { "2": { "keyword": "xyz", "aggs": [{ "keyword": "123" }, { "keyword": "456" }] }}

Well, you could iterate over these results in Mustache pretty easily, but this has an inherent flaw in building a JSON object.

{
  "device": {
     "name": "xyz",
     "properties": ["123", "456",] <----- 
  }

Example here is invalid JSON and most recipient webhooks won’t accept it because of the trailing comma. You can build the object any way you want, but no matter what, if you’re iterating over anything in the ES response object, you’ll end up with a trailing comma.

This is an inherent flaw in Mustache, not ES, but because webhooks traditionally expect JSON, normally to counter this you’d have a Mustache function, or JS to strip out the trailing comma, which we don’t have the option of doing here.

4

@dbbaughe Any ideas on this or an approach we could take to resolving this? I’m surprised more people who work with JSON via webhooks or the like aren’t experiencing a similar issue.

5

Hi @Dandy

In our backend code we convert the response from the mustache templates to a string and then send that string as a payload to the Webhooks. The web-hooks (chime / slack) currently don’t have the response payload in the json format.

Can you share the exact queries and the exact templates which you are using the build the response for the web-hook? It ll help me deep dive into this issue and debug it further.

6

@aditjind I don’t think an explicit example is needed here. I think this is just a fundamental issue with the limitations of Mustache templating offered in Opendistro.

Because you’re iterating over an object array to build a JSON. This iteration involves adding a , at the end of each iteraction of an object. E.G;

{{#ctx.results.0}}
{{#aggregations}}
{{#cluster.buckets}}
{{key}}: {{value}},
{{/cluster.buckets}}
{{/aggregations}}
{{/ctx.results.0}}

Lets pretend this outputs;

{
“key1”: “value1”,
“key2”: “value2”,
“key3”: “value3”,
“key4”: “value4”, <---- Bad comma (Expected)
}

Here’s a good discussion about it; In Mustache templating is there an elegant way of expressing a comma separated list without the trailing comma? - Stack Overflow

The issue is more Mustache as a templating engine, rather the Opendistro.

However, Opendistro offers no real solutions to workaround this issue. Meaning it is impossible to generate a valid JSON from alerts (AFAIK).

7

@Dandy I understand the problem of Mustache not generating a proper Json, but in open-distro we compile the template and store the output in the form of a string. And then that string is sent as payload for the content key. The problem I suspect you might be facing here is that when you try to unmarshal the string payload from the web-hook into a json for consumption on your end, that might be throwing error as the string might not be a valid json. But we can’t have validation of the message string because we don’t have restrictions on that string being json compliant for users, as message doesn’t necessarily have to be a json.

I hope I am understanding the problem correctly here.

8

@aditjind Yes, its the consumer that is failing because of unmarshaling errors.

The original problem isn’t that anything is being sent incorrectly.

Its that we have no valid way, as far as I can see, to produce a valid JSON payload from Opendistro ES.

9

have you tried using


{{#toJson}}ctx.results.0{{/toJson}}

you can also use painless scripts in trigger condition to format the ctx results in the way you would like and then used use simple templates in actions section

10

Hello i am facing the same issue have you found a solution ?

11

This is most likely not the answer you are looking for, but I use a regular expression on the webhook side.

I’m using a custom Webhook in OpenSearch, which I set to send text/html.

On my receiving end (node.js), I use a regular expression to remove stray commas.

replace(/,(?=\s*[}\]])/mig,'');

That leaves me with a valid JSON string, which I then parse the string to a JSON object.

Harry

1 Like
12

Finally i found a solution i do not use Opensearch Alerte anymore because it was to limitated (mustache format etc …) Now i use ElastAlerte 2 it is a better solution to build Alertes into Opensearch and export It into an other service/alerte manager

13

That’s a great point!