New to OpenSearch: Send notification on every update

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
V2.16.0
Describe the issue:
I’m new to OpenSearch; I have set up OpenSearch using fluent-bit to publish logs in OpenSearch. I have this all set up with indexes, and I can discover the results. All is working as expected.

I need to send a custom Webhook notification with the details of each new entry.

I have set up the channel notification and can send a test message.

Where I’m struggling is setting up the monitor and trigger. I have followed verious articles on the web with some success.

When the trigger fires, I receive it through the webhook, but I only have the cx.monitor info, I can not get it to send me the details.

Assuming what I’m trying to do has been done a thousand times.

fluent-bit Update OpenSearch

OpenSearch sends the new entry details to the custom webhook endpoint.

Could I get help setting up a monitor for that and configuring the message to include the details of the entry?

Happy to be pointed to a good article, all I have be able to found have been to generic.

Thanks in advanced

Harry

2

You can create a document level monitor, and use the sample_documents variables in the trigger to include the document detail in the notification message, take a look at the official documentation: Triggers - OpenSearch Documentation.

And here is my local test monitor, hope it helps:

{
   "name": "test1",
   "type": "monitor",
   "monitor_type": "doc_level_monitor",
   "enabled": false,
   "schedule": {
      "period": {
         "unit": "MINUTES",
         "interval": 1
      }
   },
   "inputs": [
      {
         "doc_level_input": {
            "description": "",
            "indices": [
               "sso_logs-nginx-prod"
            ],
            "queries": [
               {
                  "id": "052b0592-792f-4970-9b4b-83ade0e59e58",
                  "name": "xx",
                  "query": "http.response.status_code:\"200\"",
                  "tags": []
               }
            ]
         }
      }
   ],
   "triggers": [
      {
         "document_level_trigger": {
            "id": "V8zWS5EBtjpg6KVXAkoO",
            "name": "test1",
            "severity": "1",
            "condition": {
               "script": {
                  "source": "query[name=xx]",
                  "lang": "painless"
               }
            },
            "actions": [
               {
                  "id": "notification849682",
                  "name": "ssss",
                  "destination_id": "KpLVS5EBNirD1Y6eGfMF",
                  "message_template": {
                     "source": "Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n  - Trigger: {{ctx.trigger.name}}\n  - Severity: {{ctx.trigger.severity}}\n  - Period start: {{ctx.periodStart}}\n  - Period end: {{ctx.periodEnd}}\n{{#ctx.alerts}}\n    Sample documents:\n    {{#sample_documents}}\n        Index: {{_index}}\n        Document ID: {{_id}}\n       \n        Order date: {{_source.@timestamp}}\n        Order ID: {{_source.observedTimestamp}}\n        Clothing category: {{_source.span_id}}\n        -----------------\n    {{/sample_documents}}\n{{/ctx.alerts}}",
                     "lang": "mustache"
                  },
                  "throttle_enabled": false,
                  "subject_template": {
                     "source": "Alerting Notification action",
                     "lang": "mustache"
                  },
                  "action_execution_policy": {
                     "action_execution_scope": {
                        "per_alert": {
                           "actionable_alerts": []
                        }
                     }
                  }
               }
            ]
         }
      }
   ],
   "ui_metadata": {
      "schedule": {
         "timezone": null,
         "frequency": "interval",
         "period": {
            "unit": "MINUTES",
            "interval": 1
         },
         "daily": 0,
         "weekly": {
            "tue": false,
            "wed": false,
            "thur": false,
            "sat": false,
            "fri": false,
            "mon": false,
            "sun": false
         },
         "monthly": {
            "type": "day",
            "day": 1
         },
         "cronExpression": "0 */1 * * *"
      },
      "monitor_type": "doc_level_monitor",
      "doc_level_input": {
         "queries": [
            {
               "id": "052b0592-792f-4970-9b4b-83ade0e59e58",
               "queryName": "xx",
               "field": "http.response.status_code",
               "operator": "is",
               "query": "200",
               "tags": []
            }
         ]
      },
      "search": {
         "searchType": "graph"
      }
   }
}
3

@gaobinlong ,

Thank you for the quick response.

I came up with this, and it works with some caveats.

{
	"alerts": [
		{{#ctx.alerts}}
		{{#sample_documents}}
		{
			"index": "{{_index}}",
			"documentID": "{{_id}}",
			"timestamp": "{{_source.@timestamp}}",
			"event": {
				"nodeId": "{{_source.event.nodeId}}",
				"filespace": "{{_source.event.filespace}}",
				"filespaceUuid": "{{_source.event.filespaceUuid}}"
			},
			"source": {
				"offset": "{{_source.source.offset}}",
				"filename": "{{_source.source.filename}}"
			},
			"operation": {
				"action": "{{_source.operation.action}}",
				"entryPath": "{{_source.operation.entryPath}}",
				"fileId": "{{_source.operation.fileId}}"
			},
			"device": {
				"hostName": "{{_source.device.hostName}}",
				"osVersion": "{{_source.device.osVersion}}",
				"osName": "{{_source.device.osName}}"
			},
			"user": {
				"name": "{{_source.user.name}}",
				"id": "{{_source.user.id}}"
			}
		}{{^-last}}, {{/-last}}
		{{/sample_documents}}
		{{/ctx.alerts}}
	]
}

The mustache {{^-last}}, {{/-last}} is not honored; the last trailing slash causes an issue with receiving JSON.

I ended up sending htnl/text to my endpoint, running a quick regex to remove any hanging commas, and parsing the string to JSON.

incomingPayload.replace(/,(?=\s*[}\]])/mig,'');
incomingPayload = JSON.parse(incomingPayload);

This works and is a great starting point.

Again, due to the limits of the implemented mustache, I was unable to parse

"source": {
  "offset": "{{_source.source.offset}}",
  "filename": "{{_source.source.filename}}"
 },

The property names in OpenSearch have dots; I’m unsure if it’s possible to rename these.

“source.offset”: “Value”,
“source.filename”: “Value”

One more question;
As these are alerts coming through, I’m assuming it is best practice to do a follow-up call once the payload is received and acknowledge these alerts via the API.

Thanks
Harry