Hi everyone,
I made an alarm definition on OpenSearch Dashboard. In a simple way, I want it to generate an alarm when it finds the word “error” in my specific index. Then it should send me an email after the alarm is generated.
I made all the necessary definitions on OpenSearch Dashboard. When I check with preview, I can confirm that my definitions are correct. I enable the alarm definition. Then, when I watch it on Dashboard, I see that it generates an alarm about the error and the trigger function works. But when it sends me an email, it cannot establish a connection with my smtp server. I see this error on the screen.
I defined “Channels”, “Email senders” and “Email recipient groups” on the Notification menu. When I click on Channels > Notification Definition > Action / “send test message” button, it sends an email. I can see the test email in my mailbox. Similarly, I can send a test email in my alarm definition.
But the system finds the word “error”, runs the trigger function but cannot send an email using my smtp channel.
I am attaching all screenshots and related logs. How can I find a solution to this issue, can you support me?
** yum list installed | grep opensearch**
opensearch.x86_64 2.11.1-1 @opensearch-2.x
opensearch-dashboards.x86_64 2.11.1-1 @opensearch-dashboards-2.x
** tail -f /Products/opensearch/logs/my-opensearch-cluster.log**
tail -f /Products/opensearch/logs/my-opensearch-cluster.log
[2024-02-07T10:56:34,443][INFO ][o.o.a.t.TransportExecuteMonitorAction] [myserver3] Executing monitor from API - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-07T07:55:00Z, periodEnd: 2024-02-07T07:56:34.442Z, dryrun: true
[2024-02-07T10:56:34,444][INFO ][o.o.a.MonitorRunnerService] [myserver3] Executing scheduled monitor - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-07T07:55:00Z, periodEnd: 2024-02-07T07:56:34.442Z, dryrun: true, executionId: _2024-02-07T07:56:34.444399112_7ceea165-042e-489e-85c6-ec464090f368
[2024-02-07T10:56:34,635][INFO ][o.o.n.r.NotificationConfigRestHandler] [myserver3] notifications:executeGetRequest from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group}
[2024-02-07T10:56:34,636][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-get org.opensearch.commons.notifications.action.GetNotificationConfigRequest@58d26ff1
[2024-02-07T10:56:34,636][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-getAll
[2024-02-07T10:56:34,641][INFO ][o.o.n.i.NotificationConfigIndex] [myserver3] notifications:getAllNotificationConfigs from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group} retCount:4, totalCount:4
[2024-02-07T10:56:34,664][INFO ][o.o.n.r.NotificationConfigRestHandler] [myserver3] notifications:executeGetRequest from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group}
[2024-02-07T10:56:34,665][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-get org.opensearch.commons.notifications.action.GetNotificationConfigRequest@57db5ef1
[2024-02-07T10:56:34,665][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-getAll
[2024-02-07T10:56:34,670][INFO ][o.o.n.i.NotificationConfigIndex] [myserver3] notifications:getAllNotificationConfigs from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group} retCount:4, totalCount:4
[2024-02-07T10:56:44,434][INFO ][o.o.n.r.NotificationConfigRestHandler] [myserver3] notifications:executeGetRequest from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group}
[2024-02-07T10:56:44,435][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-get org.opensearch.commons.notifications.action.GetNotificationConfigRequest@d7c8ce1
[2024-02-07T10:56:44,435][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-getAll
[2024-02-07T10:56:44,441][INFO ][o.o.n.i.NotificationConfigIndex] [myserver3] notifications:getAllNotificationConfigs from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group} retCount:4, totalCount:4
[2024-02-07T10:56:58,689][INFO ][o.o.n.r.NotificationConfigRestHandler] [myserver3] notifications:executeGetRequest from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group}
[2024-02-07T10:56:58,690][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-get org.opensearch.commons.notifications.action.GetNotificationConfigRequest@7c19f5ad
[2024-02-07T10:56:58,690][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-getAll
[2024-02-07T10:56:58,695][INFO ][o.o.n.i.NotificationConfigIndex] [myserver3] notifications:getAllNotificationConfigs from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group} retCount:4, totalCount:4
[2024-02-07T10:57:07,203][INFO ][o.o.n.r.NotificationConfigRestHandler] [myserver3] notifications:executeGetRequest from:0, maxItems:10, sortField:name, sortOrder=asc, filters={config_type=slack,email,chime,microsoft_teams,webhook,sns}
[2024-02-07T10:57:07,204][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-get org.opensearch.commons.notifications.action.GetNotificationConfigRequest@6a3deb3f
[2024-02-07T10:57:07,204][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-getAll
[2024-02-07T10:57:07,210][INFO ][o.o.n.i.NotificationConfigIndex] [myserver3] notifications:getAllNotificationConfigs from:0, maxItems:10, sortField:name, sortOrder=asc, filters={config_type=slack,email,chime,microsoft_teams,webhook,sns} retCount:1, totalCount:1
[2024-02-07T10:57:08,295][INFO ][o.o.n.r.NotificationConfigRestHandler] [myserver3] notifications:executeGetRequest from:0, maxItems:10, sortField:name, sortOrder=asc, filters={config_type=slack,email,chime,microsoft_teams,webhook,sns}
[2024-02-07T10:57:08,296][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-get org.opensearch.commons.notifications.action.GetNotificationConfigRequest@3160917
[2024-02-07T10:57:08,296][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-getAll
[2024-02-07T10:57:08,299][INFO ][o.o.n.i.NotificationConfigIndex] [myserver3] notifications:getAllNotificationConfigs from:0, maxItems:10, sortField:name, sortOrder=asc, filters={config_type=slack,email,chime,microsoft_teams,webhook,sns} retCount:1, totalCount:1
[2024-02-07T10:57:17,580][INFO ][o.o.n.r.NotificationConfigRestHandler] [myserver3] notifications:executeGetRequest from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group}
[2024-02-07T10:57:17,580][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-get org.opensearch.commons.notifications.action.GetNotificationConfigRequest@7d4d7b7b
[2024-02-07T10:57:17,581][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-getAll
[2024-02-07T10:57:17,585][INFO ][o.o.n.i.NotificationConfigIndex] [myserver3] notifications:getAllNotificationConfigs from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group} retCount:4, totalCount:4
[2024-02-07T10:57:22,801][INFO ][o.o.n.r.NotificationConfigRestHandler] [myserver3] notifications:executeGetRequest from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group}
[2024-02-07T10:57:22,802][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-get org.opensearch.commons.notifications.action.GetNotificationConfigRequest@d062388
[2024-02-07T10:57:22,802][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-getAll
[2024-02-07T10:57:22,806][INFO ][o.o.n.i.NotificationConfigIndex] [myserver3] notifications:getAllNotificationConfigs from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group} retCount:4, totalCount:4
[2024-02-07T10:57:25,232][INFO ][o.o.n.r.NotificationConfigRestHandler] [myserver3] notifications:executeGetRequest from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group}
[2024-02-07T10:57:25,232][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-get org.opensearch.commons.notifications.action.GetNotificationConfigRequest@363142fc
[2024-02-07T10:57:25,232][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-getAll
[2024-02-07T10:57:25,237][INFO ][o.o.n.i.NotificationConfigIndex] [myserver3] notifications:getAllNotificationConfigs from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group} retCount:4, totalCount:4
[2024-02-07T10:57:38,555][INFO ][o.o.n.r.NotificationConfigRestHandler] [myserver3] notifications:executeGetRequest from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group}
[2024-02-07T10:57:38,556][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-get org.opensearch.commons.notifications.action.GetNotificationConfigRequest@7fae6cc0
[2024-02-07T10:57:38,556][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-getAll
[2024-02-07T10:57:38,560][INFO ][o.o.n.i.NotificationConfigIndex] [myserver3] notifications:getAllNotificationConfigs from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group} retCount:4, totalCount:4
[2024-02-07T10:57:41,366][INFO ][o.o.n.r.NotificationConfigRestHandler] [myserver3] notifications:executeGetRequest from:0, maxItems:5, sortField:name, sortOrder=asc, filters={config_type=smtp_account}
[2024-02-07T10:57:41,367][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-get org.opensearch.commons.notifications.action.GetNotificationConfigRequest@22539e3d
[2024-02-07T10:57:41,367][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-getAll
[2024-02-07T10:57:41,371][INFO ][o.o.n.r.NotificationConfigRestHandler] [myserver3] notifications:executeGetRequest from:0, maxItems:5, sortField:name, sortOrder=asc, filters={config_type=ses_account}
[2024-02-07T10:57:41,372][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-get org.opensearch.commons.notifications.action.GetNotificationConfigRequest@5780363c
[2024-02-07T10:57:41,372][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-getAll
[2024-02-07T10:57:41,380][INFO ][o.o.n.i.NotificationConfigIndex] [myserver3] notifications:getAllNotificationConfigs from:0, maxItems:5, sortField:name, sortOrder=asc, filters={config_type=smtp_account} retCount:1, totalCount:1
[2024-02-07T10:57:41,383][INFO ][o.o.n.i.NotificationConfigIndex] [myserver3] notifications:getAllNotificationConfigs from:0, maxItems:5, sortField:name, sortOrder=asc, filters={config_type=ses_account} retCount:0, totalCount:0
[2024-02-07T10:57:45,427][INFO ][o.o.n.r.NotificationConfigRestHandler] [myserver3] notifications:executeGetRequest from:0, maxItems:5, sortField:name, sortOrder=asc, filters={config_type=email_group}
[2024-02-07T10:57:45,430][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-get org.opensearch.commons.notifications.action.GetNotificationConfigRequest@5a609047
[2024-02-07T10:57:45,430][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-getAll
[2024-02-07T10:57:45,439][INFO ][o.o.n.i.NotificationConfigIndex] [myserver3] notifications:getAllNotificationConfigs from:0, maxItems:5, sortField:name, sortOrder=asc, filters={config_type=email_group} retCount:2, totalCount:2
[2024-02-07T10:57:48,393][INFO ][o.o.n.r.NotificationConfigRestHandler] [myserver3] notifications:executeGetRequest from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group}
[2024-02-07T10:57:48,394][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-get org.opensearch.commons.notifications.action.GetNotificationConfigRequest@73b62e0e
[2024-02-07T10:57:48,394][INFO ][o.o.n.i.ConfigIndexingActions] [myserver3] notifications:NotificationConfig-getAll
[2024-02-07T10:57:48,398][INFO ][o.o.n.i.NotificationConfigIndex] [myserver3] notifications:getAllNotificationConfigs from:0, maxItems:5000, sortField:name, sortOrder=asc, filters={config_type=slack,chime,microsoft_teams,webhook,email,sns,ses_account,smtp_account,email_group} retCount:4, totalCount:4
[2024-02-07T10:57:53,505][INFO ][o.o.a.t.TransportIndexMonitorAction] [myserver3] Updating monitor, 3H0BYI0BJ92cGwouc26Z, from: {"monitor":{"type":"monitor","schema_version":8,"name":"prod Tenant - Exception or MyappException Monitor","monitor_type":"query_level_monitor","user":{"name":"admin","backend_roles":["admin"],"roles":["own_index","all_access"],"custom_attribute_names":[],"user_requested_tenant":null},"enabled":true,"enabled_time":1707292416417,"schedule":{"cron":{"expression":"*/5 * * * *","timezone":"Europe/Istanbul"}},"inputs":[{"search":{"indices":["Myapp-prod-*"],"query":{"query":{"bool":{"must":[{"range":{"@timestamp":{"from":"now-2h/h","to":"now/h","include_lower":true,"include_upper":true,"boost":1.0}}},{"script":{"script":{"source":"doc['message.keyword'].size() > 0 && (doc['message.keyword'].value.contains('MyappException') || doc['message.keyword'].value.contains('exception'))","lang":"painless"},"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}}}],"triggers":[{"query_level_trigger":{"id":"7ILdco0BJ92cGwouDd7f","name":"prod Tenant - Exception or MyappException Trigger","severity":"2","condition":{"script":{"source":"ctx.results[0].hits.total.value > 0","lang":"painless"}},"actions":[{"id":"notification633183","name":"prod Tenant - Exception or MyappException Action","destination_id":"ynSLP40BJ92cGwouWjfc","message_template":{"source":"Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n - Trigger: {{ctx.trigger.name}}\n - Severity: {{ctx.trigger.severity}}\n - Period start: {{ctx.periodStart}}\n - Period end: {{ctx.periodEnd}}","lang":"mustache"},"throttle_enabled":false,"subject_template":{"source":"OpenSearch Informer prod Indices Exception Alert","lang":"mustache"}}]}}],"last_update_time":1707292416417,"ui_metadata":{"schedule":{"cronExpression":"*/5 * * * *","period":{"unit":"MINUTES","interval":1},"timezone":"Europe/Istanbul","daily":0,"monthly":{"type":"day","day":1},"weekly":{"tue":false,"wed":false,"thur":false,"sat":false,"fri":false,"mon":false,"sun":false},"frequency":"cronExpression"},"search":{"searchType":"query","bucketValue":1,"timeField":"","bucketUnitOfTime":"h","filters":[],"aggregations":[],"cleanedGroupBy":[]},"triggers":{"prod Tenant - Exception or MyappException Trigger":{"value":10000,"enum":"ABOVE"}},"monitor_type":"query_level_monitor"},"data_sources":{"query_index":".opensearch-alerting-queries","findings_index":".opensearch-alerting-finding-history-write","findings_index_pattern":"<.opensearch-alerting-finding-history-{now/d}-1>","alerts_index":".opendistro-alerting-alerts","alerts_history_index":".opendistro-alerting-alert-history-write","alerts_history_index_pattern":"<.opendistro-alerting-alert-history-{now/d}-1>","query_index_mappings_by_type":{},"findings_enabled":false},"owner":"alerting"}}
to: {"monitor":{"type":"monitor","schema_version":8,"name":"prod Tenant - Exception or MyappException Monitor","monitor_type":"query_level_monitor","user":{"name":"admin","backend_roles":["admin"],"roles":["own_index","all_access"],"custom_attribute_names":[],"user_requested_tenant":null},"enabled":true,"enabled_time":1707292416417,"schedule":{"cron":{"expression":"*/5 * * * *","timezone":"Europe/Istanbul"}},"inputs":[{"search":{"indices":["Myapp-prod-*"],"query":{"query":{"bool":{"must":[{"range":{"@timestamp":{"from":"now-2h/h","to":"now/h","include_lower":true,"include_upper":true,"boost":1.0}}},{"script":{"script":{"source":"doc['message.keyword'].size() > 0 && (doc['message.keyword'].value.contains('MyappException') || doc['message.keyword'].value.contains('exception'))","lang":"painless"},"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}}}],"triggers":[{"query_level_trigger":{"id":"7ILdco0BJ92cGwouDd7f","name":"prod Tenant - Exception or MyappException Trigger","severity":"2","condition":{"script":{"source":"ctx.results[0].hits.total.value > 0","lang":"painless"}},"actions":[{"id":"notification633183","name":"prod Tenant - Exception or MyappException Action","destination_id":"5nSLP40BJ92cGwou5DcJ","message_template":{"source":"Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n - Trigger: {{ctx.trigger.name}}\n - Severity: {{ctx.trigger.severity}}\n - Period start: {{ctx.periodStart}}\n - Period end: {{ctx.periodEnd}}","lang":"mustache"},"throttle_enabled":false,"subject_template":{"source":"OpenSearch Informer prod Indices Exception Alert","lang":"mustache"}}]}}],"last_update_time":1707292673313,"ui_metadata":{"schedule":{"cronExpression":"*/5 * * * *","period":{"unit":"MINUTES","interval":1},"timezone":"Europe/Istanbul","daily":0,"monthly":{"type":"day","day":1},"weekly":{"tue":false,"wed":false,"thur":false,"sat":false,"fri":false,"mon":false,"sun":false},"frequency":"cronExpression"},"search":{"searchType":"query","bucketValue":1,"timeField":"","bucketUnitOfTime":"h","filters":[],"aggregations":[],"cleanedGroupBy":[]},"triggers":{"prod Tenant - Exception or MyappException Trigger":{"value":10000,"enum":"ABOVE"}},"monitor_type":"query_level_monitor"},"data_sources":{"query_index":".opensearch-alerting-queries","findings_index":".opensearch-alerting-finding-history-write","findings_index_pattern":"<.opensearch-alerting-finding-history-{now/d}-1>","alerts_index":".opendistro-alerting-alerts","alerts_history_index":".opendistro-alerting-alert-history-write","alerts_history_index_pattern":"<.opendistro-alerting-alert-history-{now/d}-1>","query_index_mappings_by_type":{},"findings_enabled":false},"owner":"alerting"}}
[2024-02-07T10:57:54,683][INFO ][o.o.a.r.RestSearchMonitorAction] [myserver3] The monitor parsing failed. Will return response as is.
[2024-02-07T10:57:54,713][INFO ][o.o.a.r.RestSearchMonitorAction] [myserver3] The monitor parsing failed. Will return response as is.
[2024-02-07T10:58:16,158][INFO ][o.o.a.r.RestSearchMonitorAction] [myserver3] The monitor parsing failed. Will return response as is.
[2024-02-07T10:59:09,078][INFO ][o.o.a.r.RestSearchMonitorAction] [myserver3] The monitor parsing failed. Will return response as is.
[2024-02-07T11:00:55,897][INFO ][o.o.a.r.RestSearchMonitorAction] [myserver3] The monitor parsing failed. Will return response as is.
[2024-02-07T11:01:04,794][INFO ][o.o.a.r.RestSearchMonitorAction] [myserver3] The monitor parsing failed. Will return response as is.
[2024-02-07T11:01:12,598][INFO ][o.o.a.t.TransportIndexMonitorAction] [myserver3] Updating monitor, 3H0BYI0BJ92cGwouc26Z, from: {"monitor":{"type":"monitor","schema_version":8,"name":"prod Tenant - Exception or MyappException Monitor","monitor_type":"query_level_monitor","user":{"name":"admin","backend_roles":["admin"],"roles":["own_index","all_access"],"custom_attribute_names":[],"user_requested_tenant":null},"enabled":true,"enabled_time":1707292416417,"schedule":{"cron":{"expression":"*/5 * * * *","timezone":"Europe/Istanbul"}},"inputs":[{"search":{"indices":["Myapp-prod-*"],"query":{"query":{"bool":{"must":[{"range":{"@timestamp":{"from":"now-2h/h","to":"now/h","include_lower":true,"include_upper":true,"boost":1.0}}},{"script":{"script":{"source":"doc['message.keyword'].size() > 0 && (doc['message.keyword'].value.contains('MyappException') || doc['message.keyword'].value.contains('exception'))","lang":"painless"},"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}}}],"triggers":[{"query_level_trigger":{"id":"7ILdco0BJ92cGwouDd7f","name":"prod Tenant - Exception or MyappException Trigger","severity":"2","condition":{"script":{"source":"ctx.results[0].hits.total.value > 0","lang":"painless"}},"actions":[{"id":"notification633183","name":"prod Tenant - Exception or MyappException Action","destination_id":"5nSLP40BJ92cGwou5DcJ","message_template":{"source":"Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n - Trigger: {{ctx.trigger.name}}\n - Severity: {{ctx.trigger.severity}}\n - Period start: {{ctx.periodStart}}\n - Period end: {{ctx.periodEnd}}","lang":"mustache"},"throttle_enabled":false,"subject_template":{"source":"OpenSearch Informer prod Indices Exception Alert","lang":"mustache"}}]}}],"last_update_time":1707292673313,"ui_metadata":{"schedule":{"cronExpression":"*/5 * * * *","period":{"unit":"MINUTES","interval":1},"timezone":"Europe/Istanbul","daily":0,"monthly":{"type":"day","day":1},"weekly":{"tue":false,"wed":false,"thur":false,"sat":false,"fri":false,"mon":false,"sun":false},"frequency":"cronExpression"},"search":{"searchType":"query","bucketValue":1,"timeField":"","bucketUnitOfTime":"h","filters":[],"aggregations":[],"cleanedGroupBy":[]},"triggers":{"prod Tenant - Exception or MyappException Trigger":{"value":10000,"enum":"ABOVE"}},"monitor_type":"query_level_monitor"},"data_sources":{"query_index":".opensearch-alerting-queries","findings_index":".opensearch-alerting-finding-history-write","findings_index_pattern":"<.opensearch-alerting-finding-history-{now/d}-1>","alerts_index":".opendistro-alerting-alerts","alerts_history_index":".opendistro-alerting-alert-history-write","alerts_history_index_pattern":"<.opendistro-alerting-alert-history-{now/d}-1>","query_index_mappings_by_type":{},"findings_enabled":false},"owner":"alerting"}}
to: {"monitor":{"type":"monitor","schema_version":8,"name":"prod Tenant - Exception or MyappException Monitor","monitor_type":"query_level_monitor","user":{"name":"admin","backend_roles":["admin"],"roles":["own_index","all_access"],"custom_attribute_names":[],"user_requested_tenant":null},"enabled":false,"enabled_time":null,"schedule":{"cron":{"expression":"*/5 * * * *","timezone":"Europe/Istanbul"}},"inputs":[{"search":{"indices":["Myapp-prod-*"],"query":{"query":{"bool":{"must":[{"range":{"@timestamp":{"from":"now-2h/h","to":"now/h","include_lower":true,"include_upper":true,"boost":1.0}}},{"script":{"script":{"source":"doc['message.keyword'].size() > 0 && (doc['message.keyword'].value.contains('MyappException') || doc['message.keyword'].value.contains('exception'))","lang":"painless"},"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}}}],"triggers":[{"query_level_trigger":{"id":"7ILdco0BJ92cGwouDd7f","name":"prod Tenant - Exception or MyappException Trigger","severity":"2","condition":{"script":{"source":"ctx.results[0].hits.total.value > 0","lang":"painless"}},"actions":[{"id":"notification633183","name":"prod Tenant - Exception or MyappException Action","destination_id":"5nSLP40BJ92cGwou5DcJ","message_template":{"source":"Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n - Trigger: {{ctx.trigger.name}}\n - Severity: {{ctx.trigger.severity}}\n - Period start: {{ctx.periodStart}}\n - Period end: {{ctx.periodEnd}}","lang":"mustache"},"throttle_enabled":false,"subject_template":{"source":"OpenSearch Informer prod Indices Exception Alert","lang":"mustache"}}]}}],"last_update_time":1707292871917,"ui_metadata":{"schedule":{"cronExpression":"*/5 * * * *","period":{"unit":"MINUTES","interval":1},"timezone":"Europe/Istanbul","daily":0,"monthly":{"type":"day","day":1},"weekly":{"tue":false,"wed":false,"thur":false,"sat":false,"fri":false,"mon":false,"sun":false},"frequency":"cronExpression"},"search":{"searchType":"query","bucketValue":1,"timeField":"","bucketUnitOfTime":"h","filters":[],"aggregations":[],"cleanedGroupBy":[]},"triggers":{"prod Tenant - Exception or MyappException Trigger":{"value":10000,"enum":"ABOVE"}},"monitor_type":"query_level_monitor"},"data_sources":{"query_index":".opensearch-alerting-queries","findings_index":".opensearch-alerting-finding-history-write","findings_index_pattern":"<.opensearch-alerting-finding-history-{now/d}-1>","alerts_index":".opendistro-alerting-alerts","alerts_history_index":".opendistro-alerting-alert-history-write","alerts_history_index_pattern":"<.opendistro-alerting-alert-history-{now/d}-1>","query_index_mappings_by_type":{},"findings_enabled":false},"owner":"alerting"}}
[2024-02-07T11:01:19,439][INFO ][o.o.j.s.JobSweeper ] [myserver3] Running full sweep
[2024-02-07T11:03:21,801][INFO ][o.o.a.r.RestSearchMonitorAction] [myserver3] The monitor parsing failed. Will return response as is.
[2024-02-07T11:03:30,732][INFO ][o.o.a.t.TransportIndexMonitorAction] [myserver3] Updating monitor, 3H0BYI0BJ92cGwouc26Z, from: {"monitor":{"type":"monitor","schema_version":8,"name":"prod Tenant - Exception or MyappException Monitor","monitor_type":"query_level_monitor","user":{"name":"admin","backend_roles":["admin"],"roles":["own_index","all_access"],"custom_attribute_names":[],"user_requested_tenant":null},"enabled":false,"enabled_time":null,"schedule":{"cron":{"expression":"*/5 * * * *","timezone":"Europe/Istanbul"}},"inputs":[{"search":{"indices":["Myapp-prod-*"],"query":{"query":{"bool":{"must":[{"range":{"@timestamp":{"from":"now-2h/h","to":"now/h","include_lower":true,"include_upper":true,"boost":1.0}}},{"script":{"script":{"source":"doc['message.keyword'].size() > 0 && (doc['message.keyword'].value.contains('MyappException') || doc['message.keyword'].value.contains('exception'))","lang":"painless"},"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}}}],"triggers":[{"query_level_trigger":{"id":"7ILdco0BJ92cGwouDd7f","name":"prod Tenant - Exception or MyappException Trigger","severity":"2","condition":{"script":{"source":"ctx.results[0].hits.total.value > 0","lang":"painless"}},"actions":[{"id":"notification633183","name":"prod Tenant - Exception or MyappException Action","destination_id":"5nSLP40BJ92cGwou5DcJ","message_template":{"source":"Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n - Trigger: {{ctx.trigger.name}}\n - Severity: {{ctx.trigger.severity}}\n - Period start: {{ctx.periodStart}}\n - Period end: {{ctx.periodEnd}}","lang":"mustache"},"throttle_enabled":false,"subject_template":{"source":"OpenSearch Informer prod Indices Exception Alert","lang":"mustache"}}]}}],"last_update_time":1707292871917,"ui_metadata":{"schedule":{"cronExpression":"*/5 * * * *","period":{"unit":"MINUTES","interval":1},"timezone":"Europe/Istanbul","daily":0,"monthly":{"type":"day","day":1},"weekly":{"tue":false,"wed":false,"thur":false,"sat":false,"fri":false,"mon":false,"sun":false},"frequency":"cronExpression"},"search":{"searchType":"query","bucketValue":1,"timeField":"","bucketUnitOfTime":"h","filters":[],"aggregations":[],"cleanedGroupBy":[]},"triggers":{"prod Tenant - Exception or MyappException Trigger":{"value":10000,"enum":"ABOVE"}},"monitor_type":"query_level_monitor"},"data_sources":{"query_index":".opensearch-alerting-queries","findings_index":".opensearch-alerting-finding-history-write","findings_index_pattern":"<.opensearch-alerting-finding-history-{now/d}-1>","alerts_index":".opendistro-alerting-alerts","alerts_history_index":".opendistro-alerting-alert-history-write","alerts_history_index_pattern":"<.opendistro-alerting-alert-history-{now/d}-1>","query_index_mappings_by_type":{},"findings_enabled":false},"owner":"alerting"}}
to: {"monitor":{"type":"monitor","schema_version":8,"name":"prod Tenant - Exception or MyappException Monitor","monitor_type":"query_level_monitor","user":{"name":"admin","backend_roles":["admin"],"roles":["own_index","all_access"],"custom_attribute_names":[],"user_requested_tenant":null},"enabled":true,"enabled_time":1707293009738,"schedule":{"cron":{"expression":"*/5 * * * *","timezone":"Europe/Istanbul"}},"inputs":[{"search":{"indices":["Myapp-prod-*"],"query":{"query":{"bool":{"must":[{"range":{"@timestamp":{"from":"now-2h/h","to":"now/h","include_lower":true,"include_upper":true,"boost":1.0}}},{"script":{"script":{"source":"doc['message.keyword'].size() > 0 && (doc['message.keyword'].value.contains('MyappException') || doc['message.keyword'].value.contains('exception'))","lang":"painless"},"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}}}],"triggers":[{"query_level_trigger":{"id":"7ILdco0BJ92cGwouDd7f","name":"prod Tenant - Exception or MyappException Trigger","severity":"2","condition":{"script":{"source":"ctx.results[0].hits.total.value > 0","lang":"painless"}},"actions":[{"id":"notification633183","name":"prod Tenant - Exception or MyappException Action","destination_id":"5nSLP40BJ92cGwou5DcJ","message_template":{"source":"Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n - Trigger: {{ctx.trigger.name}}\n - Severity: {{ctx.trigger.severity}}\n - Period start: {{ctx.periodStart}}\n - Period end: {{ctx.periodEnd}}","lang":"mustache"},"throttle_enabled":false,"subject_template":{"source":"OpenSearch Informer prod Indices Exception Alert","lang":"mustache"}}]}}],"last_update_time":1707293009738,"ui_metadata":{"schedule":{"cronExpression":"*/5 * * * *","period":{"unit":"MINUTES","interval":1},"timezone":"Europe/Istanbul","daily":0,"monthly":{"type":"day","day":1},"weekly":{"tue":false,"wed":false,"thur":false,"sat":false,"fri":false,"mon":false,"sun":false},"frequency":"cronExpression"},"search":{"searchType":"query","bucketValue":1,"timeField":"","bucketUnitOfTime":"h","filters":[],"aggregations":[],"cleanedGroupBy":[]},"triggers":{"prod Tenant - Exception or MyappException Trigger":{"value":10000,"enum":"ABOVE"}},"monitor_type":"query_level_monitor"},"data_sources":{"query_index":".opensearch-alerting-queries","findings_index":".opensearch-alerting-finding-history-write","findings_index_pattern":"<.opensearch-alerting-finding-history-{now/d}-1>","alerts_index":".opendistro-alerting-alerts","alerts_history_index":".opendistro-alerting-alert-history-write","alerts_history_index_pattern":"<.opendistro-alerting-alert-history-{now/d}-1>","query_index_mappings_by_type":{},"findings_enabled":false},"owner":"alerting"}}
[2024-02-07T11:06:19,440][INFO ][o.o.j.s.JobSweeper ] [myserver3] Running full sweep
[2024-02-07T11:06:58,203][INFO ][o.o.a.r.RestSearchMonitorAction] [myserver3] The monitor parsing failed. Will return response as is.
[2024-02-07T11:11:19,440][INFO ][o.o.j.s.JobSweeper ] [myserver3] Running full sweep
