Auto resolve for Pagerduty not always auto resolving - how to debug?

[phlegx]

Hi there!

I created extra rules that provide me with auto resolve capability. It works actually pretty good, but sometimes the auto resolve - for example for checking if an HTTP endpoint is again available - is not working. Then of course the Alert remains unresolved in Pagerduty until I resolve it manually.

Any ideas on how to debug this so I can fix this? Is there another way of implementing auto resolve with Elastalert2?

Here is my auto-resolve rule for HTTP endpoints (This is a template for Terraform):

name: "[${project_name}]][${environment}] HTTP ${name} ${severity} resolved"
description: Monitors HTTP status ${environment} environment
type: frequency
index: ${project_shortcut}-heartbeat-${environment}*
num_events: 1
timeframe:
  minutes: 1

- query:
    query_string:
      query: "monitor.status: up AND url.full: \"${full_url}\" AND monitor.type: http"


alert_subject: "[${project_name}]][${environment}] HTTP ${name} - {0} resolved"
alert_subject_args:
- url.full
- host.name

realert:
  minutes: 1
alert:
  - pagerduty

pagerduty_service_key: "${pagerduty_integration_key}"
pagerduty_client_name: "Elastalert"
pagerduty_event_type: "resolve"

pagerduty_api_version: "v2"
pagerduty_v2_payload_severity: "${severity}"
pagerduty_v2_payload_source: host.name
pagerduty_incident_key: "${project_shortcut}-${environment}-http-${full_url}-${severity}"

thanks in advance!

[jertel]

This documentation describes a way to receive resolve alerts: https://elastalert2.readthedocs.io/en/latest/recipes/faq.html#how-can-i-get-a-resolve-event

However, in your situation, since your method does work sometimes, I suggest continuing down the debug path because changing to the documented method could result in the same behavior: works sometimes, but not always. What you might find is that ElastAlert2 is detecting the resolve event correctly already, and something else, such as a network glitch, PagerDuty glitch, etc is preventing the alert from getting to PagerDuty.

Also, if you're missing an occassional resolve trigger, consider that you might be missing outage triggers too, due to the same underlying problem.

Next steps would be to enable verbose logging for ElastAlert2 and when the problem occurs, inspect the logs to track down whether ElastAlert2 detected the resolve event or not. The verbose logs can be overwhelming so be prepared to sit down and sort through the logs for a while.

To enable verbose logging: https://elastalert2.readthedocs.io/en/latest/elastalert.html?highlight=verbose#logging

If you need even more logging you can adjust the log levels in the config.yaml. See the example config (near the bottom).