Alert repeatedly sends "back to normal" message despite never triggering

Hello all :).

I’ve got a scheduled query that runs every minute and produces a single scalar value. I’ve created an alert based on that query with the following settings:

  • trigger said value is < 65000
  • notifications are sent “just once, until back to normal”
  • slack channel set up as alert destination
  • custom description that includes the {{ALERT_STATUS}} and {{QUERY_RESULT_VALUE}} macros (for debugging purposes).

I’m seeing two kinds of strange behavior:

  • Often (but unfortunately for our debugging purposes, not always) the slack channel will receive a message every minute saying “{{ALERT_NAME}} went back to normal}}”, coupled with an “OK” status and a value that is clearly above the 65000 alert threshold. These messages even when the alert was not recently in a “TRIGGERED” state. Put another way, the slack channel multiple “back to normal” messages for a single “triggered” message, if a “triggered” message appears at all.
  • This behavior persists if I tell the alert to send notifications “at most every 5 minutes”. The only difference is that the green notifications are sent every 5 minutes instead of every minute when the underlying query refreshes.

Other troubleshooting steps I’ve tried:

  • I’ve debugged the underlying query to the point that I’m pretty confident it’s not returning values below the threshold for brief periods of time. The query computes rolling average anyways, so that kind of behavior shouldn’t happy given what I know about the underlying timeseries it’s pulling from. Plus, wouldn’t I see a red “triggered” message in slack if the alert entered the “triggered” state?
  • I’ve tried creating a new query and alert with the same query text, refresh rate, and alert settings and the same problem exists.

Questions:

  • Any other troubleshooting steps I should try?
  • Any other ways I could configure this alert to stop this behavior?

Relevant platform information:

  • self-hosted redash
  • redash/preview:9.0.0-beta.b49483 image

For clarification, this setting “at most every 5 minutes” only affects the frequency of TRIGGERED messages. You will always receive a notification when the alert is first triggered and you will always receive one notification when the value goes back to normal. You shouldn’t see multiple back-to-normal notifications, though, so this does look like a bug.

Does this reproduce using the V10 beta?