Updates
Resolved
The fix has been fully deployed, and alerts on insights are back to normal.
Monitoring
All but a handful of alerts are caught up. We're working to get those alerts processed now and continuing to monitor. For almost all customers there should be no impact at this time.
Monitoring
The fix to unblock the backlog is now deployed and in-progress. We're continuing to monitor the issue.
Identified
We’ve identified that the EU insight alert scheduler is getting overloaded by an unbounded batch of due alerts, which can cause scheduler runs to time out and prevent new alert checks from being scheduled. We have a mitigation ready in PR #97378. This is intended to let the EU cluster drain the backlog gradually once deployed.
Investigating
Alerts on insights are not being sent, the team is investigating