Extra Notifications

Resolved

None impact

Started

Lasted 2h 19m

View official incident ↗

Updates

  1. Resolved

    All systems have returned to normal.

  2. Monitoring

    The email notification backlog was overwhelmed by the extraneous notifications. As a result, we have cleared out the backlog of notifications that were queued up between 16:30 and 16:52 UTC to resume sending email notifications without a delay. Non-email notifications were not affected by this action.

  3. Monitoring

    The breaking change has been rolled back. A backlog of email notifications is clearing out so notifications will continue to be sent for a short time. The backlog is causing a short delay for new notifications to be sent. We are monitoring the system closely for any other issues.

  4. Identified

    We've identified an issue causing notifications to be sent on all test runs regardless of settings. We're rolling back the change causing the issue. The behavior of test steps are not impacted by this incident, only notifications are affected.