Delayed task execution on some apps

Resolved

Started

Lasted 7h 18m

View official incident ↗

Updates

  1. Resolved

    Tasks have returned to normal execution, and error rates are at normal levels.

  2. Monitoring

    The configuration has been fixed and the replication errors have ceased. We are monitoring to make sure that everything is stable.

  3. Identified

    We have identified the problematic configuration and are waiting for AWS to perform manual database cleanup to allow us to proceed.

  4. Identified

    We’ve identified the problem and the fix is being rolled out. We still have a % of our apps on the main cluster that are not running scheduled tasks or able to delete records, while the rollout is ongoing.

  5. Investigating

    We are investigating system behavior that may result in delayed task execution