Slow workflow processing and message delivery

Resolved

Started

Lasted 2h 59m

View official incident ↗

Updates

  1. Resolved

    All behavior remains normal, we are considering this incident resolved.

  2. Identified

    We are seeing a recovery in all workflow and broadcast processing delays. We are monitoring closely.

  3. Identified

    We've taken steps to shuffle around work in our system which has meant that we're processing new work for all customers again. We still have some queued workflow executions that we're working through and expect to be fully caught up in the next 40 minutes.

  4. Identified

    One of our workflow queues is currently still backlogged, but we're seeing good resolution across other areas of our service and we have effectively cordoned off the source of the issue. We're burning through remaining work and are exploring ways to shed the load further to get back to stability.

  5. Investigating

    We're seeing some recovery on our services but are dealing with delays in some workflow and broadcast processing still. Our logging service is also currently delayed, so may not reflect the latest as we catch up and work through a huge influx of work.

  6. Investigating

    We're seeing an extremely high amount of load on our services which is causing workflow and broadcast delays. We're working to relieve the pressure immediately.

  7. Investigating

    We're currently investigating an issue related to slow notification processing and message delivery. Some workflows will be delayed