Delayed Ingestion for Some Customers

Resolved

Critical impact

Started

Lasted 26h 22m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Monitoring

    We've isolated the issue to a single account. We're in contact with that customer and have restarted ingestion for all other accounts.

  3. Investigating

    We've identified a handful of incoming messages that have slowed our ingestion processing, causing it to fall behind. This has caused tripped circuit breakers in other parts of our app. All data is stored, but ingestion as a whole is paused

  4. Investigating

    One of our ingestion servers is falling a little behind, so ingestion for any customers on that server will be delayed. All data is safe and is being processed.