Error ingestion degraded in US region

Resolved

Critical impact

Started

Lasted 4h 51m

View official incident ↗

Updates

  1. Resolved

    We have processed the backlog.

  2. Monitoring

    We have processed the majority of our backlog. We are now recovering the tail of the backlog.

  3. Identified

    We have processed the majority of our backlog. We are now recovering an additional 25M error events ranging from 22:20 to 23:20 UTC, which should complete in the next 2 hours

  4. Identified

    We expect recovery of historical data to be complete within the next 2 hours.

  5. Identified

    We are still working on recovery. New errors should be processed in real time, and we are working on recovering historical data.

  6. Identified

    We have identified the problem and are working on recovery. New errors are being processed in real time and we are in the process of consuming backlogged errors.

  7. Identified

    We have identified the problem and are working on a fix, ingested errors are delayed and will be processed once the fix is in place.

  8. Investigating

    We are currently investigating this issue.