Resolved incident

EU Region Error, Attachment, and Feedback latency

Sentry · Critical · Started

Duration
5h 46m
Severity
Critical
Status
Resolved

Summary

Critical · Resolved

We have fully burned the backlog of EU error events ingested during the incident. Latency is normal across all event types.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    Sentry

    we are investigating an issue affecting errors ingestion in the DE/EU region

  2. Identified

    Sentry

    We have identified the issue causing errors, attachments, and feedback events to have high ingest latency in our EU region and are working on a fix.

  3. Identified

    Sentry

    We have identified the issue causing errors, attachments, and feedback events to have high ingest latency in our EU region and are continuing to work on a fix.

  4. Identified

    Sentry

    We have identified the issue causing errors, attachments, and feedback events to have high ingest latency in our EU region and are continuing to work on a fix.

  5. Identified

    Sentry

    We have identified the issue causing errors, attachments, and feedback events to have high ingest latency in our EU region and are continuing to work on a fix.

  6. Identified

    Sentry

    We have implemented a fix for errors and feedback ingestion in our EU region, and latency for those pipelines has returned to normal. We are still working on fixing our attachments ingestion pipeline in EU.

  7. Monitoring

    Sentry

    We have implemented a fix for our attachment ingestion issue in our EU region, and are monitoring while the backlog burns. We will update here when latency is back to normal.

  8. Monitoring

    Sentry

    The backlog on attachments in our EU region is fully burned, and latency is back to normal. While new error events are being processed with normal latency, we have a backlog of old error events that we are now burning through.

  9. Monitoring

    Sentry

    We are still burning the backlog of EU region error events received during the incident. We are around halfway through the backlog.

  10. Resolved

    Sentry

    We have fully burned the backlog of EU error events ingested during the incident. Latency is normal across all event types.