Delays in event ingestion

Resolved

Minor impact

Started

Lasted 97h 43m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Monitoring

    We have made major progress. We identified that a small percentage of event data was lost during two windows: June 9, 11:41 - 11:59 UTC and June 10, 23:55 UTC - June 11, 00:21 UTC. Some event-based features may show incomplete data while backfills and recovery continue. We are continuing to monitor and will provide another update when recovery is complete.

  3. Monitoring

    We have made major progress. We identified that a small percentage of event data was lost during two windows: June 9, 11:41 - 11:59 UTC and June 10, 23:55 UTC - June 11, 00:21 UTC. Some event-based features may show incomplete data while backfills and recovery continue. We are continuing to monitor and will provide another update when recovery is complete.

  4. Monitoring

    We have made partial progress. We identified that a small percentage of event data was lost during two windows: June 9, 11:41 - 11:59 UTC and June 10, 23:55 UTC - June 11, 00:21 UTC. Some event-based features may show incomplete data while backfills and recovery continue. We are continuing to monitor and will provide another update when recovery is complete.

  5. Monitoring

    We have made partial progress. We have identified a brief period of event data loss, and some event-based features may show incomplete data while we continue recovery. We are continuing to monitor.

  6. Monitoring

    We have made partial progress. We have identified a brief period of event data loss, and some event-based features may show incomplete data while we continue recovery. We are continuing to monitor.

  7. Monitoring

    We are continuing to monitor for any further issues.

  8. Monitoring

    We have made partial progress. We have identified a brief period of event data loss, and some event-based features may show incomplete data while we continue recovery. We are continuing to monitor.

  9. Monitoring

    We have made partial progress. Some features may show data gaps while we complete recovery. We are continuing to monitor.

  10. Monitoring

    We have made partial progress. Some features may show data gaps while we complete recovery. We are continuing to monitor.

  11. Monitoring

    We are continuing to monitor for any further issues.

  12. Monitoring

    We are monitoring the status of our services.

  13. Monitoring

    A fix has been implemented and we are monitoring the results.

  14. Identified

    We are currently experiencing delays in event ingestion impacting multiple product areas starting at 12:40pm PT. We have identified the issue and are working on a fix. No data has been lost.