Mux Data Missing Some EU Views

Resolved

Major impact

Started

Lasted 9h 18m

View official incident ↗

Updates

  1. Resolved

    Backfill complete and this incident has been resolved. Any missing views older than 7 days are unrecoverable. Time of impact for this incident is June 17th-November 5th for EU collection customers. We estimate 1/3 of views during this window are missing for affected customers, with more views affected in later months. Real-time metrics and alerting remain unaffected. We apologize for this data loss and are taking corrective action on our systems to prevent recurrence.

  2. Identified

    We are currently re-processing the last 7 days of available views that were missed. We expect this step to take several hours.

  3. Identified

    We are collecting and storing all views from the EU as of 22:16 UTC. We are evaluating if any previous missing views are recoverable.

  4. Identified

    The issue has been identified and a fix is being implemented.

  5. Investigating

    We are seeing roughly half of all collected views from the EU are missing, and have been missing for several months. Realtime metrics and alerting are unaffected. We are continuing to investigate.

  6. Investigating

    We are investigating a missing views issue affecting customers collection in the EU.