activity log for US instance is delayed

Resolved

Minor impact

Started

Lasted 80h 38m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Monitoring

    Activity Log recovered at 07:00 UTC-07:00 today (about 6.5h ago).  All streams are caught up.

  3. Monitoring

    All activity log streams are now caught up, except for the query runs table, which should have all data since the start of the incident backfilled by 1600 PDT.

  4. Monitoring

    We are continuing to monitor for any further issues.

  5. Monitoring

    We believe that the activity log should now be caught up.

  6. Monitoring

    Backfilling of the past 2h30m of data is now in progress and should complete shortly.

  7. Identified

    Activity log delay is rising, and is more than an hour behind due to a MySQL replica failover. Estimated recovery by 04:00 PDT.