Degraded querying and delayed records sent to Activity Log

Resolved

Minor impact

Started

Lasted 21h 40m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Identified

    We have stabilized activity log. We have confirmed activity log entries lost from 4:54pm-7:04pm ET in the US and 9:43pm-11:55pm GMT in the EU. Event ingest was unimpacted. All systems are currently operating normally

  3. Identified

    We have restored basic activity log functionality, though some delays remain, and we will analyze further tomorrow during business hours.

  4. Identified

    We are currently expecting to lose the last three hours of data for the beta activity log feature as part of remediation of this issue.

  5. Identified

    We have resolved the issue which impacted querying but are still working to get Activity Log up to date

  6. Identified

    We are seeing a spike in querying errors and the Activity Log has delayed ingest of new activity. We have identified the issue and are working to resolve