Elevated latencies and delays

Resolved

Major impact

Started

Lasted 25h 48m

View official incident ↗

Updates

  1. Resolved

    The issue with elevated latencies and delays has been resolved. Impacted services have returned to normal operation.

  2. Monitoring

    Systems have recovered to healthy levels and we're continuing to monitor the system's metrics.

  3. Identified

    We are still experiencing delays in flag updates and event ingestion pipeline, affecting experimentation, data export, flag status metrics and others.

  4. Identified

    We are still experiencing delays in flag updates and event ingestion pipeline, affecting experimentation, data export, flag status metrics and others.

  5. Identified

    We are seeing initial recovery for the following services - Flag updates We are monitoring for the recovery of the rest of the services.

  6. Identified

    We are continuing to work on the issue. Additionally impacted services - Delayed flag updates to SDKs - Dropped SDK events impacting Experimentation, Data Export

  7. Identified

    We have identified issue with elevated error rates and event pipelines. Currently impacted services are - Guarded rollouts - Scheduled flag changes - Experimentation - Data export - Flag usage metrics - Emails and notifications - Integrations web hooks