Delays in Drains, Observability, and Workflow in Washington (iad1)

Resolved

Major impact

Started

Lasted 1h 21m

View official incident ↗

Updates

  1. Resolved

    All services have recovered.

  2. Monitoring

    Delayed Drains deliveries, Observability data, and Build logs have been processed now. We are continuing to monitor and we will provide additional updates as they become available.

  3. Identified

    A fix has been implemented and delivery has resumed. Delayed Drains deliveries and Observability data are being processed. We are continuing to monitor.

  4. Investigating

    We've identified an issue where some customers may experience delayed drain delivery, delayed logs, metrics, and traces, and Workflow runs that remain pending in Washington (iad1). We are investigating this issue. We will provide additional updates as they become available.