Elevated connection errors in us-east-1

Resolved

Minor impact

Started

Lasted 14h 21m

View official incident ↗

Updates

  1. Resolved

    us-east-1 is reenabled and the service is operating normally. The root cause has been confirmed to be a defective Network Load Balancer that was causing a small fraction of connection attempts in the region to fail. We are awaiting a postmortem from AWS on the issue, and will publish our own postmortem here once we have that.

  2. Monitoring

    us-east-1 remains disabled. All traffic is being routed to other regions so there is no ongoing impact on service.

  3. Monitoring

    Traffic is no longer being routed to us-east-1, but instead to us-east-2 and us-west-1. We are continuing to investigate the root cause of the networking issue in us-east-1.

  4. Investigating

    We are seeing an elevated rate of connection errors in the us-east-1 region, seemingly as a result of a partial loadbalancer failure. We are disabling the region so traffic will be routed to other US sites.