Machine Creates Failing in ORD Region

Resolved

Started

Lasted 2h 33m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Monitoring

    We've implemented a fix and have seen error rates for machine creates in ORD drop off. We're continuing to monitor the results.

  3. Identified

    We've identified the cause of this increased failure rate and a fix is in progress. We are seeing most creates in ORD succeed at this time, though failure rate is still above baseline.

  4. Investigating

    We are continuing to investigate this issue. We are seeing 408 errors decreasing in ORD, though still above baseline.

  5. Investigating

    We are currently investigating elevated errors creating machines in the ORD (Chicago, Illinois) region. Users may see failed to launch VM: request returned non-2xx status: 408 errors when creating, updating, or scaling machines in ORD. Existing, already running machines in the ORD region continue to run as normal.