Elevated Machines errors in ORD

Resolved

Started

Lasted 7h 42m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved. Machine placement improvements are now deployed in all regions.

  2. Monitoring

    The fix has been deployed to ORD and we are monitoring results. New Machines will be placed using stricter memory thresholds. Existing Machines on impacted hosts will be migrated to new hosts on start and update. We are also provisioning additional capacity in ORD.

  3. Identified

    We are deploying a fix to enforce stricter memory thresholds for Machine placements. This will steer new and migrated workloads towards hosts with optimal capacity.

  4. Investigating

    We are investigating high memory utilization on some hosts in ORD. Customers may notice timeouts or errors in reserving resources when updating or creating new Machines in the region. We are rebalancing workloads in ORD to relieve memory pressure on impacted hosts.