Updates
Resolved
This incident has been resolved.
Monitoring
We're continuing to monitor after having added more capacity to our DFW and IAD regions. Deploys or machine starts using existing volumes in these regions may still hit a capacity issue. Users should use fly volume fork --vm-memory <size> to fork the volume to a host with more capacity, then retry the deploy or start command using the new volume.
Identified
We have added additional capacity in DFW and IAD regions and are monitoring the impact. New machine creates and deploys without volumes are seeing improved success rates. Deploys using depot builders in those regions are also improving, with much quicker builder start times. Deploys or machine starts using existing volumes in these regions may still hit a capacity issue. Users should use fly volume fork --vm-memory <size> to fork the volume to a host with more capacity, then retry the deploy or start command using the new volume.
Identified
We've identified some newly created Managed Postgres clusters are failing to come up healthy in these regions.
Identified
New machine creates in these regions might fail temporarily, and Depot builders may be unavailable. If you are having issues with Depot builders, consider moving them to a different region, or try --depot=false.
Identified
We have identified the problem and are working on a fix.