Scheduled provider maintenance: delayed GPU node provisioning in US East (Crusoe) and US South Central (Crusoe

Event timeline

How this incident unfolded, from the official status page.

1. Unknown

   about 2 months ago

   Cerebrium

   Maintenance ended

2. Unknown

   about 2 months ago

   Cerebrium

   Our infrastructure provider Crusoe is performing maintenance on the Kubernetes control plane for the clusters behind our US East and US South Central Crusoe regions. What to expect: - Running apps and live traffic are not affected. - Scaling onto GPUs that are already provisioned continues to work normally. - Requests to provision new GPU nodes may fail for 5 to 10 seconds at a time and then retry. If your app needs to scale beyond the GPUs currently available in the region during this window, new replicas may take a few minutes longer than usual to become ready and requests may queue briefly. - Apps with cross-region enabled will spill over automatically as normal. What you can do: - Avoid deploying new versions of your apps in this window if you can. A deploy runs old and new replicas side by side and is the most common reason an app needs extra nodes. - If you have a known traffic peak in this window, consider raising your minimum replicas or scaling buffer from about 16:15 UTC so capacity is warm before the maintenance starts, and lowering it again afterwards.