Resolved incident

VM Creation Failure for A100 Infiniband Type VMs in us-east1-a Region

Crusoe · Minor · Started

Duration
16h 50m
Severity
Minor
Status
Resolved

Summary

Minor · Resolved

This incident is now resolved.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Resolved

    Crusoe

    This incident is now resolved.

  2. Monitoring

    Crusoe

    A fix has been implemented, and we are monitoring the environment.

  3. Identified

    Crusoe

    We have identified an issue that is preventing new or restarted Virtual Machines from booting successfully on our A100 Infiniband hardware fleet. Any new VM provisioning request for this hardware type will also fail. Additionally, any existing VM on an A100 Infiniband node that is stopped and started (or rebooted) will also fail to come back online. Existing, currently running VMs are not affected and will continue to operate normally. We advise customers to avoid rebooting critical workloads on this hardware until a resolution is in place. Our engineering teams are actively investigating the root cause and are working to restore normal provisioning operations as quickly as possible.