Infrastructure Instability

Resolved

Major impact

Started

Lasted 6h 12m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Monitoring

    All systems are currently operational and the team is continuing to monitor while we investigate.

  3. Identified

    Production services are back online. We are monitoring their state and continuing to work on the Sandbox API and other services.

  4. Identified

    We've restored most customer facing services. We are still actively restoring the Sandbox API and other internal services. Thanks.

  5. Identified

    We've restored half of our infrastructure, and are presently restoring services for the other half.

  6. Identified

    Nameservers are currently recovering, and we are bringing other services online.

  7. Identified

    Restores are completing and we will be bringing DNS services back online soon.

  8. Identified

    We've started restoring backend services. We expect to have services restored shortly!!

  9. Identified

    We are still working diligently to restore service in our new production orchestration cluster. We will continue to provide updates as the status changes.

  10. Identified

    Our initial attempts to restore our cluster has failed. We are working on bringing the services up in a fresh cluster. Thank you for your patience

  11. Identified

    The issue has been identified and a fix is being implemented.

  12. Investigating

    We are running restores to bring back services online

  13. Investigating

    We are currently investigating an issue affecting our infrastructure that is causing some applications to restart unexpectedly. As a result, customers may experience intermittent disruptions or degraded performance across our systems. Our team is actively working to stabilize the situation and restore normal operation as quickly as possible. We’ll continue to share updates here as we learn more. Thank you for your patience.