Degradation of Creations and API Operations

Resolved

Started

Lasted 2h 17m

View official incident ↗

Updates

  1. Resolved

    Between 01:20 and 03:57 UTC on 12 november, the daytona api was slow or timing out, and sandbox create, start, stop and archive operations were delayed or failed. Cause: very high traffic to the snapshot list endpoint (GET /api/snapshots) ran expensive queries on the primary database instead of a read replica, and a long-running transaction kept the database under load. We have rate-limited the traffic and service has been normal since 03:57 UTC as a temporary measure. Snapshot listing and similar read-only queries will be moved to read replicas. if you saw failed operations during this window that did not recover after retrying, please contact support.

  2. Resolved

    This incident has been resolved. Sandbox creation and API operations are back to normal. The cause was unusually high load on one of our backend systems. We've put mitigations in place and are working on longer-term changes to stop it from happening again. We apologize for the disruption.

  3. Unknown

    We are investigating an incident causing degraded sandbox creations and API operations across our fleet. The team is investigating