Resolved incident
Migration Script Race Condition
Doppler · Critical · Started
- Duration
- < 1m
- Severity
- Critical
- Status
- Resolved
Summary
Critical · Resolved
We had a migration script that failed to run before the rest of our services deployed. This caused the newly deployed services to crash because the database was missing a column in a critical table. Our deployment orchestrator quickly detected the issue and reverted to the previous build. Total downtime was approximately 2 minutes. We plan to mitigate future failures by requiring our migrations to complete before rolling out our remaining services.
Event timeline
How this incident unfolded, from the official status page.
Resolved
Doppler
We had a migration script that failed to run before the rest of our services deployed. This caused the newly deployed services to crash because the database was missing a column in a critical table. Our deployment orchestrator quickly detected the issue and reverted to the previous build. Total downtime was approximately 2 minutes. We plan to mitigate future failures by requiring our migrations to complete before rolling out our remaining services.