Resolved incident
We are fully down for a subset of customers. We're investigating the issue, and are rolling back a deployment.
Asana · Major · Started
- Duration
- 42m
- Severity
- Major
- Status
- Postmortem
Summary
Major · Postmortem
Incident: An internal system responsible for automatically scaling backend server capacity in one of our compute clusters stopped replacing capacity that had been cycled out during routine maintenance. This caused a gradual reduction in available capacity over several hours. A subsequent deployment was activated with insufficient capacity, causing all new requests for that compute cluster to fail. To mitigate the impact, we reverted to a previous release revision, which still had sufficient capacity. Impact: For approximately 30 minutes, customers whose traffic was handled by this compute cluster saw full downtime; other customers were unaffected. No customer data was lost. Moving forward: We have added additional monitoring to detect this type of capacity-scaling failure much earlier, and have added safeguards to prevent deployments from shifting traffic before sufficient healthy capacity is confirmed. _Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average._
Event timeline
How this incident unfolded, from the official status page.
Investigating
Asana
We are currently investigating this issue.
Investigating
Asana
We are continuing to investigate this issue.
Investigating
Asana
The rollback seems to have completely resolved errors. We're monitoring closely, but users should be able to access Asana again.
Monitoring
Asana
A fix has been implemented and we are monitoring the results.
Resolved
Asana
This incident has been resolved.
Postmortem
Asana
Incident: An internal system responsible for automatically scaling backend server capacity in one of our compute clusters stopped replacing capacity that had been cycled out during routine maintenance. This caused a gradual reduction in available capacity over several hours. A subsequent deployment was activated with insufficient capacity, causing all new requests for that compute cluster to fail. To mitigate the impact, we reverted to a previous release revision, which still had sufficient capacity. Impact: For approximately 30 minutes, customers whose traffic was handled by this compute cluster saw full downtime; other customers were unaffected. No customer data was lost. Moving forward: We have added additional monitoring to detect this type of capacity-scaling failure much earlier, and have added safeguards to prevent deployments from shifting traffic before sufficient healthy capacity is confirmed. _Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average._