Resolved incident

Errors loading the Asana web app for some users in US

Asana · Major · Started

Duration
35m
Severity
Major
Status
Postmortem

Summary

Major · Postmortem

**Incident**: A configuration change inadvertently removed a critical resource required by one of our services. While existing infrastructure continued to operate normally, newly scaled infrastructure could not become ready. As morning traffic increased, there was a growing gap between incoming requests and our system's capacity to handle them, resulting in service degradation. We resolved the issue by manually restoring the missing resource and correcting the configuration change that caused its removal. ‌ **Impact**: For approximately 1.5 hours, a subset of customers experienced a full outage of our web application and desktop applications. During this time, affected users were unable to access Asana. No customer data was lost. ‌ **Moving forward**: We are implementing additional safeguards in our configuration management process to prevent critical resources from being inadvertently removed. We are also enhancing our monitoring systems to detect degradation earlier, before it impacts customer access. ‌ Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Identified

    Asana

    We are seeing errors for part (about 10%) of users in US for the Asana webapp. We've identified the root cause and are working on identifying a resolution.

  2. Monitoring

    Asana

    We have implemented a mitigation and are monitoring to ensure recovery.

  3. Monitoring

    Asana

    We are continuing to monitor for any further issues.

  4. Resolved

    Asana

    This incident has been resolved.

  5. Postmortem

    Asana

    **Incident**: A configuration change inadvertently removed a critical resource required by one of our services. While existing infrastructure continued to operate normally, newly scaled infrastructure could not become ready. As morning traffic increased, there was a growing gap between incoming requests and our system's capacity to handle them, resulting in service degradation. We resolved the issue by manually restoring the missing resource and correcting the configuration change that caused its removal. ‌ **Impact**: For approximately 1.5 hours, a subset of customers experienced a full outage of our web application and desktop applications. During this time, affected users were unable to access Asana. No customer data was lost. ‌ **Moving forward**: We are implementing additional safeguards in our configuration management process to prevent critical resources from being inadvertently removed. We are also enhancing our monitoring systems to detect degradation earlier, before it impacts customer access. ‌ Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.