Resolved incident

Investigating connectivity issues across accounts and devices

monday.com · Critical · Started

Duration
38m
Severity
Critical
Status
Postmortem

Summary

Critical · Postmortem

**What Happened:** On February 16th \(09:04 AM EST\), our US customers experienced a 28-minute system downtime due to an application-level bug. ‌ **What caused it:** A recent code change in the billing subsystem responsible for updating account plans introduced an edge-case bug that dramatically increased platform load, which caused system downtime. ‌ **How we responded:** Our monitoring system promptly identified the issue, triggering an immediate response from our engineering team. By activating failover mechanisms, reverting the problematic code change, and blocking the affected endpoint, we restored normal service by 09:32 AM EST. ‌ **What we're doing to prevent recurrence:** To enhance system resilience, we are implementing database optimizations, additional guardrails, and limits to prevent load spikes on the platform. It also includes strengthening our error detection and recovery processes. ‌ We sincerely apologize for any disruption this caused to your workflow. Thank you for your understanding as we work to continuously improve our platform's reliability. Your team at [monday.com](http://monday.com)

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    monday.com

    We are currently investigating reports of connectivity and loading issues across the platform. Our team is working to resolve this promptly.

  2. Identified

    monday.com

    Our team has identified the root cause of the issue and is working to resume regular service usage. We will continue to provide updates on their progress

  3. Monitoring

    monday.com

    A fix has been implemented, and we are monitoring results to restore full system stability. Automations may still be delayed.

  4. Resolved

    monday.com

    The platform is now back to regular service. Please refresh your browser to access the platform. Thank you for your patience

  5. Postmortem

    monday.com

    **What Happened:** On February 16th \(09:04 AM EST\), our US customers experienced a 28-minute system downtime due to an application-level bug. ‌ **What caused it:** A recent code change in the billing subsystem responsible for updating account plans introduced an edge-case bug that dramatically increased platform load, which caused system downtime. ‌ **How we responded:** Our monitoring system promptly identified the issue, triggering an immediate response from our engineering team. By activating failover mechanisms, reverting the problematic code change, and blocking the affected endpoint, we restored normal service by 09:32 AM EST. ‌ **What we're doing to prevent recurrence:** To enhance system resilience, we are implementing database optimizations, additional guardrails, and limits to prevent load spikes on the platform. It also includes strengthening our error detection and recovery processes. ‌ We sincerely apologize for any disruption this caused to your workflow. Thank you for your understanding as we work to continuously improve our platform's reliability. Your team at [monday.com](http://monday.com)