Resolved incident

Refactored Hubs Down

ReadMe · Major · Started

Duration
99h 42m
Severity
Major
Status
Resolved

Summary

Major · Resolved

Our systems have been stable since Friday, May 29th. We're monitoring closely and improving spike detection to prevent recurrence and adding proactive capacity monitoring to detect admin dashboard degradation before it impacts customers. Full root cause analysis coming soon. Thanks for your patience and understanding.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    ReadMe

    We're actively investigating this issue.

  2. Identified

    ReadMe

    The issue has been identified and a fix is being implemented.

  3. Identified

    ReadMe

    Systems are getting back up. We're continuing to work on a permanent fix for this issue.

  4. Monitoring

    ReadMe

    A fix has been implemented and we are monitoring the results.

  5. Monitoring

    ReadMe

    We are continuing to monitor for any further issues.

  6. Monitoring

    ReadMe

    We are receiving reports of degraded performance and are actively investigating. We will provide updates as more information becomes available.

  7. Monitoring

    ReadMe

    Systems appear stable and we are actively rolling out a series of fixes. We will continue to monitor and provide updates

  8. Monitoring

    ReadMe

    Systems remain stable. Improvements and fixes have been rolled out and we are continuing to monitor.

  9. Monitoring

    ReadMe

    Systems remain stable, and we are continuing to monitor the situation. Bi-directional sync was on a maintenance job from 9:50 - 11:59 AM (ET) and is now operational.

  10. Monitoring

    ReadMe

    Systems remain stable and our team continues to deploy targeted fixes. Work is ongoing to address degraded read performance across the system, including serving all 404s from cache. We’ll be actively monitoring over the weekend, and we will provide a full RCA next week.

  11. Resolved

    ReadMe

    Our systems have been stable since Friday, May 29th. We're monitoring closely and improving spike detection to prevent recurrence and adding proactive capacity monitoring to detect admin dashboard degradation before it impacts customers. Full root cause analysis coming soon. Thanks for your patience and understanding.