Resolved incident

Intermittent Impact from AWS Outage

Convex · Major · Started

Duration
1h 49m
Severity
Major
Status
Resolved

Summary

Major · Resolved

We’ve mitigated the impact from AWS’s Availability Zone issue, and service has remained stable. We have not observed further node failures or customer impact, so we’re marking this incident as resolved. We’ll continue to monitor AWS’s recovery separately.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    Convex

    We’re experiencing intermittent downtime from an upstream infrastructure provider. Some customer workloads may see brief error spikes while affected jobs are reallocated to healthy hardware and databases fail over. We’re monitoring recovery and will provide updates as mitigation progresses.

  2. Identified

    Convex

    AWS has identified an Availability Zone-level issue affecting our infrastructure. Some customer workloads may continue to experience intermittent errors while affected jobs are reallocated to healthy hardware and databases fail over. We’re continuing to monitor recovery and will provide updates as AWS mitigation progresses.

  3. Identified

    Convex

    AWS’s Availability Zone-level issue is continuing to cause impact to our infrastructure. Some customers may experience extended downtime or elevated error rates while affected workloads recover on healthy infrastructure. We’re actively working to restore service for impacted customers and will continue to provide updates as recovery progresses.

  4. Monitoring

    Convex

    We’ve moved affected infrastructure out of the impacted AWS Availability Zone, and traffic should now be restored. We’re continuing to monitor for any residual impact while AWS’s recovery progresses.

  5. Resolved

    Convex

    We’ve mitigated the impact from AWS’s Availability Zone issue, and service has remained stable. We have not observed further node failures or customer impact, so we’re marking this incident as resolved. We’ll continue to monitor AWS’s recovery separately.