Resolved incident

elevated latency/errors for a subset of customers

Convex · Major · Started

Duration
8h 8m
Severity
Major
Status
Resolved

Summary

Major · Resolved

This incident has been resolved.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Resolved

    Convex

    This incident has been resolved.

  2. Monitoring

    Convex

    We are continuing to monitor for any further issues.

  3. Monitoring

    Convex

    There will be sporadic availability issues for a small subset of non-pro customers. We apologize for this impact. There was an issue with a downstream provider which resulted in a database cluster running with an inefficient table format. We're in the process of rewriting this data format but have had to shed load to provide headroom while reoptimizing the database.

  4. Monitoring

    Convex

    All deployments should be restored. We are monitoring the database to ensure it remains stable.

  5. Identified

    Convex

    We're continuing to work on a fix for the issue. Expect another update in ~90 minutes.

  6. Identified

    Convex

    we've brought back a subset of free deployments. Continuing to remediate.

  7. Identified

    Convex

    We've restored affected pro deployments. Working on bringing back affected free deployments.

  8. Identified

    Convex

    We're investigating an issue with a database cluster leading to degraded performance and higher error rates for affected deployments.