Resolved incident
Small percentage of deployments experiencing outage
Convex · Major · Started
- Duration
- 43m
- Severity
- Major
- Status
- Resolved
Summary
Major · Resolved
This issue is considered resolved. Impact: A small percentage of Convex deployments experienced sporadic availability for approximately 17 minutes, from 22:42 to 22:59 PM PT. Root cause: one of our database clusters experienced a routine primary failover event. During the primary promotion process, the orchestration system failed to provision a replica that was ready to accept new database writes. Out of an abundance of caution, Convex will only use database clusters that support semi-sync replication, where at least two machines persist the writes before considering the write successful. Since the replicas were not ready, this cluster was not usable by Convex. Human operators were paged to intervene and fix the broken automation. We're working with our database partners to better understand what went wrong and what changes will prevent it in the future.
Event timeline
How this incident unfolded, from the official status page.
Resolved
Convex
This issue is considered resolved. Impact: A small percentage of Convex deployments experienced sporadic availability for approximately 17 minutes, from 22:42 to 22:59 PM PT. Root cause: one of our database clusters experienced a routine primary failover event. During the primary promotion process, the orchestration system failed to provision a replica that was ready to accept new database writes. Out of an abundance of caution, Convex will only use database clusters that support semi-sync replication, where at least two machines persist the writes before considering the write successful. Since the replicas were not ready, this cluster was not usable by Convex. Human operators were paged to intervene and fix the broken automation. We're working with our database partners to better understand what went wrong and what changes will prevent it in the future.
Monitoring
Convex
A fix has been implemented and we are monitoring the results.
Investigating
Convex
Seems related to a problematic database failover. Updates soon.