Resolved incident
elevated error rates on convex for pro instances
Convex · None · Started
- Duration
- 38m
- Severity
- None
- Status
- Postmortem
Summary
None · Postmortem
At approximately 6am Pacific we had a high rate of server errors for approximately 30 minutes for a subset of customer backends. This was caused by a 20x increase in traffic that saturated one of our databases and led to resource exhaustion on one of our function execution clusters, which is why the impact spilled beyond just that specific database. We’ve scaled up our infrastructure further and are deploying some changes to load shedding and retries to minimize the impact of such an event in future. While we aim to have no cascading impact on other customers from a load spike we’re also deploying dedicated isolated hardware for customers on our upcoming Business plan.
Event timeline
How this incident unfolded, from the official status page.
Postmortem
Convex
At approximately 6am Pacific we had a high rate of server errors for approximately 30 minutes for a subset of customer backends. This was caused by a 20x increase in traffic that saturated one of our databases and led to resource exhaustion on one of our function execution clusters, which is why the impact spilled beyond just that specific database. We’ve scaled up our infrastructure further and are deploying some changes to load shedding and retries to minimize the impact of such an event in future. While we aim to have no cascading impact on other customers from a load spike we’re also deploying dedicated isolated hardware for customers on our upcoming Business plan.
Resolved
Convex
This incident has been resolved.
Monitoring
Convex
A fix has been implemented and we are monitoring the results.
Investigating
Convex
We are currently investigating an issue leading to higher error rates