Resolved incident
Performance degradation on QStash
Upstash · Minor · Started
- Duration
- 34m
- Severity
- Minor
- Status
- Postmortem
Summary
Minor · Postmortem
On 09:43 UTC, QStash experienced degraded service when a high number of requests to a specific domain were throttled, resulting in timeouts during an unexpected phase of the TCP connection establishment. These requests and resulting retries triggered excessive consumption of network resources and negatively impacted all users. We have added more resources to QStash as a quick remediation and as for the resolution, we have improved the timeout mechanism to detect and fail-faster in such cases.
Event timeline
How this incident unfolded, from the official status page.
Investigating
Upstash
We are currently investigating this issue.
Monitoring
Upstash
A fix has been implemented and we are monitoring the results.
Resolved
Upstash
This incident has been resolved.
Postmortem
Upstash
On 09:43 UTC, QStash experienced degraded service when a high number of requests to a specific domain were throttled, resulting in timeouts during an unexpected phase of the TCP connection establishment. These requests and resulting retries triggered excessive consumption of network resources and negatively impacted all users. We have added more resources to QStash as a quick remediation and as for the resolution, we have improved the timeout mechanism to detect and fail-faster in such cases.