Resolved incident

Region ap-northeast-1 outage on Upstash Global

Upstash · Critical · Started

Duration
1h
Severity
Critical
Status
Postmortem

Summary

Critical · Postmortem

On March 6, between approximately 13:44–14:07 UTC, some databases experienced elevated latency and connection errors in the Tokyo \(ap-northeast-1\) region. The issue was caused by a sudden spike in traffic that significantly increased network utilization and connection load on a subset of nodes. Our team mitigated the incident by scaling up capacity in the region and redistributing load across additional nodes. Service recovered once the additional capacity was brought online. Resolution We have increased the number of machines in the Tokyo region to provide additional headroom and reduce the likelihood of similar incidents during traffic spikes. Next Steps We are continuing to review capacity safeguards and connection-handling limits to improve resilience against sudden traffic surges.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    Upstash

    We are currently investigating this issue.

  2. Monitoring

    Upstash

    A fix has been implemented and we are monitoring the results.

  3. Resolved

    Upstash

    This incident has been resolved.

  4. Postmortem

    Upstash

    On March 6, between approximately 13:44–14:07 UTC, some databases experienced elevated latency and connection errors in the Tokyo \(ap-northeast-1\) region. The issue was caused by a sudden spike in traffic that significantly increased network utilization and connection load on a subset of nodes. Our team mitigated the incident by scaling up capacity in the region and redistributing load across additional nodes. Service recovered once the additional capacity was brought online. Resolution We have increased the number of machines in the Tokyo region to provide additional headroom and reduce the likelihood of similar incidents during traffic spikes. Next Steps We are continuing to review capacity safeguards and connection-handling limits to improve resilience against sudden traffic surges.