Region ap-northeast-1 outage on Upstash Global

Postmortem

Critical impact

Started

Lasted 1h

View official incident ↗

Updates

  1. Postmortem

    On March 6, between approximately 13:44–14:07 UTC, some databases experienced elevated latency and connection errors in the Tokyo \(ap-northeast-1\) region. The issue was caused by a sudden spike in traffic that significantly increased network utilization and connection load on a subset of nodes. Our team mitigated the incident by scaling up capacity in the region and redistributing load across additional nodes. Service recovered once the additional capacity was brought online. Resolution We have increased the number of machines in the Tokyo region to provide additional headroom and reduce the likelihood of similar incidents during traffic spikes. Next Steps We are continuing to review capacity safeguards and connection-handling limits to improve resilience against sudden traffic surges.

  2. Resolved

    This incident has been resolved.

  3. Monitoring

    A fix has been implemented and we are monitoring the results.

  4. Investigating

    We are currently investigating this issue.