Resolved incident

Fly.io Upstash Redis service distruption (FRA region)

Upstash · Major · Started

Duration
3h 17m
Severity
Major
Status
Postmortem

Summary

Major · Postmortem

On May 12th and 13th at various times, a subset of Upstash Redis instances on [Fly.io](http://Fly.io) experienced intermittent hangs and elevated error rates. The Redis process would stall inside a logging syscall — alive but not making progress — which made the issue hard to spot from our usual telemetry. After investigating with Fly's team, we identified the root cause as a bad interaction between a recent guest kernel update on Fly's newer machines and an upstream Cloud Hypervisor bug \([cloud-hypervisor#7672](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/7672)\) affecting log writes from inside the VM. We mitigated by disabling the affected logging paths, and Fly has since rolled out a hypervisor-side patch, fully resolving the issue. No data was lost. Sorry for the disruption.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    Upstash

    Some databases may experience increased latency or timeouts in Fly.io’s FRA region.

  2. Investigating

    Upstash

    We are continuing to investigate the issue.

  3. Investigating

    Upstash

    We are working with Fly team to investigate the root cause.

  4. Resolved

    Upstash

    The incident has been resolved. We are working with Fly team on RCA.

  5. Postmortem

    Upstash

    On May 12th and 13th at various times, a subset of Upstash Redis instances on [Fly.io](http://Fly.io) experienced intermittent hangs and elevated error rates. The Redis process would stall inside a logging syscall — alive but not making progress — which made the issue hard to spot from our usual telemetry. After investigating with Fly's team, we identified the root cause as a bad interaction between a recent guest kernel update on Fly's newer machines and an upstream Cloud Hypervisor bug \([cloud-hypervisor#7672](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/7672)\) affecting log writes from inside the VM. We mitigated by disabling the affected logging paths, and Fly has since rolled out a hypervisor-side patch, fully resolving the issue. No data was lost. Sorry for the disruption.