ec2 instance disk falure in us-west-2, migrating projects to another pageserver

Resolved

Started

Lasted < 1m

View official incident ↗

Updates

  1. Resolved

    Postmortem. On 2023-06-10, from 13:31 - 17:31 UTC, we experienced EC2 instance store volume degradation on a Pageserver in the US West region. Up to 10% of the projects in that region were unavailable for reads and writes during this period. 60 projects experienced stuck management operations. No user data was lost in this incident. ## Timeline T+0: instance store volume degradation observed on one Pageserver T+10min: monitoring detects problem, an alert fires, but at wrong priority level T+1h30min: the mid-prioritized alert is observed by on-call engineer. T+1h40min: issue triaged, initiated. T+2h15min: 15 projects fixes T+3h20min: pageserver migrated, all but 45 projects fixed. T+4h: remaining projects fixed, incident resolved ## Investigation, Root cause & Analysis ### Root Cause The root cause was degradation of the EC2 instance volume degradation. This resulted in the NVMe kernel driver reporting IO errors to the filesystem (ext4). GetPage@LSN requests that missed the page cache would fail with **`Input/output error (os error 5)`** from that point on. The filesystem quickly remounted as read-only in response to the NVMe errors. This prevented ingest of new WAL and also caused any Pageserver management operations to fail, e.g., branching or branch/project deletion. Failing management operations manifest as “stuck operations” in the Neon Console UI, while Console will retry such management operations in the background. ### Discovery / Altering / Monitoring The alert for stuck projects fired within 10 minutes after EC2 notified us about the issue, but was tagged with the wrong priority level. This delayed discovery of the incident by 1h30m. ### Mitigation Resolving the issue took too long (1h40min for majority of tenants, 2h20min for the last 60). We had recently added a Control Plane operation to relocate tenants between Pageservers at the push of a button, or API call. However, this functionality was developed for a different purpose, and not prepared to handle a dying source Pageserver. Hence, we had to improvise a bulk relocation script. The script worked well for all tenants except the last 60. These were stuck in management operation retry loops that depended on the affected Pageserver. Examples for management operations are start & suspend of computes or branch creation & deletion. ## Follow-Ups **Speed Up Detection**: We corrected the alerting rule to avoid such needless (and embarrassing!) delays in the future. **Speed Up Mitigation**: We were already working on the HA story for Pageserver. However, this incident has shown that we need to improve our stop-gap solution until that project goes live. These improvements are: • Streamlined on-call playbook for disabling a Pageserver completely and declaring it dead. • Automated relocation of tenants from a declared-dead Pageserver to other Pageservers in the same region. The implementation of these improvements is straight-forward and scheduled for delivery within the coming weeks. This will bring down mitigation time to the scale of minutes for projects that do not execute management operations at the time of Pageserver failure. In this incident, all but the last 60 tenants in this incident would have been fixed within minutes. Projects with ongoing management operations at the time of Pageserver failure will continue to require human inspection for the time being. We will improve the on-call playbooks to triage these cases faster, while working on a long-term solution. **Pageserver HA:** We are already developing the Pageserver HA story and will have more to share in the coming months.

  2. Investigating

    Identified: (15:08) We are continuing to work on a fix for this incident. Monitoring: (17:31) All stuck projects are now available. We are performing a few more checks before fully resolving the incident.