Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Jun 14, 2023

Jun 13, 2023

Jun 12, 2023

Jun 10, 2023

  1. Neonec2 instance disk falure in us-west-2, migrating projects to another pageserver

    Postmortem. On 2023-06-10, from 13:31 - 17:31 UTC, we experienced EC2 instance store volume degradation on a Pageserver in the US West region. Up to 10% of the projects in that region were unavailable for reads and writes during this period. 60 projects experienced stuck management operations. No user data was lost in this incident. ## Timeline T+0: instance store volume degradation observed on one Pageserver T+10min: monitoring detects problem, an alert fires, but at wrong priority level T+1h30min: the mid-prioritized alert is observed by on-call engineer. T+1h40min: issue triaged, initiated. T+2h15min: 15 projects fixes T+3h20min: pageserver migrated, all but 45 projects fixed. T+4h: remaining projects fixed, incident resolved ## Investigation, Root cause & Analysis ### Root Cause The root cause was degradation of the EC2 instance volume degradation. This resulted in the NVMe kernel driver reporting IO errors to the filesystem (ext4). GetPage@LSN requests that missed the page cache would fail with **`Input/output error (os error 5)`** from that point on. The filesystem quickly remounted as read-only in response to the NVMe errors. This prevented ingest of new WAL and also caused any Pageserver management operations to fail, e.g., branching or branch/project deletion. Failing management operations manifest as “stuck operations” in the Neon Console UI, while Console will retry such management operations in the background. ### Discovery / Altering / Monitoring The alert for stuck projects fired within 10 minutes after EC2 notified us about the issue, but was tagged with the wrong priority level. This delayed discovery of the incident by 1h30m. ### Mitigation Resolving the issue took too long (1h40min for majority of tenants, 2h20min for the last 60). We had recently added a Control Plane operation to relocate tenants between Pageservers at the push of a button, or API call. However, this functionality was developed for a different purpose, and not prepared to handle a dying source Pageserver. Hence, we had to improvise a bulk relocation script. The script worked well for all tenants except the last 60. These were stuck in management operation retry loops that depended on the affected Pageserver. Examples for management operations are start & suspend of computes or branch creation & deletion. ## Follow-Ups **Speed Up Detection**: We corrected the alerting rule to avoid such needless (and embarrassing!) delays in the future. **Speed Up Mitigation**: We were already working on the HA story for Pageserver. However, this incident has shown that we need to improve our stop-gap solution until that project goes live. These improvements are: • Streamlined on-call playbook for disabling a Pageserver completely and declaring it dead. • Automated relocation of tenants from a declared-dead Pageserver to other Pageservers in the same region. The implementation of these improvements is straight-forward and scheduled for delivery within the coming weeks. This will bring down mitigation time to the scale of minutes for projects that do not execute management operations at the time of Pageserver failure. In this incident, all but the last 60 tenants in this incident would have been fixed within minutes. Projects with ongoing management operations at the time of Pageserver failure will continue to require human inspection for the time being. We will improve the on-call playbooks to triage these cases faster, while working on a long-term solution. **Pageserver HA:** We are already developing the Pageserver HA story and will have more to share in the coming months.

    Resolved

Jun 9, 2023

  1. PandaDocSlight Performance Degradation in Editor

    We're all set! Users can now create documents with no degradation. If you continue to experience any issues with this, please reach out to us at support@pandadoc.com.

    Resolved

Jun 8, 2023

Jun 7, 2023

Jun 6, 2023

  1. ApigeeIssue with Cassandra

    No further issues encountered since the last status page update. Marking this incident as resolved. Post incident analysis is in flight to determine preventative actions and other improvement opportunities to reduce the risk of recurrence. Thank you for your patience and partnership through this.

    Resolved

Jun 5, 2023

  1. LiveblocksLiveblocks

    The incident has been resolved. There was a problem with SSL.

    Resolved

Jun 2, 2023

Jun 1, 2023

May 31, 2023

May 29, 2023

May 26, 2023

  1. KeeperKCM, iOS and Android updates pending

    Issues related to record field changes is resolved in production, however new updates are still being published over the next 24 hours to take into account the upcoming feature updates planned for next week.

    Resolved

May 25, 2023

May 23, 2023

May 21, 2023

May 18, 2023

May 17, 2023

  1. CodaIntermittent failures

    Coda experienced a partial outage due to problems with our networking infrastructure. The AWS team worked diligently with us to narrow down the issue and identified a DNS limit that surfaced on larger than usual load. We have already applied changes to improve the stability of this infrastructure and we are continuing to improve our systems to ensure this problem does not recur.

    Resolved

May 16, 2023

  1. KeeperGovCloud SSO login 403 error - Resolved

    Some SSO logins on the GovCloud environment were generating a 403 error. An overly aggressive WAF rule in the Amazon Shield system blocked the requests. The DevOps team applied the necessary WAF rule policy changes to resolve the issue.

    Resolved

May 12, 2023

  1. RubyGems.orgGoogle Safe Browsing flag as malware

    Success! Google Safe Browsing has removed the malware flag from rubygems.org. Special thanks are owed to everyone who connected us to the Chrome and Safe Browsing teams, and the team members who assisted in clearing this up.

    Resolved

Older incidents