BeamImage build service degraded due to AWS outage in us-east-1
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
The backlog data has been processed. This is now resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Lambda is back in business, and the backlog of events triggered during the outage is being processed.
Resolved
This incident has been resolved.
Resolved
The incident has been resolved
Resolved
Postmortem. On 2023-06-10, from 13:31 - 17:31 UTC, we experienced EC2 instance store volume degradation on a Pageserver in the US West region. Up to 10% of the projects in that region were unavailable for reads and writes during this period. 60 projects experienced stuck management operations. No user data was lost in this incident. ## Timeline T+0: instance store volume degradation observed on one Pageserver T+10min: monitoring detects problem, an alert fires, but at wrong priority level T+1h30min: the mid-prioritized alert is observed by on-call engineer. T+1h40min: issue triaged, initiated. T+2h15min: 15 projects fixes T+3h20min: pageserver migrated, all but 45 projects fixed. T+4h: remaining projects fixed, incident resolved ## Investigation, Root cause & Analysis ### Root Cause The root cause was degradation of the EC2 instance volume degradation. This resulted in the NVMe kernel driver reporting IO errors to the filesystem (ext4). GetPage@LSN requests that missed the page cache would fail with **`Input/output error (os error 5)`** from that point on. The filesystem quickly remounted as read-only in response to the NVMe errors. This prevented ingest of new WAL and also caused any Pageserver management operations to fail, e.g., branching or branch/project deletion. Failing management operations manifest as “stuck operations” in the Neon Console UI, while Console will retry such management operations in the background. ### Discovery / Altering / Monitoring The alert for stuck projects fired within 10 minutes after EC2 notified us about the issue, but was tagged with the wrong priority level. This delayed discovery of the incident by 1h30m. ### Mitigation Resolving the issue took too long (1h40min for majority of tenants, 2h20min for the last 60). We had recently added a Control Plane operation to relocate tenants between Pageservers at the push of a button, or API call. However, this functionality was developed for a different purpose, and not prepared to handle a dying source Pageserver. Hence, we had to improvise a bulk relocation script. The script worked well for all tenants except the last 60. These were stuck in management operation retry loops that depended on the affected Pageserver. Examples for management operations are start & suspend of computes or branch creation & deletion. ## Follow-Ups **Speed Up Detection**: We corrected the alerting rule to avoid such needless (and embarrassing!) delays in the future. **Speed Up Mitigation**: We were already working on the HA story for Pageserver. However, this incident has shown that we need to improve our stop-gap solution until that project goes live. These improvements are: • Streamlined on-call playbook for disabling a Pageserver completely and declaring it dead. • Automated relocation of tenants from a declared-dead Pageserver to other Pageservers in the same region. The implementation of these improvements is straight-forward and scheduled for delivery within the coming weeks. This will bring down mitigation time to the scale of minutes for projects that do not execute management operations at the time of Pageserver failure. In this incident, all but the last 60 tenants in this incident would have been fixed within minutes. Projects with ongoing management operations at the time of Pageserver failure will continue to require human inspection for the time being. We will improve the on-call playbooks to triage these cases faster, while working on a long-term solution. **Pageserver HA:** We are already developing the Pageserver HA story and will have more to share in the coming months.
Resolved
This incident has been resolved.
Resolved
We're all set! Users can now create documents with no degradation. If you continue to experience any issues with this, please reach out to us at support@pandadoc.com.
Resolved
This incident has been resolved. A postmortem will be coming by EOD
Resolved
This incident has been resolved.
Resolved
Postmortem: [https://www.svix.com/blog/outage-us-june-2023/](https://www.svix.com/blog/outage-us-june-2023/)
Resolved
This incident has been resolved.
Resolved
Our engineering team has fixed the problem and we are closely monitoring the platform. Thank you for your patience.
Resolved
No further issues encountered since the last status page update. Marking this incident as resolved. Post incident analysis is in flight to determine preventative actions and other improvement opportunities to reduce the risk of recurrence. Thank you for your patience and partnership through this.
Resolved
There was an issue between 9:00-9:40 EST that would have seen new user sign ups and password resets fail to send verification emails and would have blocked this functionality. The issue has been fully resolved.
Resolved
The incident has been resolved.
Resolved
The incident has been resolved. There was a problem with SSL.
Resolved
Amazon has resolved the issue.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We observed delays in propagating flag updates to SDKs from 20:01 to 21:23 UTC. This issue is now resolved.
Resolved
Issues related to record field changes is resolved in production, however new updates are still being published over the next 24 hours to take into account the upcoming feature updates planned for next week.
Resolved
The backlog has been completely processed and all systems are now up to date.
Resolved
This incident has been resolved.
Resolved
We observed minor delays in the processing of context data. Impact was from 13:34 UTC - 15:12 UTC. This incident has been resolved.
Resolved
Maintenance on our J1-Jumbo
Resolved
Maintenance on our J1-Jumbo
Resolved
Maintenance on our J1-Jumbo
Resolved
Suspension has been lifted.
Resolved
This incident has been resolved.
Resolved
The issue resolved itself after 30 minutes.
Resolved
Coda experienced a partial outage due to problems with our networking infrastructure. The AWS team worked diligently with us to narrow down the issue and identified a DNS limit that surfaced on larger than usual load. We have already applied changes to improve the stability of this infrastructure and we are continuing to improve our systems to ensure this problem does not recur.
Resolved
A fix has been deployed for the issue that was impacting Brave browsers. Brave browser users should now be able to edit docs as before. The issue is now resolved.
Resolved
Some SSO logins on the GovCloud environment were generating a 403 error. An overly aggressive WAF rule in the Amazon Shield system blocked the requests. The DevOps team applied the necessary WAF rule policy changes to resolve the issue.
Resolved
This incident has been resolved.
Resolved
Success! Google Safe Browsing has removed the malware flag from rubygems.org. Special thanks are owed to everyone who connected us to the Chrome and Safe Browsing teams, and the team members who assisted in clearing this up.
Resolved