Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

May 6, 2026

  1. Help ScoutDocs and AI Answers Issues

    Resolved - The loading issues with AI Answers and Docs have been resolved, so we're closing this out. Sorry for the interruption today, folks!

    Resolved

  2. 1Password1Password Hosted Service in the US experiencing high latency and errors

    # 2026-04-14 SEV1 Incident Postmortem - Hosted Service in the US experiencing high latency and errors # Incident Postmortem - Elevated Application Error **Date of Incident:** 2026-04-14 **Time of Incident \(UTC\):** 17:25 - 18:00 **Service\(s\) Affected:** Sign in, Admin console **Impact Duration:** Approximately 35 minutes ## Summary On April 14, 2026, [1Password.com](http://1Password.com) experienced elevated error rates due to an increase in traffic to our USA/Global infrastructure. Requests arrived more quickly than they could be serviced in our USA/global region. Customers experienced errors and high latency. This was not a security incident, and there was no loss of data. ## Impact on Customers * **Sign-in and account access:** Some customers in the USA/Global region experienced errors when attempting to sign in or access their 1Password accounts between 17:25 and 18:00 UTC on April 14, 2026. * **All 1Password clients:** The disruption applied to all client types, including the web app, browser extensions, and desktop and mobile applications, for customers connecting through the USA/Global region. * **Number of affected requests:** Approximately 2.95% of requests over the incident window to [1Password.com](http://1Password.com) \(USA/Global\) during the incident window resulted in errors or timeouts, though for some brief spikes up to 45% of requests failed. * **Geographic regions affected:** Customers on our USA/Global only, other regions were not affected. ## What Happened? An unusually high volume of traffic exceeded the capacity of our controls, and triggered latency and errors in the underlying applications. * **Timeline of events:** * **2026-04-14 17:25** - Start of incident. * **2026-04-14 17:36** - Issue identified. * **2026-04-14 17:51** - Fix implemented. * **2026-04-14 18:01** - Recovery complete. * **Root cause analysis:** An increase in traffic exceeded the capacity of our traffic management layer, causing degraded service for customers in the USA/Global region. ## How Was It Resolved? * Established runbooks for load shedding and system upgrades mitigated the issue. ## What We Are Doing to Prevent Future Incidents * **Resolution:** We made two key changes to prevent recurrence: 1. **Increased capacity of internal systems** 2. **Upgraded traffic controls at the network edge** * **Verification:** Since implementing these changes, we have observed several additional instances of unusually high traffic matching this incident. In each case, the new controls handled the traffic automatically with no impact to customers or our systems. ## Next Steps and Communication * No action is required from our customers at this time. We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding. Sincerely, The 1Password Team

    Resolved

  3. GradleFailed upgrade on the Plugin Portal

    While trying to deploy an upgrade, the application was down for 20 minutes from 11:19 GMT+2. We will investigate the root cause and attempt the upgrade at a later date.

    Resolved

  4. UpstashQStash US Region: Schedule Degradation

    # **Incident Postmortem: Scheduled Jobs Inconsistency in US Region** On May 1, 2026, we experienced an incident affecting a subset of schedules in the US region following a recent infrastructure update. The issue has been resolved, and all affected schedules have been restored. ## **Summary** As part of an ongoing scalability improvement, we recently updated scheduling infrastructure in the US region to a new architecture. During this transition, a legacy execution path remained in the codebase as a fallback mechanism. On May 1, a bug caused the system to revert to the legacy path. This resulted in inconsistent state between the old and new scheduling systems for some users. ## **Impact** The incident affected a limited number of users in the US region. **Most users were not affected**, and the vast majority of schedules continued operating normally throughout the incident. Users who did not update schedules during the transition window continued operating normally throughout the incident. **A subset of users who created, edited, paused, or deleted schedules between April 24 and May 1 may have experienced one or more of the following:** * Schedule updates not being reflected * Paused schedules becoming active again * Deleted schedules reappearing * Newly created schedules not executing as expected Schedules created after the transition may have stopped executing briefly before recovery. ## **Root Cause** During the transition, the new scheduling infrastructure became the source of truth for schedule state. Due to a bug, the system unexpectedly reverted traffic to the legacy scheduling path, which began accepting updates independently from the new system. This caused the two systems to diverge and resulted in inconsistent schedule state for affected users. ## **Resolution** After identifying the issue, we: 1. Restored the new scheduling system as the active source of truth 2. Reconciled data between the legacy and new systems 3. Updated missing schedule changes back into the new infrastructure 4. Performed conflict resolution to preserve user data and schedule continuity In some cases, schedules that had previously been paused or deleted were restored to avoid permanent data loss. ## **Preventive Measures** We are implementing several changes to prevent similar incidents: * Removing obsolete fallback execution paths after transitions complete * Adding automated safeguards and alerts for unexpected system fallback behavior * Improving consistency validation between systems * Expanding rollback and reconciliation testing We apologize for the disruption and appreciate everyone’s patience while we resolved the issue.

    Resolved

  5. TrueLayerPLN Delayed Payment Settlement

    We experienced an issue with PLN payments where clients may have seen delayed settlement of payins and payouts. The issue has been resolved and payments are settling as normal. Impact start time: 06/05/2026, 01:46 UTC Impact end time: 06/05/2026, 07:08 UTC Client actions: None

    Resolved

  6. XeroGlobal: Xero- some customers may be experiencing issues in Xero

    Our team has resolved the issue affecting customers trying to access Xero. We apologise for the disruption this, and the other incidents today, have caused. We know how important reliable access to Xero is for you, and we appreciate your patience.

    Resolved

  7. ZapierErrors related to Klaviyo app integration

    The issue causing 522 timeout errors with the Klaviyo app integration has been resolved. This was related to an upstream service disruption on Klaviyo's side: https://status.klaviyo.com/incidents/yyh7s59brj67 Zaps using Klaviyo should now be functioning as expected. No data loss occurred — any Zap runs that failed during the incident can be replayed from Zap History. If you are still experiencing issues with your Klaviyo integration, please contact Support: https://zapier.com/app/get-help Thank you for your patience.

    Resolved

  8. XeroGlobal - some customers would have experienced issues using Xero

    We've resolved the issue that was preventing some customers from navigating Xero and completing tasks. We're sorry for the disruption this caused. A fix is now in place, and we'll be taking time to learn from this incident to help prevent it from happening again. Thank you for your patience.

    Resolved

  9. BraintrustHigh rate of errors on control plane

    The issue was caused by our firewall provider unexpectedly blocking traffic despite safeguards we had in place to prevent this behavior. We are actively working with the vendor on additional mitigation measures to reduce the likelihood of this occurring again.

    Resolved

  10. ZapierSession initialization failed error when trying to connect apps

    We’ve resolved an issue where some users could see a “Session initialization failed” error when creating new app connections. The issue was caused by an unintended staged rollout configuration and was mitigated by rolling back the rollout. Existing Zaps and workflows were not impacted. If you saw a failed connection attempt during the incident window, please retry creating the connection now. If you’re still experiencing issues creating new app connections, please contact Zapier Support.

    Resolved

May 5, 2026

  1. GitHubIncomplete pull request results in repositories

    On April 28, 2026, at approximately 14:07 UTC, GitHub received reports that pull requests were missing from search results across global and repository /pulls pages. <br /><br />The issue was caused by a manually invoked repair job intended for a single repository, which was executed without the required safety flags. During execution of the repair job, the database query remained correctly scoped to the repo’s PR IDs. However, the Elasticsearch reconciliation logic did not apply the same scope. It interpreted the min and max PR IDs as a continuous range, causing unrelated PR documents across other repos to be marked for deletion. This resulted in the removal of 1,789,756,838 PR documents from the search index, approximately 49% of indexed PR documents. <br /><br />Customer impact was limited to PR search and list discoverability. Primary storage was unaffected, and there was no impact to opening, updating, or merging PRs. <br /><br />The issue was identified ~10 minutes after initial customer reports. Because it affected search index completeness rather than service availability, it was not caught by existing monitoring. <br /><br />The root cause was a flaw in the search document repair framework: it allowed a scoped reconciliation to run without enforcing a matching Elasticsearch query scope. This created a destructive mismatch between the source-of-truth and the index. The issue was compounded by the ability to trigger the job from the production console without safety defaults. Prior testing focused only on safe backfill scenarios and did not cover this reconciliation path. Additionally, there was no automated detection for large-volume deletions in Elasticsearch. <br /><br />We mitigated the incident through three parallel actions: (1) Deployed a MySQL-backed search fallback for the most active repos by traffic to restore PR visibility for highly impacted users (2) Initiated a snapshot restore and reindex process to repopulate missing pull request documents in Elasticsearch (3) Added a degradation notice on PR pages to inform users of incomplete search results while recovery was in progress. The incident was resolved on May 1, 2026 at 4:15 UTC, following completion and validation of the reindex process. <br /><br />To prevent recurrence, we are prioritizing improvements to the repair framework and safeguards. These include enforcing scoped query alignment between primary storage and Elasticsearch, preventing destructive operations without explicit opt-in, strengthening guardrails for manual repair jobs, and evaluating restrictions on production console access. <br /><br />In parallel, we are expanding automated test coverage for reconciliation safety invariants and introducing detection for anomalous deletion patterns in Elasticsearch so similar issues can be identified or blocked earlier. <br /><br />We are committed to improving the safety and reliability of our repair systems and ensuring that operational workflows are resilient to both software defects and manual invocation risks.

    Resolved

  2. Linode(Copy Fail) Linux Kernel Local Privilege Escalation Vulnerability [CVE-2026-31431]

    We have completed our investigation and response to the “Copy Fail” Linux kernel local privilege escalation vulnerability (CVE-2026-31431). We have published a documentation article with detailed guidance on available mitigations and recommended actions for affected systems. Customers can find more information and step-by-step instructions here: https://www.linode.com/docs/guides/cve-2026-31431-copy-fail-mitigation/ We encourage all customers to review the article and apply the appropriate mitigations to their environments. If you have questions or need assistance, please contact us at 855-454-6633 (+1-609-380-7100 Intl.) or email support@linode.com for assistance.

    Resolved

  3. XeroGlobal: Xero - some customers may be experiencing errors in Xero

    Our team has resolved the issue impacting customers trying to navigate and use Xero. We apologise for any inconvenience this has caused. A product release caused Xero's accessing and core functions to become unavailable, and a rollback has restored service.

    Resolved

Older incidents