StripeElevated API error rates from 18:15 - 18:34 UTC
From 18:15 - 18:33 UTC we saw elevated error rates and response times with requests to Invoicing and Billing-related APIs. This is now resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
From 18:15 - 18:33 UTC we saw elevated error rates and response times with requests to Invoicing and Billing-related APIs. This is now resolved.
Resolved
This incident has been resolved.
Resolved
Resolved - The loading issues with AI Answers and Docs have been resolved, so we're closing this out. Sorry for the interruption today, folks!
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The delays in SIM card management actions have been resolved. All operations are processing normally.
Resolved
This incident has been resolved.
Resolved
# 2026-04-14 SEV1 Incident Postmortem - Hosted Service in the US experiencing high latency and errors # Incident Postmortem - Elevated Application Error **Date of Incident:** 2026-04-14 **Time of Incident \(UTC\):** 17:25 - 18:00 **Service\(s\) Affected:** Sign in, Admin console **Impact Duration:** Approximately 35 minutes ## Summary On April 14, 2026, [1Password.com](http://1Password.com) experienced elevated error rates due to an increase in traffic to our USA/Global infrastructure. Requests arrived more quickly than they could be serviced in our USA/global region. Customers experienced errors and high latency. This was not a security incident, and there was no loss of data. ## Impact on Customers * **Sign-in and account access:** Some customers in the USA/Global region experienced errors when attempting to sign in or access their 1Password accounts between 17:25 and 18:00 UTC on April 14, 2026. * **All 1Password clients:** The disruption applied to all client types, including the web app, browser extensions, and desktop and mobile applications, for customers connecting through the USA/Global region. * **Number of affected requests:** Approximately 2.95% of requests over the incident window to [1Password.com](http://1Password.com) \(USA/Global\) during the incident window resulted in errors or timeouts, though for some brief spikes up to 45% of requests failed. * **Geographic regions affected:** Customers on our USA/Global only, other regions were not affected. ## What Happened? An unusually high volume of traffic exceeded the capacity of our controls, and triggered latency and errors in the underlying applications. * **Timeline of events:** * **2026-04-14 17:25** - Start of incident. * **2026-04-14 17:36** - Issue identified. * **2026-04-14 17:51** - Fix implemented. * **2026-04-14 18:01** - Recovery complete. * **Root cause analysis:** An increase in traffic exceeded the capacity of our traffic management layer, causing degraded service for customers in the USA/Global region. ## How Was It Resolved? * Established runbooks for load shedding and system upgrades mitigated the issue. ## What We Are Doing to Prevent Future Incidents * **Resolution:** We made two key changes to prevent recurrence: 1. **Increased capacity of internal systems** 2. **Upgraded traffic controls at the network edge** * **Verification:** Since implementing these changes, we have observed several additional instances of unusually high traffic matching this incident. In each case, the new controls handled the traffic automatically with no impact to customers or our systems. ## Next Steps and Communication * No action is required from our customers at this time. We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding. Sincerely, The 1Password Team
Resolved
We'll continue monitoring
Resolved
This incident has been resolved
Resolved
This incident has been resolved.
Resolved
Our team has resolved the issue causing some customers to experience errors when accessing Xero. We apologise for any inconvenience this has caused.
Resolved
While trying to deploy an upgrade, the application was down for 20 minutes from 11:19 GMT+2. We will investigate the root cause and attempt the upgrade at a later date.
Resolved
We caught up with our queues and have regular processing times again.
Resolved
# **Incident Postmortem: Scheduled Jobs Inconsistency in US Region** On May 1, 2026, we experienced an incident affecting a subset of schedules in the US region following a recent infrastructure update. The issue has been resolved, and all affected schedules have been restored. ## **Summary** As part of an ongoing scalability improvement, we recently updated scheduling infrastructure in the US region to a new architecture. During this transition, a legacy execution path remained in the codebase as a fallback mechanism. On May 1, a bug caused the system to revert to the legacy path. This resulted in inconsistent state between the old and new scheduling systems for some users. ## **Impact** The incident affected a limited number of users in the US region. **Most users were not affected**, and the vast majority of schedules continued operating normally throughout the incident. Users who did not update schedules during the transition window continued operating normally throughout the incident. **A subset of users who created, edited, paused, or deleted schedules between April 24 and May 1 may have experienced one or more of the following:** * Schedule updates not being reflected * Paused schedules becoming active again * Deleted schedules reappearing * Newly created schedules not executing as expected Schedules created after the transition may have stopped executing briefly before recovery. ## **Root Cause** During the transition, the new scheduling infrastructure became the source of truth for schedule state. Due to a bug, the system unexpectedly reverted traffic to the legacy scheduling path, which began accepting updates independently from the new system. This caused the two systems to diverge and resulted in inconsistent schedule state for affected users. ## **Resolution** After identifying the issue, we: 1. Restored the new scheduling system as the active source of truth 2. Reconciled data between the legacy and new systems 3. Updated missing schedule changes back into the new infrastructure 4. Performed conflict resolution to preserve user data and schedule continuity In some cases, schedules that had previously been paused or deleted were restored to avoid permanent data loss. ## **Preventive Measures** We are implementing several changes to prevent similar incidents: * Removing obsolete fallback execution paths after transitions complete * Adding automated safeguards and alerts for unexpected system fallback behavior * Improving consistency validation between systems * Expanding rollback and reconciliation testing We apologize for the disruption and appreciate everyone’s patience while we resolved the issue.
Resolved
This incident has been resolved.
Resolved
We experienced an issue with PLN payments where clients may have seen delayed settlement of payins and payouts. The issue has been resolved and payments are settling as normal. Impact start time: 06/05/2026, 01:46 UTC Impact end time: 06/05/2026, 07:08 UTC Client actions: None
Resolved
The issue causing Jira users inability to log time on business board has been resolved. A fix was deployed to address the problem and the service is operating normally for all affected customers.
Resolved
Processing of test execution ingestion data has successfully caught up.
Resolved
We've implemented a fix and have mitigated impact as of 9:25pm PT. Streaming SDKs should be able to connect successfully at this time.
Resolved
Our team has resolved the issue affecting customers trying to access Xero. We apologise for the disruption this, and the other incidents today, have caused. We know how important reliable access to Xero is for you, and we appreciate your patience.
Resolved
The issue causing 522 timeout errors with the Klaviyo app integration has been resolved. This was related to an upstream service disruption on Klaviyo's side: https://status.klaviyo.com/incidents/yyh7s59brj67 Zaps using Klaviyo should now be functioning as expected. No data loss occurred — any Zap runs that failed during the incident can be replayed from Zap History. If you are still experiencing issues with your Klaviyo integration, please contact Support: https://zapier.com/app/get-help Thank you for your patience.
Resolved
We've resolved the issue that was preventing some customers from navigating Xero and completing tasks. We're sorry for the disruption this caused. A fix is now in place, and we'll be taking time to learn from this incident to help prevent it from happening again. Thank you for your patience.
Resolved
We mitigated the problem. The issue only affected apps having machines in the sin region.
Resolved
The issue was caused by our firewall provider unexpectedly blocking traffic despite safeguards we had in place to prevent this behavior. We are actively working with the vendor on additional mitigation measures to reduce the likelihood of this occurring again.
Resolved
We’ve resolved an issue where some users could see a “Session initialization failed” error when creating new app connections. The issue was caused by an unintended staged rollout configuration and was mitigated by rolling back the rollout. Existing Zaps and workflows were not impacted. If you saw a failed connection attempt during the incident window, please retry creating the connection now. If you’re still experiencing issues creating new app connections, please contact Zapier Support.
Resolved
This incident has been resolved.
Resolved
The issue has been resolved.
Resolved
On April 28, 2026, at approximately 14:07 UTC, GitHub received reports that pull requests were missing from search results across global and repository /pulls pages. <br /><br />The issue was caused by a manually invoked repair job intended for a single repository, which was executed without the required safety flags. During execution of the repair job, the database query remained correctly scoped to the repo’s PR IDs. However, the Elasticsearch reconciliation logic did not apply the same scope. It interpreted the min and max PR IDs as a continuous range, causing unrelated PR documents across other repos to be marked for deletion. This resulted in the removal of 1,789,756,838 PR documents from the search index, approximately 49% of indexed PR documents. <br /><br />Customer impact was limited to PR search and list discoverability. Primary storage was unaffected, and there was no impact to opening, updating, or merging PRs. <br /><br />The issue was identified ~10 minutes after initial customer reports. Because it affected search index completeness rather than service availability, it was not caught by existing monitoring. <br /><br />The root cause was a flaw in the search document repair framework: it allowed a scoped reconciliation to run without enforcing a matching Elasticsearch query scope. This created a destructive mismatch between the source-of-truth and the index. The issue was compounded by the ability to trigger the job from the production console without safety defaults. Prior testing focused only on safe backfill scenarios and did not cover this reconciliation path. Additionally, there was no automated detection for large-volume deletions in Elasticsearch. <br /><br />We mitigated the incident through three parallel actions: (1) Deployed a MySQL-backed search fallback for the most active repos by traffic to restore PR visibility for highly impacted users (2) Initiated a snapshot restore and reindex process to repopulate missing pull request documents in Elasticsearch (3) Added a degradation notice on PR pages to inform users of incomplete search results while recovery was in progress. The incident was resolved on May 1, 2026 at 4:15 UTC, following completion and validation of the reindex process. <br /><br />To prevent recurrence, we are prioritizing improvements to the repair framework and safeguards. These include enforcing scoped query alignment between primary storage and Elasticsearch, preventing destructive operations without explicit opt-in, strengthening guardrails for manual repair jobs, and evaluating restrictions on production console access. <br /><br />In parallel, we are expanding automated test coverage for reconciliation safety invariants and introducing detection for anomalous deletion patterns in Elasticsearch so similar issues can be identified or blocked earlier. <br /><br />We are committed to improving the safety and reliability of our repair systems and ensuring that operational workflows are resilient to both software defects and manual invocation risks.
Resolved
This incident has been resolved.
Resolved
We've implemented an update so webhooks sent with a Content-Type header that specified a non-UTF-8 character set will trigger your Zaps.
Resolved
We have completed our investigation and response to the “Copy Fail” Linux kernel local privilege escalation vulnerability (CVE-2026-31431). We have published a documentation article with detailed guidance on available mitigations and recommended actions for affected systems. Customers can find more information and step-by-step instructions here: https://www.linode.com/docs/guides/cve-2026-31431-copy-fail-mitigation/ We encourage all customers to review the article and apply the appropriate mitigations to their environments. If you have questions or need assistance, please contact us at 855-454-6633 (+1-609-380-7100 Intl.) or email support@linode.com for assistance.
Resolved
We have confirmed the fix had its intended effect. This incident is resolved.
Resolved
All data is processed in time.
Resolved
Our team has resolved the issue impacting customers trying to navigate and use Xero. We apologise for any inconvenience this has caused. A product release caused Xero's accessing and core functions to become unavailable, and a rollback has restored service.
Resolved
All docs sites are online and fully-functioning. We are still investigating the root cause of this issue and will publish a more detailed report as soon as we know more details.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved