AblyElevated Realtime errors in EU regions
The incident has been resolved
Resolved
Outages and recoveries from the services you depend on.
40 incidents
The incident has been resolved
Resolved
This incident has been resolved.
Resolved
services.gradle.org is fully operational again. The outage was caused by a configuration change on services.gradle.org that was not intended to affect this service.
Resolved
This is now resolved, triggering is back to real-time.
Resolved
This incident has been resolved.
Resolved
Our load-balancing of ingestion traffic is doing its job and everything is back to normal.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Engineers have resolved the issue and service to connected emails has been restored.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
## **Incident Postmortem - Sign-in Service Degradation** * **Date of Incident:** 2025-08-05 * **Time of Incident \(UTC\):** 20:20 - 21:20 * **Service\(s\) Affected:** Sign-in, Web Application, Command Line Interface \(CLI\), Single Sign On \(SSO\), API’s * **Impact Duration:** 1 hour ### **Summary** On August 5, 2025, 1Password experienced a service degradation that impacted customers' ability to sign in and access the web application. The incident was triggered during a planned architectural improvement when a misconfigured rollback attempt caused an overload of traffic and a subsequent database connection bottleneck. The issue was resolved by correcting the misconfiguration and restarting the web application servers, fully restoring service. ### **Impact on Customers** During the service disruption, some customers experienced degraded performance when accessing their 1Password vaults and signing in. * **Sign-in Issues**: Customers may have experienced sign-in slowness or timeouts. * **Error Messages**: Customers may have seen error messages when attempting to sign in, such as "Can't sign in", "Failed to determine sign in methods for email", or "Upstream connect error". * **Vault Access**: Some customers experienced degraded performance when accessing their 1Password vaults. * **Geographic Regions Affected:** USA/Global ### **What Happened?** The incident was part of ongoing improvements to the 1Password infrastructure and was not the result of a security incident. Customer data was not affected. **Timeline of Events \(UTC\):** * **17:12**: A planned, phased rollout of an architectural improvement to authentication systems begins. * **18:20**: Engineers monitoring the rollout begin to observe higher latency during sign-in for a small subset of accounts. * **20:17**: A rollback of the change is initiated. An error in the rollback configuration sends a high volume of traffic to the new code path, causing a database connection bottleneck. Engineers observing the deployment immediately observe service impact. * **20:21**: A corrective action is deployed to revert the system to its previous state before the misconfigured rollback. * **20:39**: While impact is still being observed, a failover from the primary database to a secondary database is initiated. This action has no effect. * **20:58**: A restart of the service that manages incoming traffic to our services is initiated to reset connections. * **21:13**: A rolling restart of the web application servers is initiated. * **21:20**: Service is fully restored for all customers. **Root Cause Analysis:** The root cause was an error in the configuration of an attempted rollback. This misconfiguration incorrectly routed a high volume of sign-in traffic through a new, slower code path, which created a bottleneck of connections to our primary database and made the web application unresponsive. ### **How Was It Resolved?** **Resolution Steps:** The issue was fully resolved through two key actions: 1. The rollback misconfiguration was identified and corrected, which stopped traffic from flowing to the problematic new code. 2. A rolling restart of the web application servers was performed to clear the backlog of stuck database connections. **Verification of Resolution:** Monitoring systems were closely observed for 30 minutes to ensure error rates returned to normal. ### **What We Are Doing to Prevent Future Incidents** We are working to implement the following improvements: * **Improve configuration testing**: We will improve testing procedures of configuration updates and their rollbacks prior to being pushed to production. * **Improve our deployment tooling**: We will add additional validation to our traffic management tools to prevent similar configuration errors. * **Review our incident response procedures**: We have updated the runbook used to respond to this type of incident with guidance that will enable faster recovery. * **Enhance our monitors**: We will add more specific alerts that will help us more quickly distinguish between different application tiers, allowing for faster diagnosis and time to resolution. ### **Next Steps and Communication** No action is required from our customers. 1Password applications are designed to be resilient, with local copies of vault data always available on customer devices, even without a connection to the 1Password service. If you are still experiencing issues, please contact our support team. We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding. Sincerely, The 1Password Team
Resolved
This incident has been resolved.
Resolved
# **What happened** On July 31, between approximately 12:30 AM and 2:00 AM EDT, some logged-out Vimeo users encountered an issue when visiting the Vimeo home page. Instead of the expected content, impacted users received a blank screen with a brief error message. # **What caused it** Corrupted memory on the server side caused some isolated web server processes to fail. As a result, requests for the logged-out Vimeo home page that relied on these processes resulted in the error message. # **How we responded** We became aware of the outage within 30 minutes of the initial event and immediately began investigating. Once we confirmed the issue, we applied a manual fix to restore normal functionality. # **What we’re doing to prevent a recurrence** We’re improving our internal reporting to alert us to similar conditions sooner. We’re also making sure that future errors of this type generate our standard HTTP 500 error page instead of a blank screen with a technical readout.
Resolved
The issue impacting the delivery of website assets via our CDN (cdn.prod.website-files.com) for users in Singapore has been resolved. We’ve verified the fix implemented by the third-party provider, and asset requests are now functioning as expected. If visitors are still unable to access Webflow-hosted sites, we recommend the following: - Restart your device. - Clear your browser and DNS caches. - If the issue persists, contact your local ISP and ask them to check their DNS configuration for cdn.prod.website-files.com. We appreciate your patience throughout this incident.
Resolved
Atlassian’s engineering team has implemented a temporary fix that has successfully addressed the API rate limiting behavior impacting Airtable–Jira syncs. With this intervention in place, sync operations are now functioning as expected. Our teams will continue collaborating closely to monitor performance and support Atlassian in delivering a long-term solution. We appreciate your patience as we worked to restore full functionality and ensure a stable integration experience. — The Airtable Team
Resolved
This incident has been resolved.
Resolved
The issue has been resolved and the Admin Console content search is now working as expected.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We are no longer seeing any errors with the Help Scout web app and are confident that we've resolved the issue. We're sorry for any trouble this has caused, and thank you for your patience!
Resolved
This issue was directly related to [https://www.fastlystatus.com/incident/377793](https://www.fastlystatus.com/incident/377793)
Resolved
This incident has been resolved.
Resolved
Between 13:51 UTC and 15:20 UTC, some customers in the US and EU region may have experienced errors sending golden signal data to New Relic, issues in viewing data in real time, and delayed or missing alert notifications related to real time data.
Resolved
# **Incident Postmortem - Intermittent SSO Availability** **Date of Incident:** 2025-07-14 **Time of Incident \(UTC\):** 22:00UTC - 11:23UTC **Service\(s\) Affected:** SSO **Impact Duration:** 1 hour and 23 minutes ## **Summary** On July 14, 2025 1Password’s SSO services were intermittently slow or unavailable for approximately 1 hour and 23 minutes. This was due to an outage with an upstream provider which 1Password backend services rely on to verify DNS lookups for SSO authentication. You can read the provider’s incident report [here](https://blog.cloudflare.com/cloudflare-1-1-1-1-incident-on-july-14-2025/), which outlines in detail what occurred and how they remediated the issue. For the duration of the incident 1Password customers would have experienced failed or slow SSO login attempts. We apologize for any inconvenience this issue may have caused our users. ## **Impact on customers** Customers accessing 1Password cloud services experienced delayed responses and timeout error messages. * **SSO Login:** Users logging in via SSO experienced failures, timeout errors or slow responses ## **Scope** * **Number of affected Customers \(approximate\):** Most users attempting to authenticate using SSO * **Geographic Regions Affected:** All ## **What Happened?** At 22:00UTC on July 14, 2025 1Password engineers were alerted to customer timeouts and failures when trying to use SSO as an authentication method. After diagnosing the issue, engineers discovered the source of the issue was an outage with an upstream provider when looking up and verifying OIDC provider hostnames. * **Timeline of Events \(UTC\):** * 22:00 - Incident initially detected by 1Password * 22:13 - Upstream provider identifies issue * 22:59 - Upstream provider implements a fix * 23:30 - Incident resolved * **Root Cause Analysis:** The third party provider 1Password uses for DNS lookup and verification failed. ## **How Was It Resolved?** * **Resolution Steps:** Our third party provider mitigated the issue. * **Verification of Resolution:** We closely monitored systems to ensure SSO logins were successful. ## **What We Are Doing to Prevent Future Incidents** * We are investigating the addition of an additional DNS provider to provide redundancy.
Resolved
We've continued to monitor this incident and can confirm it has now been resolved. We apologise for any inconvenience caused. If you are still facing issues with the service, please reach out to our support team here https://support.speechmatics.com/hc/en-gb.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The issue where emails were not syncing with G Suite has been resolved. The incident occurred from 12 pm PT on 8/5/2025 to 1:30 pm PT on 8/6/2025. There should be no further impact at this time, but please reach out to us at Support if any additional assistance is needed: https://help.lever.co/hc/en-us/requests/new
Resolved
We identified and resolved a Workflow UI issue that occurred between 3:57 AM and 4:29 AM PST. This may have also impacted other services like data query that tried to query on the workflow object.
Resolved
This incident has been resolved.
Resolved
The issue is resolved.
Resolved
Cloud Provider has confirmed that the issue has been resolved. No further service disruptions were observed since the last update.
Resolved
This incident has been resolved.
Resolved
Resolved