Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Sep 18, 2025

Sep 17, 2025

  1. PrismaAccelerate is unable to provision new query engines in eu-central-1

    Description: We identified an upstream infrastructure provider issue affecting the eu-central-1 region. Accelerate experienced issues provisioning resources for new query engine instances. Affected users: Users provisioning new Accelerate query engines in the eu-central-1 region. Root cause: Upstream infrastructure provider Workarounds: None identified

    Resolved

Sep 16, 2025

  1. FrontSome conversations cause error page

    Front has now completed the fix and cleanup associated with this incident. If you continue to see errors loading conversations, please make sure you've done a browser or app refresh, and if the issue persists please contact support. Summary: Conversations for which Front attempted to provide a Suggested Reply between 21:30 – 21:50 UTC may have been unreadable until the fix was completed at 22:49 UTC. During that initial time, suggested replies were generated with an additional field that caused an unrecoverable error in the frontend. No data or messages were lost or corrupted during the incident. Suggested replies for conversations that arrived during that time may need to be regenerated.

    Resolved

  2. AssemblyAIDegraded Streaming Performance

    We experienced an issue with our Streaming API between 09:00 GMT and 10:04 GMT. This resulted in reduced quality of service for streaming sessions and some sessions were closing unexpectedly. A fix was implemented and our Streaming API is now functioning normally.

    Resolved

  3. BitbucketDelays in running Bitbucket Pipelines

    Between 03:30 UTC to 03:50 UTC, we experienced delyas in running pipelines for Atlassian Bitbucket. The issue has been resolved and the service is operating normally.

    Resolved

Sep 15, 2025

  1. MezmoSearch performance issues

    The issues with Search in Live Tail have been resolved, and Search performance is back to normal.

    Resolved

  2. ConfluenceDegraded performance in Confluence and Jira for Microsoft Edge users

    An update has been released by Microsoft to resolve this issue for users of the Edge browser. To verify that you are running the required version of Edge, you can navigate to edge://components/ in the address bar, click the 'Check for update' button for the 'Trust Protections List', ensuring that you are on version 1.0.0.31 or higher to ensure you are running the version that contains the fixed release. All Edge browsers should otherwise receive this update within 10 hours without user intervention.

    Resolved

Sep 13, 2025

Sep 12, 2025

  1. MailgunSending errors in US

    All systems are operating normally at this time. Our engineers continue to monitor our systems for any recurrence or new issues.

    Resolved

Sep 11, 2025

  1. TrelloDegraded performance of Trello

    On Thursday, September 11, 2025 1:00PM UTC, Trello users may have experienced performance degradation. The issues has now been resolved, and the service is operating normally for all affected customers.

    Resolved

  2. VimeoHomepage is not loading

    # **What happened** On September 4, between approximately 10:00 AM and 11:00 AM EDT, some Vimeo users experienced partially rendered pages across the site. Affected pages displayed the top menu bar but were missing portions of the expected page content. # **What caused it** During a site update, some required files weren’t uploaded correctly. # **How we responded** We received internal notification of the issue almost as it occurred and began investigating immediately. We quickly confirmed the outage and reverted to a prior stable version of Vimeo. This restored all affected pages within one hour. We then investigated the root cause and implemented a permanent fix by roughly 1:30 PM. # **What we’re doing to prevent a recurrence** We changed the order in which Vimeo source code is uploaded and checked so that lagging updates won’t cause similar outages in the future.

    Resolved

Sep 10, 2025

  1. 1PasswordElevated API Errors

    # Performance Degradation **Date of Incident:** 2025-09-03 **Time of Incident \(UTC\):** 11:06 - 12:07 **Service\(s\) Affected:** All APIs **Impact Duration:** 61 minutes ## Summary For 61 minutes on the morning of September 3rd, 2025, all 1Password APIs in the US/Global environment had degraded performance or returned an error for approximately 20% of requests. 92% of the impact was mitigated within 13 minutes at 11:19 by automation scaling up infrastructure. By 12:06 a manual restart of the remaining infrastructure completed mitigation. A permanent fix was implemented and deployed to prevent the issue from reoccurring. ## Impact on Customers * **APIs:** High latency, or a 500 Internal Server Error. * **Number of Affected Customers:** 20% of all requests returned errors for 13 minutes, 1% thereafter. * **Geographic Regions Affected \(if applicable\):** 1Password USA/Global ## What Happened? * **Timeline of Events \(UTC\):** * 11:05: A customer started a stream of an unusually high volume of requests to an API with sub-optimal performance. * 11:06: Some servers started consuming abnormally high memory, causing slow response times and high error rates. * 11:19: Automation scaled up infrastructure to service additional load * 11:30: Increased errors trigger escalation, on-call engineer begins investigation * 11:51: Engineers declare an incident and alert response teams * 12:02: Response team begins restarting affected servers. * 12:07: All servers completed restarts, and error rates returned to normal levels * **Root Cause Analysis:** A poorly performing cache operation was triggered repeatedly in a short period of time across multiple servers, leading directly to greatly delayed responses. ## How Was It Resolved? * **Mitigation Steps:** Automatic instance scaling restored over 98% of operational capacity after 13 minutes. Full capacity was restored through manual intervention * **Resolution Steps:** We refactored the poorly performing query. * **Verification of Resolution:** We tested the affected API to confirm refactoring of query produced the desired performance improvement. We deployed the fix and monitored it for 24 hours to assert the issue was resolved. ## What We Are Doing to Prevent Future Incidents * We are auditing services for sub-optimal query performance. ## Next Steps and Communication * No action is required from our customers at this time. ‌ We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding. Sincerely, The 1Password Team

    Resolved

Older incidents