Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Oct 11, 2025

Oct 10, 2025

  1. New RelicData Irregularities - US Region

    Between 13:21 UTC and 19:09 UTC, a subset of customers in the US region may have experienced delayed browser monitoring data for queries, charts, and dashboards as well as delayed or missing alert notifications for related data when alerts were not suppressed. We have resolved this interruption and services have returned to normal operations.

    Resolved

  2. UpstashConnectivity issue on us-east-1

    Between 15:46–15:55 UTC, some client connection attempts to databases in us-east-1 timed out due to unexpected high load on a server. The node was recovered at 15:50 UTC, and the updated DNS record propagated by 15:55 UTC. Services are operating normally.

    Resolved

Oct 9, 2025

  1. New RelicData Irregularities - US Region

    Between 16:20 UTC and 17:28 UTC on Tuesday, October 7th, 2025, some customers in the US region may have experienced delays in data appearing within the New Relic UI. Alert notifications may have also been delayed as a result. We have investigated and resolved this service interruption and impacted services have returned to normal operations.

    Resolved

  2. AblyAbly website, customer dashboard & Control API issues

    Customers may be experiencing connectivity issues affecting our website (ably.com), customer dashboards, the Control API (control.ably.net), the Ably blog and documentation. Our realtime services are not affected by this issue at this time.

    Resolved

  3. imgixIncrease in intermittent 503 errors

    Between 2025-10-08 23:50 UTC and 2025-10-09 01:50 UTC, a subset of origin fetch requests failed and returned errors. The issue has been identified and resolved.

    Resolved

Oct 8, 2025

  1. 1PasswordSome users are unable to interact with the admin console

    # Incident Postmortem - Some customers are unable to interact with the admin console **Date of Incident:** 2025-09-24 **Time of Incident \(UTC\):** 02:27 - 17:16 **Service\(s\) Affected:** Admin console, Sign in **Impact Duration:** 36:49 ## Summary Some customers with certain account configurations were placed on a blocklist and presented with a 403 error page after accessing the admin console. ## Impact on Customers * **Admin console:** Affected customers were presented with a 403 error page whenever they tried to interact with any of the admin console pages. * **Log in:** Affected customers were also unable to log in to the application. * **Number of Affected Customers \(approximate\):** 515 * **Geographic Regions Affected \(if applicable\):** All regions ## What Happened? * **Timeline of Events \(UTC\):** * Sep 24th 2:27am: Spike in application monitoring alerted engineers to increased rates of IP blocking * Sep 24th 3:00am: Cause identified as a change to requests in the application, which had been partially rolled out via a feature flag. * Sep 24th 4:03am: The feature flag was enabled to all customers which reduced the spike, but IP blocks continued throughout the day. * Sep 24th 10:03pm: Merged an application change to revert the change to prevent the issue reoccurring. * Sep 25th 5:03pm: The change was deployed with scheduled application release, error rate dropped off shortly after. * **Root Cause Analysis:** The issue was caused by GET requests to the Users API exceeding the URL length limit due to a recent change to append a list of UUIDs to the request parameters to resolve customer reported performance issues. * **Contributing factors:** * Requests were switched from GET to POST to prevent requests from exceeding the URL limit, however an issue with the feature flag configuration was causing UUIDs to be sent with the GET endpoint. * An underlying issue with the feature flag not resolving as expected in the application. ## How Was It Resolved? * **Mitigation Steps:** Customers were manually removed from the blocklist at multiple points in time as we evaluated the root cause and worked to patch the root issue. * **Resolution Steps:** The issue was mitigated by removing UUIDs at the API level if a GET request is used. Additional logging has been added to identify the root cause of the feature flag configuration issue. * **Verification of Resolution:** We monitored our server logs to ensure that we did not observe any additional GET requests to the affected URL. ## What We Are Doing to Prevent Future Incidents * **Audit additional admin console API requests:** We’re performing a sweep of admin console API requests to ensure the utilization of POST requests with highly parameterized URLs. * **Remove the feature flag misconfiguration:** We’re correcting the way the feature flag is configured to ensure consistent outcomes. ## Next Steps and Communication No action is required from our customers at this time. ‌ We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding. Sincerely, The 1Password Team

    Resolved

  2. ConfluenceIssues for some partners using Connect Javascript APIs

    On September 29-30 2025, for about 24 hours, Confluence Cloud experienced an incident due to the unintended deprecation of ACJS methods (https://developer.atlassian.com/cloud/confluence/about-the-connect-javascript-api/#connect-javascript-api) linked to retired V1 APIs (https://developer.atlassian.com/cloud/confluence/rest/v1/), resulting in 410 errors affecting partners reliant on these APIs. This issue is resolved, we will be working on a longer-term solution to prevent a similar incident from occurring again.

    Resolved

  3. CerebrasPartial Service Disruption

    A temporary network issue caused partial degradation for the Qwen-3-480B-Coder from 10:36 AM - 11:09 AM PT. The problem has been mitigated and normal performance has been restored.

    Resolved

  4. PrismaPrisma Postgres connection issues in US East region

    We identified and resolved an issue that caused new TCP connections from the US East region to fail, while existing connections continued to function normally. As a result, only a subset of users in the region were affected. The root cause has been identified, and a mitigation is currently in place while we implement a permanent fix.

    Resolved

  5. MailgunUS Email Ingest Errors

    Between 12:55 UTC and 13:07 UTC, customers may have experienced email sending failures in the US region. A deployment of new code caused our sending pipeline to temporarily reject some email submissions with HTTP 500 or SMTP 451 errors (~ 2.5% of requests). Our team quickly identified the issue, rolled back the deployment, and restored normal service. Traffic was immediately shifted from the affected US region to the non impacted region to further mitigate customer impact while the rollback took place.

    Resolved

Oct 7, 2025

  1. PlivoIntermittent Message failures in India Region

    Dear Valued Customer, We recently experienced intermittent SMS failures. Our engineering and operations teams promptly investigated and resolved the issue. The root cause was identified, and corrective actions have been implemented to prevent recurrence. Incident Start Time: 6th Oct 2025 14:30 UTC Incident End Time: 7th Oct 2025 03:30 UTC If you have any questions or concerns regarding this incident, please contact Plivo Support. Support portal: https://support.plivo.com/hc/en-us

    Resolved

Older incidents