Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Nov 4, 2025

Nov 3, 2025

  1. PusherUS3 cluster - major outage

    ## Root Cause Analysis: Redis Cluster Startup Failures – US3 Cluster **Incident Date:** October 20, 2025 **Duration:** 18:36 UTC – 20:41 UTC **Status:** Resolved ## **Summary** On October 20, 2025, customers experienced a significant outage affecting the US3 cluster. The disruption began with increased latency and 503 errors before escalating into full service downtime as Redis clusters in the US3 region failed to start successfully during an infrastructure update. Service was fully restored at 20:41 UTC after engineers identified and resolved the underlying startup issue, confirming stability across all Redis clusters. ## **Impact** Between **18:36 UTC and 20:41 UTC**, customers experienced: * Major outage in the **US3** cluster, impacting message delivery and connection reliability * Elevated error rates and timeouts across dependent APIs * Temporary need for customers to switch to alternate clusters to maintain service continuity No customer data was lost. However, applications relying solely on the affected cluster experienced full downtime for a large portion of the incident. ## **Root Cause** The outage was caused by a failure of Redis clusters to start correctly following an infrastructure update due to a missing configuration flag upon startup. An unexpected upgrade to the Docker runtime running our Redis cluster introduced a breaking change that prevented container startup for certain Redis deployments. When replacement Redis instances in the US3 redis-main cluster were launched, they failed initialization checks and repeatedly restarted, rendering the cluster unavailable. The incompatibility remained undetected until a routine node replacement in the US3 cluster introduced new Redis instances to the cluster. ## **Detection and Response** Monitoring systems first detected increased error rates and latency at **18:36 UTC**, followed by a rise in 503 responses from the affected APIs. **Timeline of events:** * **18:36 UTC** – Increased latency and 503 errors observed in US3 cluster * **19:01 UTC** – Engineering began investigation into Redis startup failures * **19:04 UTC** – Incident declared a major outage; mitigation efforts initiated * **19:59 UTC** – Root cause identified and configuration fix applied * **20:20 UTC** – Services operational; monitoring for recovery stability * **20:41 UTC** – All Redis nodes confirmed healthy; incident resolved ## **Resolution** The engineering team: * Implemented a temporary fix to the Redis environments ensure Redis instances could initialize successfully * Blocked further automated replacements in other clusters until validated * Verified recovery and stability across all Redis clusters After deployment of the fix, Redis instances in the US3 redis-main cluster started correctly and full service was restored. ## **Preventative Actions** To prevent recurrence, the team has: * Rolled out a permanent fix across all Redis clusters used in all regions * Planned a long-term remediation to modernize Redis image packaging for compatibility with current and future Docker releases ## **Next Steps and Commitment** We are conducting a broader review of infrastructure upgrade processes to better detect runtime incompatibilities before they impact production. We apologize for the disruption this incident caused. Ensuring reliability and transparency remains our highest priority, and we continue to strengthen our processes to maintain consistent, predictable service for all Pusher customers.

    Resolved

  2. ConfluenceDegraded performance and intermittent errors

    On November 3, 2025, some Confluence, Jira Service Management, and Jira Cloud users may have experienced performance degradation and errors in the Global Automation UI page. Automation executions were not impacted by this incident. The issue has now been resolved, and the service is operating normally for all affected customers.

    Resolved

Nov 2, 2025

Nov 1, 2025

Oct 31, 2025

  1. AsanaIssues in with Asana in EU region

    This incident has been resolved. During the incident, users in EU experienced slow application performance, and automations, webhooks, and background actions were delayed.

    Resolved

  2. Black Forest LabsGlobal Outage

    Our third party had resolved their issue be we kept monitoring closely, we trust the stability now.

    Resolved

  3. MuxEU Data ingest collection issues

    We have confirmed a transient issue with our cloud service provider from 6:41 UTC until 7:00 UTC for Mux Data collection in the EU. This issue has been resolved.

    Resolved

  4. GOV.UKWhitehall publishing activity significantly delayed

    A large spike in republish activity, that had been misconfigured to run in the high priority queue, caused other publishing activity to stall. Attachment uploads were stuck 'processing' for the duration of the incident. The incident was identified at 4:05pm and resolved by 7:15pm.

    Resolved

Oct 30, 2025

  1. LeverEmail drafts deleted when scheduling interview

    **Impact Summary**  Between **October 22, 2025** and **October 28, 2025**, an issue with the Lever email-message-edito**r** component caused the email body to be unintentionally cleared in several workflows, including the interview scheduler and offer edito**r**. As a result, emails may have been sent without any body content, potentially leading to confusion or miscommunication with candidates and stakeholders.   \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ **Root Cause**  An update was introduced to support upcoming functionality for the Candidate Transparency feature. This change altered how the email-message-editor initialized its content:   \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ **Resolution**  * On **October 27, 2025**, the issue was identified and diagnosed.  * A fix was implemented to restore the original behavior.  * The fix was deployed to production on **October 28, 2025 at 17:11 ET**, restoring expected functionality.   \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  **Preventative Actions**  To prevent similar issues in the future, improved regression testing will be implemented for shared components like the email-message-editor.

    Resolved

Older incidents