SparkPostOutbound message delivery is delayed/slow for some SparkPost customers (US hosted only)
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Resolved
All services should be functioning normally at this time.
Resolved
We've restored the ReverseETL delivery counts in the US
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The incident with our API has been resolved. If you are still having issues please reach out to our support via live chat or at support@upcloud.com. We apologize for the issues caused by this incident and appreciate your understanding.
Resolved
## Root Cause Analysis: Redis Cluster Startup Failures – US3 Cluster **Incident Date:** October 20, 2025 **Duration:** 18:36 UTC – 20:41 UTC **Status:** Resolved ## **Summary** On October 20, 2025, customers experienced a significant outage affecting the US3 cluster. The disruption began with increased latency and 503 errors before escalating into full service downtime as Redis clusters in the US3 region failed to start successfully during an infrastructure update. Service was fully restored at 20:41 UTC after engineers identified and resolved the underlying startup issue, confirming stability across all Redis clusters. ## **Impact** Between **18:36 UTC and 20:41 UTC**, customers experienced: * Major outage in the **US3** cluster, impacting message delivery and connection reliability * Elevated error rates and timeouts across dependent APIs * Temporary need for customers to switch to alternate clusters to maintain service continuity No customer data was lost. However, applications relying solely on the affected cluster experienced full downtime for a large portion of the incident. ## **Root Cause** The outage was caused by a failure of Redis clusters to start correctly following an infrastructure update due to a missing configuration flag upon startup. An unexpected upgrade to the Docker runtime running our Redis cluster introduced a breaking change that prevented container startup for certain Redis deployments. When replacement Redis instances in the US3 redis-main cluster were launched, they failed initialization checks and repeatedly restarted, rendering the cluster unavailable. The incompatibility remained undetected until a routine node replacement in the US3 cluster introduced new Redis instances to the cluster. ## **Detection and Response** Monitoring systems first detected increased error rates and latency at **18:36 UTC**, followed by a rise in 503 responses from the affected APIs. **Timeline of events:** * **18:36 UTC** – Increased latency and 503 errors observed in US3 cluster * **19:01 UTC** – Engineering began investigation into Redis startup failures * **19:04 UTC** – Incident declared a major outage; mitigation efforts initiated * **19:59 UTC** – Root cause identified and configuration fix applied * **20:20 UTC** – Services operational; monitoring for recovery stability * **20:41 UTC** – All Redis nodes confirmed healthy; incident resolved ## **Resolution** The engineering team: * Implemented a temporary fix to the Redis environments ensure Redis instances could initialize successfully * Blocked further automated replacements in other clusters until validated * Verified recovery and stability across all Redis clusters After deployment of the fix, Redis instances in the US3 redis-main cluster started correctly and full service was restored. ## **Preventative Actions** To prevent recurrence, the team has: * Rolled out a permanent fix across all Redis clusters used in all regions * Planned a long-term remediation to modernize Redis image packaging for compatibility with current and future Docker releases ## **Next Steps and Commitment** We are conducting a broader review of infrastructure upgrade processes to better detect runtime incompatibilities before they impact production. We apologize for the disruption this incident caused. Ensuring reliability and transparency remains our highest priority, and we continue to strengthen our processes to maintain consistent, predictable service for all Pusher customers.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
On November 3, 2025, some Confluence, Jira Service Management, and Jira Cloud users may have experienced performance degradation and errors in the Global Automation UI page. Automation executions were not impacted by this incident. The issue has now been resolved, and the service is operating normally for all affected customers.
Resolved
This incident has now been resolved. Thank you for your patience.
Resolved
This incident has been resolved.
Resolved
The incident has been resolved.
Resolved
The incident is now resolved. All delayed data for Facebook Integrations, which occurred between 1:00 AM and 2:00 AM, has been fully restored.
Resolved
The issue when sending money or paying a bill via international wire in non-USD currencies has been resolved.
Resolved
Customers should no longer see any impact in creating Anyscale Clouds.
Resolved
All cluster operations are working normally.
Resolved
This incident has been resolved. During the incident, users in EU experienced slow application performance, and automations, webhooks, and background actions were delayed.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Our third party had resolved their issue be we kept monitoring closely, we trust the stability now.
Resolved
This incident has been resolved.
Resolved
We have confirmed a transient issue with our cloud service provider from 6:41 UTC until 7:00 UTC for Mux Data collection in the EU. This issue has been resolved.
Resolved
This incident has been resolved.
Resolved
A large spike in republish activity, that had been misconfigured to run in the high priority queue, caused other publishing activity to stall. Attachment uploads were stuck 'processing' for the duration of the incident. The incident was identified at 4:05pm and resolved by 7:15pm.
Resolved
This incident has been resolved.
Resolved
We have resolved this incident. We’re sorry for the inconvenience.
Resolved
We have continued to see good performance and success rates for Slam-1 and are now marking this issue as resolved.
Resolved
From 4:58 PM EST to 5:10 PM EST, the cohere-rerank-3.5 model experienced partial unavailability.
Resolved
This incident has been resolved.
Resolved
Cluster creation is operating normally, and all delayed clusters have finished creation.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The upstream provider has reported full service recovery and Trelica is fully operational. This incident has been resolved.
Resolved
**Impact Summary** Between **October 22, 2025** and **October 28, 2025**, an issue with the Lever email-message-edito**r** component caused the email body to be unintentionally cleared in several workflows, including the interview scheduler and offer edito**r**. As a result, emails may have been sent without any body content, potentially leading to confusion or miscommunication with candidates and stakeholders. \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ **Root Cause** An update was introduced to support upcoming functionality for the Candidate Transparency feature. This change altered how the email-message-editor initialized its content: \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ **Resolution** * On **October 27, 2025**, the issue was identified and diagnosed. * A fix was implemented to restore the original behavior. * The fix was deployed to production on **October 28, 2025 at 17:11 ET**, restoring expected functionality. \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ **Preventative Actions** To prevent similar issues in the future, improved regression testing will be implemented for shared components like the email-message-editor.
Resolved