Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Oct 3, 2026

  1. HarnessIntermittent External Network Connectivity Issues Affecting Build VMs

    ## Summary CI runners in the us-west1 and us-central1 regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services such as GitHub and Bitbucket over outbound network gateways. ## Impact * CI runners in the affected regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services \(e.g., GitHub, Bitbucket\) over our outbound network gateways. * The issue was intermittent rather than constant — connections succeeded under normal load, and failures clustered during periods of high outbound traffic volume. * No data was lost or corrupted. This was a network-connectivity and capacity issue, not a data-integrity issue. * us-west1 and us-central1 were the affected regions; other regions were not impacted by this issue. ## Root Cause ‌ Our load balancer distributes outbound traffic across multiple NAT gateways using a hashing method based on connection details \(source/destination address and port\). For any single connection, these details stay constant for that connection's lifetime. We had a sustainted traffic surge for a few seconds which congested the gateways ‌ ## Action Items To prevent such issues from happening again Harness will, Increase outbound connection capacity on our NAT gateways by provisioning additional external network interfaces, giving each gateway a substantially larger pool of connections it can serve concurrently..

    Resolved

Oct 2, 2026

  1. SnowflakeINC20000237

    Snowflake Engineering has completed the postmortem of this service incident. A detailed Root Cause Analysis \(RCA\) is available on the Snowflake Community site: [https://community.snowflake.com/s/article/INC20000237](https://community.snowflake.com/s/article/INC20000237)

    Resolved

  2. HubSpotHubSpot is experiencing issues

    Between 04:32 PM EDT (UTC -04:00) and 07:21 PM EDT (UTC -04:00), some customers in North America experienced issues with customer platform tools, marketing tools, sales tools. This was caused by a bad production configuration change. As of 07:21 PM EDT (UTC -04:00), our customer platform tools, marketing tools, sales tools are working properly and the incident has been fully resolved. HubSpot conducts a thorough review after each incident to understand the cause and prevent it from happening again. Learn more about HubSpot's commitment to reliability at www.HubSpot.com/reliability. The information on this page reflects our understanding of the incident and impact at the time of the update.

    Resolved

  3. HarnessCI Linux Cloud VM Networking issues - intermittent

    ### Summary Between approximately **11:37 AM and 3:01 PM PDT \(18:37–22:01 UTC\) on September 29, 2026**, customers using Harness CI self-hosted Linux cloud runners in **prod1/prod2 \(us-west1\)** experienced intermittent networking failures during pipeline execution — primarily Git/network timeouts when fetching source code. **Root Cause** This was primarily due to saturation of our NAT which was futher exacerbated by a temporary cloud provider compute capacity, preventing new VM provisioning there. ### Impact * Customers running self-hosted Linux CI builds in us-west1 saw **intermittent build/pipeline failures**, not a full outage. * Observed customer-facing errors included Git fetch timeouts, e.g. `Failed to connect to github.com port 443: Connection timed out` **Remediation** * **Immediate:** Rebalanced ARM build-VM load from the saturated us-west1 path toward central1 \(partially limited by central1 capacity\); restarted the affected Cloud NAT gateways, which stabilized pipeline networking. **Action Items** 1. Secure additional NAT gateway/IP capacity \(In progress\) 2. Improve NAT saturation detection and alerting ahead of customer impact. 3. Evaluate cross-region capacity contingencies so a single region's compute shortage doesn't block failover mitigation.

    Resolved

  4. BrexNotifications Partial Outage

    A transient issue caused some notifications to not be sent between 18:18–18:54 UTC. This includes MFA and 3DS challenges. The issue has been resolved and we are now sending notifications correctly.

    Resolved

  5. CloudflareNetwork Performance Issues - Seoul (ICN)

    Between 17:11 and 17:22 UTC, Cloudflare experienced network performance degradation impacting Seoul, South Korea (ICN). The issue has been identified and fully mitigated, and network performance has returned to normal levels.

    Resolved

  6. HarnessInvestigating CI Pipeline Delays

    # Summary On September 22, 2026, three of our outbound network gateways in one region became unhealthy and began rejecting a small percentage of new outbound connection attempts. The issue was resolved the same day problem completely with no further recurrence or follow-on impact.The underlying cause was a saturated internal traffic-monitoring buffer on those gateways, which caused their health checks to respond too slowly and get marked unhealthy.  # Impact * Only three gateway instances in one region were affected; other regions and other gateways in the same region were not impacted. * There was no cascading failure , the issue was fully resolved with a single round of gateway restarts, and no follow-on issues occurred afterward. # Root Cause  Our outbound network gateways run an internal component that monitors network traffic for operational visibility. Under sustained high traffic, this component's internal buffer became saturated, which created backpressure severe enough to slow down the gateway's own health-check responses beyond the timeout our load balancer allows. Once that timeout was exceeded, the load balancer marked the affected gateways unhealthy and began rejecting a portion of new connections destined for them. Actual packet forwarding continued to work normally throughout the incident. This was specifically a health-check and monitoring-buffer issue, not a failure of the gateways' core forwarding capability. # Resolution The three affected gateway instances were manually restarted, which cleared the saturated monitoring buffer and immediately restored normal health-check behavior. Connection rejections stopped as soon as the restart completed, and no further intervention was needed. # Preventive Actions | **Action** | | --- | | Replace the current traffic-monitoring approach on all gateways with a more efficient, lower-overhead implementation that cannot saturate in this way, and that alerts well before any customer-visible impact. | | Implement automatic, periodic replacement of long-running gateway instances on a fixed schedule, so no instance can accumulate the kind of age-related degradation seen in this incident. | | Migrate gateway instances to network-optimized infrastructure with substantially more dedicated network capacity, better suited to sustained high-throughput traffic. | | Enhance existing monitoring buffer's utilization while the longer-term replacement is rolled out. | | Enhance alerting specifically on gateway health-check response latency, to catch early signs of degradation before it becomes customer-visible. |

    Resolved

  7. HarnessProd2: CV is failing with errors for all customers

    **Summary** Customers on Prod-2 experienced elevated failure rates in Continuous Verification pipelines. Verification jobs failed to complete or reported errors because CV data-collection workers could not communicate with the platform service that coordinates their work. This was caused by an unexpected traffic spike to one of our backend systems.  Service was restored by scaling out the affected platform component. No data loss or corruption occurred. ‌ **Customer Impact** Customers using Continuous Verification on Prod-2 saw verification pipeline jobs fail or time out during the incident window. A significant fraction of verification job instances that ran during this period ended in an operational failure. Pipelines not using CV continued to operate normally. ‌ **Root Cause** A large automated provisioning run created a significant number of persistent CV monitoring workers in a short window, which overwhelmed a shared platform component. The sudden increase in concurrent outbound calls exhausted the available network connections on each service instance.   ‌ **Mitigation** The maximum scaling limit for the CV service was raised, allowing it to expand capacity to meet demand. Failures stopped shortly after the scale-out was completed. ‌ **Next Steps** To prevent recurrence, Harness will: * **Increase capacity headroom and add provisioning guardrails:** Raise the CV service scaling limit to maintain headroom under burst loads, and introduce rate limiting for bulk provisioning of monitored services. * **Improve Resiliency:** Add exponential backoff and jitter for worker retries, and use a reusable connection pool to reduce overhead and prevent transient failures from compounding under high concurrency. * **Improve monitoring and alerting:** Enhance alerting for abnormal per-account worker growth rates, so similar conditions are detected and acted on earlier.

    Resolved

  8. LinodeService Issue - Object Storage - US-Southeast (Atlanta)

    We haven’t observed any additional issues with the Object Storage service, and will now consider this incident resolved. If you continue to experience problems, please <a href="https://cloud.linode.com/support/tickets">open a Support ticket</a> for assistance.

    Resolved

  9. AmplitudeAmplitude- Platform Loading Issues

    The issue has been resolved, and the platform is operating normally again. Our Engineering team will continue to monitor the platform closely to ensure everything remains stable. Thank you for your patience and understanding while we worked to resolve this issue.

    Resolved

  10. AkamaiEdge Delivery Issues

    We can confirm that the issue was mitigated at 14:45 UTC on October 2, 2026, and the service has resumed normal operation. Customers and partners can view additional details about the incident by logging in to: https://community.akamai.com/customers/s/feed/0D5a700001WrkrnCAB or reaching out to Akamai Support. We apologize for the impact and thank you for your patience and continued support. We are committed to making continuous improvements to make our systems better and prevent a recurrence of this issue.

    Resolved

  11. SinchCalling Degradation - PSTN

    We want to inform you that we experienced a performance degradation impacting our Calling service. Incident Start Time: 2026-10-02 05:51 UTC Incident End Time: 2026-10-02 09:05 UTC Impact: Customers may have experienced failed attempts when making PSTN outbound calls. We sincerely apologize for any inconvenience this may have caused and appreciate your understanding while we worked to resolve the issue. For further information or assistance, please contact us at support@sinch.com.

    Resolved

Older incidents