Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Sep 22, 2026

  1. HarnessDBDevOps Builds Failing intermittently

    ## Summary Between approximately **8:40 AM and 1:00 PM PDT on Wednesday, September 17, 2026**, some customers running **Database DevOps** pipelines in Production experienced intermittent failures when executing schema migration steps \(for example Apply Schema and related Database DevOps steps\). Affected runs commonly surfaced a misleading error: ‌ `Invalid request: Unable to fetch port details. Please verify pipeline yaml and check if 'Enable container based execution' toggle is on for the step group.` ‌ Failures were intermittent as the same pipeline could succeed some times. Other Harness modules and non–Database DevOps CD pipelines were not broadly impacted. ‌ ‌ ## Root Cause This issue required two separate problems to occur together. First, pipeline plan creation runs on a shared pool of background worker threads, and each thread can handle work for many different customer accounts over time. The identity used to authenticate outbound service calls was tied to the thread rather than to the specific request, so a thread could occasionally carry over the identity from a previous account's request instead of using the correct one for the current request. This happened inconsistently as the same pipeline could compile fine on one run and pick up the wrong identity on the next, which is why the issue appeared and disappeared without any customer-side change, and why retries usually worked. ‌ Second, and independently, a recent change to our Database DevOps service added a stricter check to make sure the identity on an incoming request matches the account it claims to represent. Before this change, a mismatched identity like the one above would have gone unnoticed and the request would still succeed. After this change, that same mismatch was correctly rejected, but the calling service treated that rejection as a hard failure rather than retrying, which is what turned an internal identity mix-up into a visible pipeline failure for customers. ‌ Neither problem caused issues on its own. The identity mix-up was harmless until the stricter check existed to catch it, and the stricter check only became noticeable because the identity mix-up was already happening in the background ‌ ## Impact * **Product area:** Database DevOps pipeline steps \(schema apply and related migration steps\) in Production. * **Customer experience:** Intermittent pipeline failures. Some executions succeeded while others failed within minutes of each other. The reported error pointed at container/port configuration and was misleading relative to the actual cause. Retries frequently succeeded. * **Scope:** Limited to customers running Database DevOps workloads on Production clusters where the stricter account validation was active. Other CD and platform functionality was not broadly disrupted, and no data loss or unintended schema changes resulted , affected executions failed rather than applying incorrect changes. ‌ ## Remediation * **Immediate:** Rolled back the Database DevOps service change that enforced the stricter account-binding validation in the most impacted Production environment, restoring Database DevOps pipeline execution for affected customers. * **Permanent:** Shipped a fix that explicitly sets the correct source principal and restores the previous thread context afterwards around every outbound call made by Database DevOps plan creators, including the plugin capability call. Outbound calls now always carry the account context of the pipeline being compiled, regardless of which worker thread runs it. * **Verification:** Following the rollout, service logs showed no further account-mismatch rejections across monitored clusters. ## Action Items ‌ ‌ To prevent such issues from happening again, Harness will,  * Enhance tests that compile plans for two accounts on a shared executor and assert each outbound call carries the correct account. * **Fail-fast on capability-check auth errors**: treat 403/auth failures as hard errors with an accurate message, instead of silently falling back to the legacy execution path. * **Alerting on account-binding rejections and plugin-capability 403 rate**: surface these signals before customer impact

    Resolved

  2. ClickHouse CloudWe are investigating an issue with CH servers in Azure germanywestcentral region

    The underlying AZ has recovered and we are fully back online. Since ClickHouse cloud architecture uses 3 AZs we were operational during this outage but there might have been some degradations. Azure support shared the details of the issue as following- Between 12:04 UTC and 17:46 UTC on 22 September 2022, a platform issue resulted in an impact to the Virtual Machines (VMs) and Virtual Machine Scale Sets (VMSS) in Germany West Central. Customers may have experienced their virtual machines becoming unresponsive, restarting unexpectedly, or being temporarily unavailable during the impact window.

    Resolved

  3. TrueLayerEUR Settlement Delays

    Settlement delays have now been resolved. All funds have now reached end-user accounts as expected. Regular processing timings have resumed. Impact start time: 22/09/2026, 10:26 UTC Impact end time: 22/09/2026, 13:50 UTC Client actions: None

    Resolved

  4. BufferBuffer Service Disruption

    The issue affecting Buffer has been fully resolved. Our team has confirmed that the service is now operating normally. We apologise for any disruption this may have caused and appreciate your patience throughout the resolution process. If you encounter any further issues, please let us know.

    Resolved

  5. BunnyIntermittent Login/API Timeouts in Poland

    Network performance issues caused by an upstream provider have been mitigated via traffic rerouting. All services and connection timeouts have returned to normal.

    Resolved

  6. OVHcloud - Public Cloud[GRA5/7/9/11][Public Cloud] - Some Instances incident notification

    We are pleased to inform you that the incident affecting our Public Cloud offering has been resolved. <b>Start time :</b> 22/09/2026 08:33 UTC <b>End time :</b> 22/09/2026 10:11 UTC <b>Root Cause :</b> This incident was caused by an unexpected infrastructure malfunction. We apologize for any inconvenience caused and appreciate your understanding.

    Resolved

  7. GitHubDisruption with some GitHub services

    On September 15, 2026, between 05:45 and 09:50 UTC, the Claude Fable 5.1 model in GitHub Copilot experienced intermittently degraded availability, with an average error rate of 2.8%. During brief recurring 15 minute periods that recurred every ~45 minutes, availability for Claude Fable 5.1 dropped to a maximum of ~40% before recovering completely. Other Copilot models were not affected. Users could continue to work with a different model or with 'Auto'.<br /><br />The cause was an issue with an upstream model provider that intermittently rejected requests while overloaded. GitHub worked with the provider, who acknowledged and then resolved the underlying issue at 9:50 UTC, after which the model returned to constant normal availability. Once recovery was guaranteed, we resolved the incident at 11:17 UTC.<br /><br />To reduce the chance of recurrence and customer impact, GitHub is reviewing per-model availability alerting and automatic in-product fallback so that requests to a degraded model can shift to a healthy alternative more quickly.

    Resolved

  8. SupabaseSupavisor connection disruptions in EU West 1 (Ireland)

    This incident has been resolved. Following the restart of the affected Supavisor node, error rates returned to normal and no further disruptions have been observed during the monitoring period. All services are operating normally.

    Resolved

  9. SumUpFPS transactions delayed or rejected

    The issue has been fixed and we’re confident it’s resolved SumUp services have returned to normal. You can stay up to date on the status of our services anytime here on our status page: https://status.sumup.com.

    Resolved

  10. GitHubDegradation with Gemini 3.8 Flash

    On September 16, 2026, between 04:40 and 11:45 UTC, the Gemini 3.8 Flash model in GitHub Copilot experienced degraded availability. Requests to this model failed at an average rate of 6.4%, and the impact was highest during peak traffic hours. Other Copilot models were not affected. Users could continue to work with a different model or with 'Auto'.<br /><br />The degradation was caused due to a capacity issue with an upstream model provider. Failure rates returned to normal as the daily traffic peak passed. We monitored the model until it was healthy and resolved the incident at 17:48 UTC. We are working to make our systems resilient to cover peak demand for all the Copilot models.

    Resolved

  11. AivenAiven Console and API are down

    This incident is now resolved. Earlier today a certificate problem affecting our public API endpoints caused sign-in failures and errors for customers using the Aiven Console, the Aiven API, the CLI, the Terraform provider and other API-based integrations. A corrected certificate has been in place across all API nodes since 02:05 UTC, and we have seen no further errors during our monitoring period. We are following up with a permanent fix to prevent this from recurring. We apologize for the disruption. If you are still seeing problems, please contact Aiven support at support@aiven.io: mailto:support@aiven.io

    Resolved

Older incidents