Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Sep 18, 2026

  1. VercelInvoice Generation Paused

    Invoice generation has been re-enabled. --- We've identified the issue and expect to resolve by end-of-day today. --- We've temporarily paused invoice generation. Invoices may arrive later than usual. Product availability and usage are unaffected, and no further action is required. We're working on an update to re-enable invoice generation and we'll provide another update when invoicing resumes.

    Resolved

  2. GitLabErrors Opening RepositoriesDegraded Performance

    The incident has been resolved following the completion of the rollback deployment across the Gitaly nodes. All nodes have returned to a stable state, repository access has been restored. Incident team is currently investigating the root cause.

    Resolved

Sep 17, 2026

  1. AmplitudeRealtime Data Pipeline Delay

    From approximately 11:30 AM PDT (18:30 UTC) to 2:14 PM PDT (21:14 UTC), Amplitude experienced delays affecting the realtime data pipeline. Some data may have been delayed before appearing in Amplitude in real time. The issue has been resolved. There is no data loss, and no customer action is needed.

    Resolved

  2. Cockroach LabsConnection attempts failing for multiple Basic tenants in GCP South America (southamerica-east1).

    This incident has been resolved. CockroachDB Cloud (Basic) clusters hosted in GCP South America (São Paulo / southamerica-east1) experienced network connectivity issues that have now been fully restored. Connectivity has returned to normal and all affected clusters are operating as expected. We apologize for any disruption this may have caused. If you continue to experience any issues, please contact support.

    Resolved

  3. KlaviyoFlow sending delays

    Flow message sending has fully recovered and is processing at normal speeds. Delayed messages were not lost and have since been delivered. This incident has been resolved.

    Resolved

  4. TrueLayerEdge System Degraded Performance

    Our edge system experienced a short period of degraded performance which may have caused for your requests to timeout. This is resolved. Impact start time: 17/09/2026, 18:05 BST Impact end time: 17/09/2026, 18:09 BST Client actions: None

    Resolved

  5. LiveKitCloud Dashboard, Analytics Services, and Billing API failures

    This incident has been resolved. Between 12:08 and 14:23 UTC, customers were unable to access the Cloud Dashboard and the Analytics Services APIs. The cause was a control plane failure in our last remaining legacy Kubernetes cluster, which took out a dependency of those services. Our user-facing services (RTC, SIP, and others) were unaffected, as they run redundantly across our modern clusters and have no dependency on the legacy cluster. We have no indication of data loss. Recovery took longer than we would like, as mitigation required working within the legacy cluster. Work was already underway to make the affected dependency redundant across our modern clusters, and that cluster is scheduled for decommissioning, which removes this class of failure. We will follow up with a postmortem.

    Resolved

  6. ElasticAutoOps node metrics temporarily unavailable in some regions

    The issue causing missing node-level metrics (CPU, memory, disk, thread pools) for recent time ranges in a subset of AutoOps regions has been fully resolved. Cluster health, shard, and deployment data were unaffected throughout, and no data was lost.

    Resolved

  7. WixRESOLVED: Multiple Issues with Wix Services in European Region

    On September 15th at 8:27 AM UTC, some users in Europe experienced errors when accessing [Wix.com](http://Wix.com), logging in, and viewing live sites, as well as issues using site features such as Bookings, Stores and Payments. All functionality was fully restored within approximately 30 minutes. The root cause was an infrastructure change intended to remove unused resources in one of our European data center regions. Those resources were still serving live traffic. We regularly decommission and clean up infrastructure that is no longer needed. In this case, the change was made in a region that had already been largely retired, and the resources targeted for removal were believed to be unused. That assumption was not fully verified before the change was applied, and configuration that was still routing European traffic was removed. Requests arriving in that region could not reach the services behind them, which led to errors across a number of Wix products. Our monitoring detected the errors within minutes, our on-call team was engaged immediately, but, because the change had been made in a region that was being retired, it was not an obvious candidate for causing live traffic impact, and identifying it as the cause took longer than it should have. Once identified, we mitigated by shifting European traffic away from the affected region and serving it from other active regions, and restored the removed configuration in parallel. Once service was fully restored, traffic was returned to the affected region without further disruption. Following this incident, we are introducing automated validation that verifies whether a network resource is still in use before it can be removed, blocking the change from being merged if it is. We are also strengthening review requirements for infrastructure changes, and improving high availability and automatic failover between regions, including a faster way to move traffic out of a region when needed, so that similar issues can be contained before it affects users.

    Resolved

  8. LiveKitInvestigating reports of intermittently increased API latency impacting Egress, Ingress, and SIP

    ### Summary On 15 September, LiveKit Cloud experienced three periods of degraded performance between 13:34–13:41, 16:06–16:11 and 21:10–21:22 UTC. Ingress creation was the most affected: more than half of `CreateIngress` requests failed at the worst point. A number of in-progress recordings ended prematurely, and a small percentage of inbound SIP calls failed during a one-minute period \(during each impact window\). Realtime connections were not affected and media already flowing continued normally. The cause was a dropped connection to our primary metadata database that neither the client nor server side detected, which left a transaction open and holding row locks for up to 17 minutes. Requests queued behind those locks, and the sudden release of that backlog — not the wait itself — is what briefly degraded the wider platform. ### Root Cause Our services share a distributed global metadata database that stores ingress and egress state. At the time of the incident, a connection between one of our services and that database was dropped in a way neither end observed: the client treated the connection as closed and returned an error, while the database continued to consider the session live. Because the client had a transaction open at the time, the database kept that transaction's row locks held. The impact then came in two distinct phases. **While the lock was held**, only requests that needed the same rows were affected. Ingress creation failed, and recordings that could not report their status ended early. Most other APIs were unaffected, because they never touched the locked rows. **When the lock was released**, several hundred transactions that had queued behind it — some waiting more than seventeen minutes — all executed within a fraction of a second. That burst briefly saturated the database and degraded it for every service using it, not just those touching the original rows. This is why the broadest impact, including room management APIs and SIP participant creation, appears at the very end of each window rather than during it. The database only reclaimed the abandoned session when the operating system's TCP timeout expired, roughly 17 minutes after the connection was dropped. Two design choices amplified a single stuck row into regional impact: * **Recording workers treated "cannot report status" as "cannot accept work."** Every worker in a region reports through one shared service, so when that service became slow, all workers in the region stopped accepting new work at the same time and our router saw no available capacity. * **Several internal queries scanned the affected table without a narrowing filter.** That meant one locked row could block reads that were otherwise unrelated to it, which is what allowed the backlog to grow large enough to be disruptive on release. ### Scope of Impact Ingress creation was the most affected API: at the worst point in each window, 50.3%, 62.4% and 58.1% of `CreateIngress` requests returned server errors. Many failed requests are retried automatically, so the share of ingresses that ultimately could not be created is lower than those figures suggest. Starting a new egress was largely unaffected, staying under 2% throughout. SIP was affected at the end of each window, with `CreateSIPParticipant` reaching 7.50% and between 1.3% and 2.2% of attempted inbound calls failing during call setup; calls already connected were unaffected. Room management APIs stayed below 0.5%. Realtime connections were not affected, and media already flowing continued normally. The more consequential impact was to recordings already in progress. Up to 16.5% of egresses started during an affected window ended prematurely — 0.39% of all egresses that day — and did so without surfacing an error to indicate the recording had failed. ### Mitigations and Follow-ups Completed: * We have set a database-side idle transaction timeout so an abandoned transaction can no longer hold locks for more than 10 seconds. This caps both the wait and the size of any backlog that can accumulate behind it. We have verified this against a reproduction of the original failure. Underway: * We are removing an unnecessary transaction wrapper around single-statement writes, which shrinks the window in which a dropped connection can leave locks held. * We are adding database-side statement and lock timeouts so that no query can wait indefinitely on a lock. * We are changing recording workers so that a single status-reporting failure no longer removes an entire regional fleet from service. * We are making egress startup retry transient database errors rather than aborting the job. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team.

    Resolved

Older incidents