Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Jul 30, 2026

  1. LiveKitIntermittent webhook delivery failures

    ## Summary Between 2026-07-16 and 2026-07-29, a caching bug caused webhooks to be intermittently not delivered for about one hour for projects whose configuration was updated during this period. ## Root Cause A bug in an internal configuration service caused webhook settings to be dropped from a project's cached configuration whenever that project's record was updated. Once triggered, our system stopped delivering webhooks until the cache refreshed from the database, up to one hour later. Because of this caching behavior, the impact was intermittent: webhooks would stop, then resume on their own, then stop again after the next project update. Additionally, the "test webhook" feature in the dashboard reads webhook configuration through a different path and was unaffected, so test webhooks succeeded even while real event delivery was failing. ## Timeline \(UTC\) * 2026-07-16 23:48 - Change containing the bug reached production; intermittent impact begins * 2026-07-24 - Some customer reports of missing webhooks received; investigation begins * 2026-07-28 16:23 - Root cause identified; fix developed and rollout begins * 2026-07-29 16:26 - Fix deployed to all regions; incident resolved ## Scope of Impact Customers using webhooks were affected across all regions between 2026-07-16 and 2026-07-29, with most impact occurring after 2026-07-22. A project was affected only if its configuration was updated during the window, and each occurrence lasted up to one hour. Affected projects saw `room_started` and `participant_*` webhooks silently not delivered, while `room_finished` webhooks and dashboard webhook tests often continued to work. If you observed missing webhooks and had recently made any settings changes to your project, you were likely impacted. ## Mitigations and Follow-ups * The bug has been fixed and a regression test has been added. * We added logging of webhook delivery response codes to improve visibility into delivery failures. * We are adding per-project webhook delivery monitoring and alerting that can detect missing deliveries independent of overall volume. * We are unifying the configuration paths used by webhook delivery and the dashboard webhook test, so a passing test reflects actual delivery behavior. * We are making the configuration cache lifetime adjustable so similar issues can be mitigated quickly in the future.

    Resolved

  2. LambdaInvestigating network connectivity issues in us-east-2

    Severity Level: Medium The incident has been resolved. Network traffic is now working as expected. ----------------------------------------------------------------------------- Again, in the event you’re unsure as to whether your service is still impacted by this, please open a support ticket and we’ll get back to you as soon as possible. https://support.lambdalabs.com/hc/en-us/requests/new To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button.

    Resolved

  3. GitHubIncident with Actions

    On July 25, 2026, GitHub Actions experienced two related periods of degradation that caused some workflow runs to be delayed by more than 5 minutes or end with infrastructure failures. <br /><br />First period (08:45 – 09:13 UTC): During planned maintenance on a critical-path Redis cluster for Actions, one participating region was left in a degraded state. Separately, an independent capacity operation temporarily removed another region from the cluster and redirected its traffic to the degraded region. This created cross-region inconsistencies in job-assignment state, causing workflow runs to be delayed, exhaust retries, or fail outright. At peak, about 7% of runs were delayed by more than 5 minutes, and 25% of runs failed with an infrastructure error during the course of the incident. We mitigated the incident at 09:13 UTC by returning traffic to its normal distribution. <br /><br />Second period (12:08 – 12:48 UTC): As part of mitigating the first incident, traffic was returned to the regional instance that was still undergoing its capacity increase. Multiple Redis nodes in the scaling region experienced failures, increasing traffic to healthy nodes and causing connection limits to be reached on many nodes. At peak, 30% of runs were delayed by more than 5 minutes, and 60% of runs failed with an infrastructure error during the course of the incident. We mitigated the incident at 12:48 UTC by redirecting workflow traffic away from the scaling region. <br /><br />We are adding stronger regional health and capacity checks before maintenance and requiring a stable observation period before restoring traffic. We are also improving automated connection resiliency, and partnering with our platform dependency to automatically detect and remediate unhealthy cluster members and shard imbalance. More generally, we already had work underway to improve the resiliency and scale of this piece of Actions infrastructure.

    Resolved

  4. LambdaNetwork issues in us-east-2

    Severity Level: High The incident has been resolved. Networking is now working as expected. ----------------------------------------------------------------------------- Again, in the event you’re unsure as to whether your service is still impacted by this, please open a support ticket and we’ll get back to you as soon as possible. • https://support.lambdalabs.com/hc/en-us/requests/new To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button.

    Resolved

  5. BrexNotification delays

    We’ve resolved the issue affecting notifications and the Tasks page. All services are operating normally.

    Resolved

  6. SpeechmaticsRealtime SaaS US - Degraded performance

    We identified a subset of customer session failures in our Realtime SaaS - US region between 15:15 to 15:35 UTC. Our team has restored the service to normal performance levels. We apologise for any inconvenience.

    Resolved

  7. GitHubSeveral GPT models degraded

    On July 25, 2026, between 09:07 and 10:04 UTC, the GPT-5.2, GPT-5.3-Codex, GPT-5.4, GPT-5.4 Mini, GPT-5.6 Sol, GPT-5.6 Terra and GPT-5.6 Luna models experienced degraded availability in GitHub Copilot products and IDE surfaces. Requests to these models had an average failure rate of 5.6%. Other Copilot models remained available as alternatives.<br /><br />The degradation was caused by an issue with an upstream model provider. Success rates returned to normal after the upstream issue was mitigated, and we continued monitoring before resolving the incident. We are working on improving the automated failover for the affected models to prevent similar incidents in the future.

    Resolved

  8. HootsuiteExperiencing login issues with Mozilla Firefox & Safari

    We have resolved the issues related to the login flow when accessing the Hootsuite platform via Safari 26.6 and Mozilla Firefox version 153.01. If you continue to have issues logging into your Hootsuite dashboard via Safari or Mozilla Firefox please reach out to our customer support team. Thank you for your patience.

    Resolved

  9. TrueLayerPayout Failure Retries Degradation

    This incident has been resolved. Payout acceptance rates have returned to normal rates. What happened: A change deployed on 20/07/2026 at 09:43 UTC inadvertently altered our payout retry logic, causing some errors that should have been retried to be treated as terminal. This resulted in a temporary increase in payout failure rates. What we did: We identified the root cause, deployed a fix restoring the previous retry behaviour, and confirmed via monitoring that acceptance rates are back to baseline. We apologise for any inconvenience caused and are working on preventative measures to avoid a recurrence. Impact start time: 20/07/2026, 09:43 UTC Impact end time: 29/07/2026, 17:26 UTC Client actions: None

    Resolved

  10. BufferIssue Publishing Posts to Facebook

    Thank you for all your patience with us as we continued to monitor the fix that was put in place yesterday over that small number of Facebook posts that failed. We have fully resolved this incident now, and after monitoring for the past 24 hours, we're fully confident everything is back to normal. Apologies again for any inconvenience that may have been created through this temporary issue.

    Resolved

  11. TelnyxService Degradation in Australia

    Following a period of monitoring, connectivity and service performance have remained stable. Customers should no longer experience issues across affected services in the Australia region. We apologise for the disruption and appreciate your patience.

    Resolved

  12. DigitalOceanResponse Degradation Impacting Kimi-K3

    Our Engineering team has resolved the issue causing empty or broken responses from Kimi-K3. If you continue to experience any problems, please open a ticket with our Support team. We apologize for any inconvenience this may have caused.

    Resolved

Older incidents