Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Jun 28, 2026

Jun 27, 2026

  1. GitHubDisruption with some GitHub services

    This incident was used to notify for a maintenance event. There is no specific root cause analysis. Work progressed as planned without any issues to report.

    Resolved

  2. DigitalOceanDroplet Backup Service

    Between 00:00 UTC & 16:00 UTC today, our Engineering team identified an issue affecting backup operations on Droplets. During this period, backups scheduled within this window may not have been created and may appear as missing. Our team has taken necessary measures to resolve the issue, and we can confirm that the backup service has been restored and is now functioning normally. Upcoming scheduled backups should be performed as expected. We sincerely apologize for any inconvenience this may have caused and appreciate your understanding. However, if you have any further questions or concerns, please create a support ticket for further analysis.

    Resolved

  3. SumUpSome payments failing

    The issue has been fixed and we’re confident it’s resolved SumUp services have returned to normal. You can stay up to date on the status of our services anytime here on our status page: https://status.sumup.com.

    Resolved

  4. TrueLayerPLN Payout Settlement Delays

    We experienced a service degradation for PLN Payouts due to an issue with our banking partner. Clients may have seen delayed settlement of payments. The issue has been resolved, and payments are settling as normal. Impact start time: 26/06/2026, 19:05 UTC Impact end time: 27/06/2026, 09:50 UTC Client actions: None

    Resolved

  5. DigitalOceanAnthropic Inference Model Availability

    Our Engineering team has resolved the issue that was causing HTTP 400 errors for requests to Anthropic models. Users should now be able to access Anthropic models without any issues. If you continue to experience problems, please open a ticket with our Support team so we can investigate further. We apologize for any inconvenience this may have caused.

    Resolved

  6. CircleCIDelays starting Docker jobs

    ## Summary On Tuesday, June 23, 2026, our engineering team identified that a single Docker job cluster was operating below expected performance levels. Our infrastructure is organized into routing groups, each responsible for handling a specific segment of traffic, and this cluster was the sole member of its group. To address the performance issue, they initiated a standard two-step procedure to safely take it out of rotation: first, update the routing configuration to stop directing traffic to the cluster, then take it offline. When the team moved to take the cluster offline, the first step of the procedure was skipped. The cluster was removed from the available pool, but the routing configuration was not updated, meaning traffic continued to be directed to the routing group even though it had no capacity to process work. With no other clusters in the routing group to absorb the traffic, affected jobs had nowhere to go and began queuing rather than starting. This is a routine and safe operation when the two-step procedure is followed correctly, this incident exposed a gap in our tooling that allowed the procedure to be completed out of order. The issue began at approximately 17:25 UTC and was fully resolved by 19:46 UTC, for a total impact window of roughly 2 hours and 20 minutes. Only Docker jobs were affected; other job types, including machine and macOS jobs, continued to operate normally. Beyond the missed step, several factors extended the impact window. An automated alert fired within a minute of the change deploying, but was dismissed as expected behavior. This delayed detection by nearly an hour. When engineers attempted to accelerate recovery by restoring a prior known-good version, they encountered unexpected issues with the rollback process, adding further delay before the fix could be deployed. The original status page for this incident can be found [here](https://status.circleci.com/incidents/rp9y8kszn25j). ## How Job Routing Works CircleCI uses Nomad as its job scheduling system for Docker-based jobs. When a job is triggered, it is routed to one of several Nomad clusters, organized into routing groups each responsible for handling a specific segment of traffic. This multi-cluster architecture is intentional: clusters are isolated from one another so that an issue in one does not impact the others, limiting blast radius and improving overall resiliency. Under normal circumstances, if a cluster becomes unavailable, traffic is automatically rerouted to the remaining operational clusters in its routing group, allowing jobs to continue without interruption. ## What Happened \(All times UTC\) Prior to the incident, our engineering team identified that a single Docker job cluster was operating below expected performance levels. Our infrastructure is organized into routing groups, each responsible for handling a specific segment of traffic, and this cluster was the sole member of its group. To address the performance issue, they initiated a standard two-step procedure to safely take it out of rotation. At 16:58, our engineering team applied a configuration change to remove the affected cluster from service. The first step of the two-step procedure was missed. The cluster was removed from the available pool, but the routing configuration was not updated, leaving traffic still being directed to a routing group with no capacity to process it. When the configuration change was fully deployed at 17:25, the affected jobs began to queue. At 17:26, our monitoring alerted us to reduced job throughput on the affected cluster. Because throughput on that cluster was expected to drop as part of the cluster rotation procedure, the alert was interpreted as expected behavior and no action was taken. By 18:19, a series of customer support tickets made clear that something beyond expected behavior was occurring. A formal incident was declared at 18:28 and engineers began investigating. By 18:33, the team confirmed the impact was isolated to a subset of Docker jobs and identified a large backlog of queued jobs. At 18:41, a fix was merged to restore the affected cluster back into the available pool. Deployment of that fix was delayed because the CI smoke tests assume functioning clusters, leading to some rework of the pipeline itself. In parallel, the team attempted to accelerate recovery at 18:54 by restoring a prior known-good configuration directly, but encountered a conflict that left multiple active versions running simultaneously, preventing a clean rollback. The fix was fully deployed at 19:20 and Docker jobs began processing again. By 19:28, the team confirmed the backlog was actively clearing and newly submitted jobs were starting at normal times. Customers who had accumulated queued jobs during the incident period may have experienced a surge in concurrent job starts upon recovery, potentially hitting plan concurrency limits, which would have resulted in jobs still queueing while we processed the backlog. In total, the period during which Docker jobs were unable to start lasted from approximately 17:25 to 19:20, roughly 1 hour and 55 minutes. The additional time from 19:20 to 19:46 was spent processing the accumulated backlog, during which some previously queued jobs completed with elevated wait times. ## Future Prevention and Process Improvement We are taking the following steps to prevent a recurrence and improve our response time: **We are strengthening our failover architecture.** Our multi-cluster design is built for resiliency, and while automatic failover mechanisms are already in place, this incident exposed a gap in our safeguards. We are closing that gap to ensure traffic is rerouted to healthy clusters across routing groups regardless of whether a cluster goes down unexpectedly or is intentionally taken offline. **We are enhancing our cluster rotation procedure.** This incident was possible because our tooling allowed an invalid state to exist, a cluster removed from the available pool while the routing configuration still directed traffic to it. We are updating this tooling to enforce that routing configuration and cluster availability are always updated together, and that the procedure cannot be completed out of order. **We are improving our monitoring and alerting.** The automated alert that fired during this incident was dismissed because it appeared consistent with the operation being performed. We are improving our monitoring so that unexpected side effects are surfaced clearly and are not confused with expected behavior. **We are improving our rollback capabilities.** During the incident, our team encountered delays when attempting to deploy the fix. We are auditing our deployment configurations to ensure that rollbacks can be executed more quickly and reliably during incidents. Reliability is our top priority, and we are committed to ensuring that infrastructure upgrades like this one are executed with the appropriate safeguards in place.

    Resolved

Jun 26, 2026

  1. AkamaiDataStream Configuration Issues

    We can confirm that the issue was mitigated at 20:55 UTC on 25 June 2026 and the service has resumed normal operation. Customers and partners can view additional details about the incident by logging in to: https://community.akamai.com/customers/s/feed/0D5a700001FiyHGCAZ, or reaching out to Akamai Support. We apologize for the impact and thank you for your patience and continued support. We are committed to making continuous improvements to make our systems better and prevent a recurrence of this issue.

    Resolved

  2. GitHubWe are seeing elevated errors with Next Edit Suggestions and Completions

    On June 23, 2026, between 22:45 and 23:29 UTC, GitHub Copilot Completions and Next Edit Suggestions were degraded for users in all regions. During this window, affected users may have seen failed or missing code completions and Next Edit Suggestions. On average about 25% of Completions and Next Edit Suggestions requests failed during the impact window, peaking at roughly 27%. The cause was a configuration change that prevented the Copilot service from obtaining the authentication tokens it needs to reach its model backends; this both failed requests directly and caused the service to temporarily remove backends from rotation. GitHub engineers detected the elevated error rate within minutes, declared an incident, and mitigated the issue at 23:22 UTC by redeploying the service with a known-good configuration, which restored normal operation. As a follow-up, the team disabled the affected authentication path to prevent a future deployment from re-introducing the problem, and is making the change rollout safer. We apologize for the disruption and are taking steps to reduce the likelihood of similar incidents.

    Resolved

  3. TrelloTrello - degraded performance

    # Summary On May 17th 2026 at 12:00 UTC a subset of Trello users experienced a degradation of some core functionality including but not limited to: issues with comments, viewing activity logs, accessing card details, board exports, Power-Ups, attachments, Home page updates, and search functionality. Some moved cards displayed 'Card Not Found' errors. These issues persisted for multiple days. Most of the issues encountered when interacting with Trello (commenting, viewing activity) were resolved by May 19 2026 at 08:31 UTC, however there was a lingering data integrity issue which manifested as “Card Not Found” errors when trying to open cards. >90% of these cards were repaired by May 28 at 17:20 UTC. The remainder took a little more time and were completely repaired by June 5th 2026 at 21:37 UTC. We wanted to share a more detailed analysis of the issues we encountered, as well as what we’re doing to ensure nothing similar happens in the future. # Root Cause The incident was caused by a bug in our database (MongoDB) which surfaced during a project to optimize the performance and scale of one of our largest collections. The affected collection contains card comments and other activity and is used across much of Trello’s functionality. The optimization project was to reshard the collection onto a new key. This allows us to eliminate a so-called “insert hot spot” which is where all newly created database entries end up on a single database partition (aka shard). After resharding, the load would be spread evenly across the various servers that make up the database cluster. This decreases latency and improves query performance for users. During the final stages of the resharding operation (which is a long-running, automated process), 12 of the 30 relevant shards ran out of disk space while building indexes. Rather than aborting the resharding process, the database bug caused the cluster to put the new database servers into rotation without their indexes, despite logs and metrics saying otherwise. Due to the enormous amount of data in this collection, having no indexes means that there is essentially no way to query for the data you want (i.e. “give me all comments for this card”) without performing an extremely slow and inefficient collection scan. # Remedial Actions Plan The first thing we did was initiate a full rebuild of all the database indexes for the collection. Unfortunately this takes a very long time to complete (multiple days) and is compute-intensive. We scaled all the database servers up in order to help the process move faster, however due to the single-threaded nature of this process, more compute did not equate to more speed. Additionally we enabled a database setting which caused all queries that would result in a full collection scan to fail-fast rather than drag on in the background and consume database resources before ultimately failing. This helped the parts of Trello that were still working stay responsive. Once the reindex process had completed, all functionality was restored. However, we discovered that cards which were moved to a different board during the rebuild were left in a broken state. Their activity logs and comments were essentially stranded on the old board while the card was now on the new board. This manifested as a “Card Not Found” error when trying to open such a card. The team moved into a new phase of writing code to safely remediate all of the remaining data issues in batches, with the final batch of fixes completing by June 5th 2026 at 21:37 UTC. # Next Steps We know that Trello is where your work lives and issues like this can cause a lot of disruption. To ensure we don’t run into this issue again in the future, the team is taking the following actions: - Working closely with our database vendor to ensure the original bug is fixed (update: the fix is shipping in the next version) - Working to create a standard set of tools and processes to ensure index rebuilds proceed as fast as possible in an emergency like this - Designing and implementing changes to make card move operations more atomic (either the whole operation succeeds, or the whole operation fails, rather than a partial failure)

    Resolved

  4. ExpoBuild and Workflows pages on expo.dev were not scrollable

    Between 9:04am PDT and 10:56am PDT, users experienced scrolling issues on the Build and Workflows pages of expo.dev. This was caused by a CSS issue introduced in a recent deployment. We rolled back the deployment and verified the issue is resolved. We apologize for the inconvenience.

    Resolved

  5. MuxLive Stream Disconnects

    No further issues observed during monitoring period. Live streams are operational.

    Resolved

  6. GitHubIncident with Webhooks

    On June 11, 2026, between 19:28 UTC and 21:06 UTC, GitHub webhook deliveries were delayed. Average delivery latency peaked at approximately 3.4 minutes, with some deliveries delayed by as much as 62 minutes at the 99th percentile. No events were lost — delayed events were queued and delivered once processing caught up.<br /><br />This was due to a change in how webhook traffic was distributed across regions: to relieve load on one region, a portion of processing was shifted to another, where higher latency prevented our delivery workers from keeping pace with incoming volume, creating a backlog. We mitigated the incident by rebalancing webhook traffic distribution; as load returned to normal levels, processing caught up and the delivery backlog fully drained.<br /><br />We are working on improving the accuracy of the utilization metrics used to scale our delivery worker pools, and reassess how we distribute webhook traffic across regions, to reduce our time to detection and mitigation of issues like this one in the future.

    Resolved

  7. LinodeEmerging Service Issue - GPU Instances - All Regions

    On June 23, 2026, following a global software deployment, we identified an issue causing intermittent boot failures specifically for GPU Linodes. The impact was limited to instances where multiple GPU Linodes attempted to boot simultaneously. During the impact window customers could have experienced localized disruption or elevated error rates, particularly during automated node recycling or scaling events. During the investigation it was found that a recent software update created a conflict when multiple GPU servers tried to start up at the exact same time. The servers essentially blocked one another from loading, and our system did not automatically trigger a retry. This specific interaction only happens under heavy, simultaneous workloads, which is why it wasn't caught during our standard pre-release testing. In order to mitigate the issue, we successfully deployed a hotfix directly to all active GPU hosts across our fleet at 20:32 UTC on June 24th, 2026. The affected systems are currently operating normally as expected. In order to prevent this issue from happening in the future, we have developed and integrated a comprehensive fix into our upcoming software release scheduled to roll out globally over the next week. Additionally, we are actively prioritizing the procurement of dedicated GPU testing hardware for our development cloud to improve test coverage and ensure concurrent hardware workloads are fully simulated before future updates reach production. This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing and any information herein is subject to change.

    Resolved

  8. ZapierFacebook Lead Ads intermittently missing leads

    This incident has now been resolved, and all services are operating normally. We’re working with Meta to confirm whether any missed leads will be resent or if further customer action is needed. We’ll share an update once we have more information. Update: We’ve been advised that missed lead notifications from the affected period won’t be resent or backfilled. We understand this may be disruptive for affected customers. To recover missed leads, customers can export a CSV or Excel file from Meta Ads Manager, import it into a Zapier Table, and use Transfer Data to send those leads through their Zap workflow: • Meta export instructions: https://www.facebook.com/business/help/734933888443065 • Zapier Transfer Data instructions: https://help.zapier.com/hc/en-us/articles/8496274335885-Transfer-existing-data-using-a-Zap If you need assistance, or continue to experience any further issues, please reach out to our Support team: https://zapier.com/app/get-help

    Resolved

  9. CircleCIXcode 26.6 Demand + Queueing

    We observed a surge in demand for Xcode 26.6 and have since mitigated the incident by increasing the available capacity for that Xcode version. Customers may have seen jobs queueing for ~10-15 mins.

    Resolved

  10. SlackChat functionality unavailable on Slack Help page

    We've resolved the issue and users can now use the Chat feature on the Slack Help Center. We performed a configuration change, which brought the feature back online and have validated that it's functioning as expected. We apologize for how this incident may have affected you and your business.

    Resolved

  11. SanityQuery Performance Degradation for Some Customers

    A small subset of customers experienced increased query latency and intermittent timeouts during this incident. Our engineering team identified the issue, implemented mitigations, and restored normal query performance. The incident has been resolved.

    Resolved

Older incidents