Resolved
Live outage feed
Outages and recoveries from the services you depend on.
Earlier incidents
40 incidents
Oct 10, 2026
CloudflareCloudflare API service issues
This incident has been resolved.
Resolved
SupabaseIntermittent connection errors via the connection pooler in us-east-1
Between 22:10 and 22:51 UTC on October 9, 2026, about 1% of new database connections made through the Shared Pooler (Supavisor) in us-east-1 failed with "Failed to connect to database" timeout errors. The cause of the failures was an incident at our cloud provider which affected network traffic between availability zones. Supabase Project databases were not affected, and connections that had already been open prior to the incident remained open. The issue was resolved at 22:51 UTC. Clients that retried on connection errors would have reconnected successfully.
Resolved
RunPodEmergency Network Maintenance - EUR-NO-2
Maintenance ended
Resolved
HarnessProd2 Pipelines with verify step failing intermittently
### Summary Customers on Prod 2 experienced intermittent failures of a few Verify steps at startup. Affected steps failed shortly after launch with a message indicating Continuous Verification\(CV\) could not reach an internal Harness service. The issue stemmed from a recent service update that introduced connection pooling to improve performance and reliability with a keep-alive setting incompatible with the receiving service's timeout. No customer data was lost or corrupted. ### Customer Impact Customers using CV Verify steps on Prod-2 saw intermittent step failures during the incident window. The failure rate was low — the majority of executions completed successfully throughout the window. Pipelines that did not use CV Verify steps were unaffected. ### Root Cause A recent CV service update introduced connection pooling to an internal platform component, with a keep-alive duration longer than that component's idle timeout. Connections idle long enough were silently closed by the server; the CV service did not detect this and had no retry logic for such failures, causing them to surface directly as Verify step failures. The issue was intermittent because it only affected connections that happened to be idle long enough to expire. ### Mitigation The connection keep-alive duration was reduced to fall within the receiving service's idle timeout. Failures dropped to near zero immediately, and no further step failures of this type have been observed on any production environment. ### Next Steps To prevent recurrence, Harness will: * **Fix connection handling:** Tuned the default keep-alive and enabled automatic retry on connection failures so stale-connection errors are handled transparently. * **Add resilience at step startup:** Introduce retry with backoff for the internal service call at the Verify step start, so isolated transient failures do not surface as immediate step failures. * **Improve alerting:** Enhanced monitoring for internal connection error rates.
Resolved
VercelDelays in Drains, Observability, and Workflow in Washington (iad1)
All services have recovered.
Resolved
TemporalElevated latency and errors in AWS us-east-1
This incident has been resolved. The AWS us-east-1 region is stable and all affected namespaces have been failed back.
Resolved
BubbleMobile builds failing due to 3rd party service outage
The 3rd party service is still in degraded performance but our builds are succeeding.
Resolved
ChatGPT / OpenAISome ChatGPT content may be unavailable on Android
All impacted services have now fully recovered.
Resolved
Confluent CloudPossible outage in AWS us-east-1
This incident has been resolved. All affected services have fully recovered.
Resolved
MuxElevated API Latency
This incident has been resolved.
Resolved
ReplitUsers unable to top-up
Users are able to top up again.
Resolved
Oct 9, 2026
InngestDegraded Function Execution
One of our datacenter providers, Rackspace, had a network outage from 2:05 pm Pacific to 2:49 pm Pacific (21:05 UTC to 21:49 UTC). This resulted in degraded performance. We apologize for the inconvenience. We're going to keep monitoring and update if anything changes
Resolved
ChatGPT / OpenAIWorkspace admins may be unable to manage plugins
All impacted services have now fully recovered.
Resolved
JFrogJFrog Cloud Platform outage affecting subset of customers in AWS US East region
This incident has been resolved.
Resolved
PlanetScaleAWS us-east-1 connectivity failures
AWS us-east-1 cross-zone networking has fully recovered.
Resolved
PerplexityElevated API errors and timeouts
The issue affecting the Website, App, and Computer has been resolved. All services are operational. The API remained operational throughout this incident.
Resolved
RunwayWe've identified a problem and are working on a fix
This incident has now been resolved.
Resolved
Sumo LogicProblem with Metrics Alerting in America Federal (FED)
We are no longer seeing issues with Metrics Alerting and normal operation has been restored at this time.
Resolved
ConvexFailing function calls
On 2026-10-09, between 22:11 and 22:52 UTC, some query and mutation executions running in us-east-1 failed to run. This outage was caused by a network issue at an upstream provider. The problem is now resolved.
Resolved
CursorInvestigating service degradation — Cloud Agents and Grok Bot
This incident has been resolved. New Cloud Agents and Grok Bot sessions are starting normally again.
Resolved
CircleCIDelays starting Gen 2 Docker Jobs
## **Summary** Between October 1 and October 8, 2026, CircleCI customers experienced five separate incidents in which jobs waited longer than normal to start, or failed to start. Three of the incidents were driven by limited instance availability from our cloud provider. One was caused by a configuration issue in our scheduling infrastructure, and one by memory limits on an internal service, which we raised during the incident. * **October 1, 07:57 to 10:24 UTC:** Docker Gen 2 jobs waited up to 30 minutes to start. A configuration issue prevented the Gen 2 scheduling servers from using their high-performance local disks. During a burst in traffic, the fallback disks saturated and job placement slowed. * **October 7, 13:27 to 14:06 UTC:** Machine jobs waited longer than normal because we could not acquire enough virtual machines from our cloud provider. * **October 7, 14:24 to 15:21 UTC:** Some jobs failed with an infrastructure error, and workflows were delayed. The service that gives each job its configuration at startup ran out of memory and restarted. * **October 7, 18:49 to 20:05 UTC:** Docker Gen 2 jobs waited longer than normal because we could not acquire enough instances from our cloud provider. * **October 8, 12:48 to 16:00 UTC:** Linux machine jobs and remote Docker jobs waited up to 50 minutes to start. Our cloud provider did not have enough of the instance type we request, and a defect in the logic that picks a different instance type kept us from using alternatives. Jobs that waited during these incidents were delayed, not lost. Customers whose jobs failed with an infrastructure error during the October 7 incident from 14:24 to 15:21 UTC can rerun them. The original status pages can be found below: * [October 1 - Delays starting Gen 2 Docker Jobs](https://status.circleci.com/incidents/22mrv46g6n7g) * [October 7 - Delay on starting Machine Job Tasks](https://status.circleci.com/incidents/tj6wdnfjwr64) * [October 7 - Elevated level of infra fails on customer jobs](https://status.circleci.com/incidents/0z31ldjd8m5v) * [October 7 - Increased task wait times for Docker Gen2](https://status.circleci.com/incidents/39rfrjmk5pvd) * [October 8 - Elevated wait times for machine jobs](https://status.circleci.com/incidents/jn756xtsy4xg) ## **Background** Every job that runs on CircleCI needs compute. Where that compute comes from depends on the executor. Docker jobs run on clusters of servers that schedule work onto a fleet of instances. The scheduler is the software on those servers that decides which instance runs each job. Gen 2 Docker jobs ran on one such cluster at the time of the October 1 incident. Gen 1 Docker jobs are spread across several clusters. Machine jobs, including remote Docker jobs, get a dedicated virtual machine. A remote Docker job is handled as a Linux machine job from submission through provisioning. Our machine provisioning service requests each machine from our cloud provider in one of the two regions we normally use for machine jobs. When the provider cannot supply the instance type we request, the service falls back: it requests a different instance type, or a different region, from a list we maintain. A job waits until a machine is available. When any job starts, it asks a job configuration service for its configuration. That service runs as a set of pods, which are small copies of the same program. If too many pods stop, the remaining pods carry more load, and jobs cannot start until the service recovers. These paths depend on three kinds of capacity: the throughput of our scheduling systems, the supply of instances from our cloud provider, and the memory and number of copies of the services that start jobs. A shortfall in any of them makes jobs wait or fail. ## **What Happened** _\(All times UTC\)_ ### **October 1: Docker Gen 2 jobs delayed up to 30 minutes** At 07:44, disk activity on the servers that schedule Gen 2 Docker jobs rose to 90% of capacity and stayed there. At 07:57, the number of Docker jobs waiting to start began to grow, and wait times rose with it. An automated alert paged the on-call engineer at 08:08. We declared an incident at 08:42. At 08:43, engineers identified high disk latency on the scheduling servers. They first believed the disks were too slow for the load, so they prepared a change to a faster disk type. At 09:01, the team agreed on two workstreams: move the servers to faster disks, and add a second Gen 2 cluster so that jobs would split across two clusters. At 09:41, engineers found the root cause. Each scheduling server has fast disks attached directly to it, and the scheduler is meant to keep its working data there. These servers run on a security-hardened operating system image, and on that image the fast disks were not being used. The scheduler wrote its data to slower network-attached disks instead. The script that sets up the fast disks failed without an error, and a gap in logging on these servers hid the failure. Gen 2 traffic had also become increasingly bursty. The slower disks saturated during those bursts, and the scheduler fell behind. Wait times peaked at 09:54, when jobs on the Gen 2 medium resource class waited up to 30 minutes. At 09:55, engineers applied a fix that set up the fast disks correctly. At 10:11, the scheduler began placing jobs faster, which confirmed the servers could keep up with new jobs. Wait times for Gen 2 medium fell to 13 minutes at 10:14, 7 minutes at 10:17, and 1 minute 30 seconds at 10:20. We moved the status page to monitoring at 10:24 and resolved the incident at 10:45. At 11:20, the second Gen 2 cluster began running jobs. ### **October 7, 13:27 to 14:06: Elevated wait times for machine jobs** We declared an incident at 13:27 after seeing elevated queueing and wait times. We posted to the status page at 13:31. At 13:31, engineers saw that we were struggling to acquire virtual machines for machine jobs, and that the number of jobs waiting to start was high mainly for machine jobs. At 13:34, wait times were elevated across executors. At 13:48, wait times began to recover as more machines became available, and we resolved the incident at 14:06. ### **October 7, 14:24 to 15:21: Job failures and workflow delays** Shortly after the previous incident was marked as resolved, we declared a new incident at 14:24 due to a sharp drop in the number of jobs submitted to our execution systems. At 14:26, engineers saw that pods of the job configuration service were restarting. Pods that restart leave fewer copies of the service to answer requests, and jobs that could not retrieve their configuration after repeated attempts failed with an infrastructure error. Workflows that were already running waited for updates from jobs that had not started. At 14:35, engineers increased the amount of available memory to the pods and increased the number of pods, which gave the service more copies to share the load. We also increased the amount of memory available Job submissions began recovering, and the number of messages flowing through our workflow system returned to normal levels. At 14:44, we moved the status page to monitoring. Pods in the job configuration service reached their memory limits and restarted. We increased memory allocations and pod counts, and the service recovered. At 15:13, we increased the number of instances in our Gen 2 Docker clusters so we could work through the backlog of waiting jobs quickly. We resolved the incident at 15:21. ### **October 7, 18:49 to 20:05: Elevated wait times for Docker Gen 2 jobs** We declared a new incident at 18:49 after Docker Gen 2 wait times rose. At 18:51, engineers confirmed that a shortage of available instances kept us from acquiring enough of them, which raised wait times. Wait times differed by resource class size. At 19:52, average wait times were lower than an hour earlier as more capacity became available, though brief spikes continued, and we moved the status page to monitoring. We resolved the incident at 20:05. ### **October 8: Linux machine and remote Docker jobs wait up to 50 minutes** We declared an incident at 12:48 after seeing a large number of jobs waiting to start and long wait times for machine and remote Docker jobs. At 12:49, engineers saw that our cloud provider could not supply enough instances. Delays continued to climb as we investigated. At 13:13, we began examining the logic that chooses a different instance type or region when our first choice is unavailable. At 13:24, we suspected that a defect in that logic was stopping us from using other instance types in our primary region, so we disabled our fallback region and confirmed the cause of the defect. Engineers began working on a fix and deployed it at 14:49. The service began requesting other instance types and started machines in large numbers. At 14:58, the queue of jobs waiting for a machine dropped steadily. Average wait times across resource classes continued to drop over the next hour as we worked through the backlog of waiting jobs. At 16:00, the queue of jobs waiting for a machine cleared and the number of waiting jobs returned to normal. We moved the status page to monitoring at 16:09 and resolved the incident at 16:35. ## **Future Prevention and Process Improvement** We are taking the following steps to strengthen the resilience of job start times, including supply diversification, fallback logic, and earlier detection. **We are expanding our compute regions.** In the October 7 and October 8 incidents, our cloud provider could not supply enough instances in the regions we used. We are adding new regions to our fallback logic in order handle spikes for demand that outpace our preferred regions. **We are fixing the logic that picks a different instance type.** On October 8, a defect kept the machine provisioning service from using other instance types in the same region. We deployed a fix during the incident and are now reviewing and refactoring this logic as a whole, with tests, so that a shortage of one instance type moves work to another. **We are expanding the instance types we offer and securing additional capacity.** We are working with our cloud provider to reserve additional capacity in our existing regions, and we are adding instance types that jobs can run on. Industry demand for high-performance compute is growing. We are diversifying where and how we source capacity so our customers are insulated from supply variation. **We added a second Gen 2 Docker cluster so traffic bursts are distributed across two clusters.** We are also adding alerts on scheduling server disk saturation and on how fast the scheduler places jobs, and an alert that fires when a server starts without its fast disks in use. We made the disk setup script fail with an error, and we restored log shipping from these servers so that a similar problem is visible sooner. Customer experience is our top priority, and we commit to continually improving the reliability of our systems to match the trust that our customers place in us. We thank our customers for their patience while our team worked to resolve these incidents. Please reach out to our support team with any questions or concerns.
Resolved
CircleCIDelay on starting Machine Job Tasks
## **Summary** Between October 1 and October 8, 2026, CircleCI customers experienced five separate incidents in which jobs waited longer than normal to start, or failed to start. Three of the incidents were driven by limited instance availability from our cloud provider. One was caused by a configuration issue in our scheduling infrastructure, and one by memory limits on an internal service, which we raised during the incident. * **October 1, 07:57 to 10:24 UTC:** Docker Gen 2 jobs waited up to 30 minutes to start. A configuration issue prevented the Gen 2 scheduling servers from using their high-performance local disks. During a burst in traffic, the fallback disks saturated and job placement slowed. * **October 7, 13:27 to 14:06 UTC:** Machine jobs waited longer than normal because we could not acquire enough virtual machines from our cloud provider. * **October 7, 14:24 to 15:21 UTC:** Some jobs failed with an infrastructure error, and workflows were delayed. The service that gives each job its configuration at startup ran out of memory and restarted. * **October 7, 18:49 to 20:05 UTC:** Docker Gen 2 jobs waited longer than normal because we could not acquire enough instances from our cloud provider. * **October 8, 12:48 to 16:00 UTC:** Linux machine jobs and remote Docker jobs waited up to 50 minutes to start. Our cloud provider did not have enough of the instance type we request, and a defect in the logic that picks a different instance type kept us from using alternatives. Jobs that waited during these incidents were delayed, not lost. Customers whose jobs failed with an infrastructure error during the October 7 incident from 14:24 to 15:21 UTC can rerun them. The original status pages can be found below: * [October 1 - Delays starting Gen 2 Docker Jobs](https://status.circleci.com/incidents/22mrv46g6n7g) * [October 7 - Delay on starting Machine Job Tasks](https://status.circleci.com/incidents/tj6wdnfjwr64) * [October 7 - Elevated level of infra fails on customer jobs](https://status.circleci.com/incidents/0z31ldjd8m5v) * [October 7 - Increased task wait times for Docker Gen2](https://status.circleci.com/incidents/39rfrjmk5pvd) * [October 8 - Elevated wait times for machine jobs](https://status.circleci.com/incidents/jn756xtsy4xg) ## **Background** Every job that runs on CircleCI needs compute. Where that compute comes from depends on the executor. Docker jobs run on clusters of servers that schedule work onto a fleet of instances. The scheduler is the software on those servers that decides which instance runs each job. Gen 2 Docker jobs ran on one such cluster at the time of the October 1 incident. Gen 1 Docker jobs are spread across several clusters. Machine jobs, including remote Docker jobs, get a dedicated virtual machine. A remote Docker job is handled as a Linux machine job from submission through provisioning. Our machine provisioning service requests each machine from our cloud provider in one of the two regions we normally use for machine jobs. When the provider cannot supply the instance type we request, the service falls back: it requests a different instance type, or a different region, from a list we maintain. A job waits until a machine is available. When any job starts, it asks a job configuration service for its configuration. That service runs as a set of pods, which are small copies of the same program. If too many pods stop, the remaining pods carry more load, and jobs cannot start until the service recovers. These paths depend on three kinds of capacity: the throughput of our scheduling systems, the supply of instances from our cloud provider, and the memory and number of copies of the services that start jobs. A shortfall in any of them makes jobs wait or fail. ## **What Happened** _\(All times UTC\)_ ### **October 1: Docker Gen 2 jobs delayed up to 30 minutes** At 07:44, disk activity on the servers that schedule Gen 2 Docker jobs rose to 90% of capacity and stayed there. At 07:57, the number of Docker jobs waiting to start began to grow, and wait times rose with it. An automated alert paged the on-call engineer at 08:08. We declared an incident at 08:42. At 08:43, engineers identified high disk latency on the scheduling servers. They first believed the disks were too slow for the load, so they prepared a change to a faster disk type. At 09:01, the team agreed on two workstreams: move the servers to faster disks, and add a second Gen 2 cluster so that jobs would split across two clusters. At 09:41, engineers found the root cause. Each scheduling server has fast disks attached directly to it, and the scheduler is meant to keep its working data there. These servers run on a security-hardened operating system image, and on that image the fast disks were not being used. The scheduler wrote its data to slower network-attached disks instead. The script that sets up the fast disks failed without an error, and a gap in logging on these servers hid the failure. Gen 2 traffic had also become increasingly bursty. The slower disks saturated during those bursts, and the scheduler fell behind. Wait times peaked at 09:54, when jobs on the Gen 2 medium resource class waited up to 30 minutes. At 09:55, engineers applied a fix that set up the fast disks correctly. At 10:11, the scheduler began placing jobs faster, which confirmed the servers could keep up with new jobs. Wait times for Gen 2 medium fell to 13 minutes at 10:14, 7 minutes at 10:17, and 1 minute 30 seconds at 10:20. We moved the status page to monitoring at 10:24 and resolved the incident at 10:45. At 11:20, the second Gen 2 cluster began running jobs. ### **October 7, 13:27 to 14:06: Elevated wait times for machine jobs** We declared an incident at 13:27 after seeing elevated queueing and wait times. We posted to the status page at 13:31. At 13:31, engineers saw that we were struggling to acquire virtual machines for machine jobs, and that the number of jobs waiting to start was high mainly for machine jobs. At 13:34, wait times were elevated across executors. At 13:48, wait times began to recover as more machines became available, and we resolved the incident at 14:06. ### **October 7, 14:24 to 15:21: Job failures and workflow delays** Shortly after the previous incident was marked as resolved, we declared a new incident at 14:24 due to a sharp drop in the number of jobs submitted to our execution systems. At 14:26, engineers saw that pods of the job configuration service were restarting. Pods that restart leave fewer copies of the service to answer requests, and jobs that could not retrieve their configuration after repeated attempts failed with an infrastructure error. Workflows that were already running waited for updates from jobs that had not started. At 14:35, engineers increased the amount of available memory to the pods and increased the number of pods, which gave the service more copies to share the load. We also increased the amount of memory available Job submissions began recovering, and the number of messages flowing through our workflow system returned to normal levels. At 14:44, we moved the status page to monitoring. Pods in the job configuration service reached their memory limits and restarted. We increased memory allocations and pod counts, and the service recovered. At 15:13, we increased the number of instances in our Gen 2 Docker clusters so we could work through the backlog of waiting jobs quickly. We resolved the incident at 15:21. ### **October 7, 18:49 to 20:05: Elevated wait times for Docker Gen 2 jobs** We declared a new incident at 18:49 after Docker Gen 2 wait times rose. At 18:51, engineers confirmed that a shortage of available instances kept us from acquiring enough of them, which raised wait times. Wait times differed by resource class size. At 19:52, average wait times were lower than an hour earlier as more capacity became available, though brief spikes continued, and we moved the status page to monitoring. We resolved the incident at 20:05. ### **October 8: Linux machine and remote Docker jobs wait up to 50 minutes** We declared an incident at 12:48 after seeing a large number of jobs waiting to start and long wait times for machine and remote Docker jobs. At 12:49, engineers saw that our cloud provider could not supply enough instances. Delays continued to climb as we investigated. At 13:13, we began examining the logic that chooses a different instance type or region when our first choice is unavailable. At 13:24, we suspected that a defect in that logic was stopping us from using other instance types in our primary region, so we disabled our fallback region and confirmed the cause of the defect. Engineers began working on a fix and deployed it at 14:49. The service began requesting other instance types and started machines in large numbers. At 14:58, the queue of jobs waiting for a machine dropped steadily. Average wait times across resource classes continued to drop over the next hour as we worked through the backlog of waiting jobs. At 16:00, the queue of jobs waiting for a machine cleared and the number of waiting jobs returned to normal. We moved the status page to monitoring at 16:09 and resolved the incident at 16:35. ## **Future Prevention and Process Improvement** We are taking the following steps to strengthen the resilience of job start times, including supply diversification, fallback logic, and earlier detection. **We are expanding our compute regions.** In the October 7 and October 8 incidents, our cloud provider could not supply enough instances in the regions we used. We are adding new regions to our fallback logic in order handle spikes for demand that outpace our preferred regions. **We are fixing the logic that picks a different instance type.** On October 8, a defect kept the machine provisioning service from using other instance types in the same region. We deployed a fix during the incident and are now reviewing and refactoring this logic as a whole, with tests, so that a shortage of one instance type moves work to another. **We are expanding the instance types we offer and securing additional capacity.** We are working with our cloud provider to reserve additional capacity in our existing regions, and we are adding instance types that jobs can run on. Industry demand for high-performance compute is growing. We are diversifying where and how we source capacity so our customers are insulated from supply variation. **We added a second Gen 2 Docker cluster so traffic bursts are distributed across two clusters.** We are also adding alerts on scheduling server disk saturation and on how fast the scheduler places jobs, and an alert that fires when a server starts without its fast disks in use. We made the disk setup script fail with an error, and we restored log shipping from these servers so that a similar problem is visible sooner. Customer experience is our top priority, and we commit to continually improving the reliability of our systems to match the trust that our customers place in us. We thank our customers for their patience while our team worked to resolve these incidents. Please reach out to our support team with any questions or concerns.
Resolved
CircleCIElevated level of infra fails on customer jobs
## **Summary** Between October 1 and October 8, 2026, CircleCI customers experienced five separate incidents in which jobs waited longer than normal to start, or failed to start. Three of the incidents were driven by limited instance availability from our cloud provider. One was caused by a configuration issue in our scheduling infrastructure, and one by memory limits on an internal service, which we raised during the incident. * **October 1, 07:57 to 10:24 UTC:** Docker Gen 2 jobs waited up to 30 minutes to start. A configuration issue prevented the Gen 2 scheduling servers from using their high-performance local disks. During a burst in traffic, the fallback disks saturated and job placement slowed. * **October 7, 13:27 to 14:06 UTC:** Machine jobs waited longer than normal because we could not acquire enough virtual machines from our cloud provider. * **October 7, 14:24 to 15:21 UTC:** Some jobs failed with an infrastructure error, and workflows were delayed. The service that gives each job its configuration at startup ran out of memory and restarted. * **October 7, 18:49 to 20:05 UTC:** Docker Gen 2 jobs waited longer than normal because we could not acquire enough instances from our cloud provider. * **October 8, 12:48 to 16:00 UTC:** Linux machine jobs and remote Docker jobs waited up to 50 minutes to start. Our cloud provider did not have enough of the instance type we request, and a defect in the logic that picks a different instance type kept us from using alternatives. Jobs that waited during these incidents were delayed, not lost. Customers whose jobs failed with an infrastructure error during the October 7 incident from 14:24 to 15:21 UTC can rerun them. The original status pages can be found below: * [October 1 - Delays starting Gen 2 Docker Jobs](https://status.circleci.com/incidents/22mrv46g6n7g) * [October 7 - Delay on starting Machine Job Tasks](https://status.circleci.com/incidents/tj6wdnfjwr64) * [October 7 - Elevated level of infra fails on customer jobs](https://status.circleci.com/incidents/0z31ldjd8m5v) * [October 7 - Increased task wait times for Docker Gen2](https://status.circleci.com/incidents/39rfrjmk5pvd) * [October 8 - Elevated wait times for machine jobs](https://status.circleci.com/incidents/jn756xtsy4xg) ## **Background** Every job that runs on CircleCI needs compute. Where that compute comes from depends on the executor. Docker jobs run on clusters of servers that schedule work onto a fleet of instances. The scheduler is the software on those servers that decides which instance runs each job. Gen 2 Docker jobs ran on one such cluster at the time of the October 1 incident. Gen 1 Docker jobs are spread across several clusters. Machine jobs, including remote Docker jobs, get a dedicated virtual machine. A remote Docker job is handled as a Linux machine job from submission through provisioning. Our machine provisioning service requests each machine from our cloud provider in one of the two regions we normally use for machine jobs. When the provider cannot supply the instance type we request, the service falls back: it requests a different instance type, or a different region, from a list we maintain. A job waits until a machine is available. When any job starts, it asks a job configuration service for its configuration. That service runs as a set of pods, which are small copies of the same program. If too many pods stop, the remaining pods carry more load, and jobs cannot start until the service recovers. These paths depend on three kinds of capacity: the throughput of our scheduling systems, the supply of instances from our cloud provider, and the memory and number of copies of the services that start jobs. A shortfall in any of them makes jobs wait or fail. ## **What Happened** _\(All times UTC\)_ ### **October 1: Docker Gen 2 jobs delayed up to 30 minutes** At 07:44, disk activity on the servers that schedule Gen 2 Docker jobs rose to 90% of capacity and stayed there. At 07:57, the number of Docker jobs waiting to start began to grow, and wait times rose with it. An automated alert paged the on-call engineer at 08:08. We declared an incident at 08:42. At 08:43, engineers identified high disk latency on the scheduling servers. They first believed the disks were too slow for the load, so they prepared a change to a faster disk type. At 09:01, the team agreed on two workstreams: move the servers to faster disks, and add a second Gen 2 cluster so that jobs would split across two clusters. At 09:41, engineers found the root cause. Each scheduling server has fast disks attached directly to it, and the scheduler is meant to keep its working data there. These servers run on a security-hardened operating system image, and on that image the fast disks were not being used. The scheduler wrote its data to slower network-attached disks instead. The script that sets up the fast disks failed without an error, and a gap in logging on these servers hid the failure. Gen 2 traffic had also become increasingly bursty. The slower disks saturated during those bursts, and the scheduler fell behind. Wait times peaked at 09:54, when jobs on the Gen 2 medium resource class waited up to 30 minutes. At 09:55, engineers applied a fix that set up the fast disks correctly. At 10:11, the scheduler began placing jobs faster, which confirmed the servers could keep up with new jobs. Wait times for Gen 2 medium fell to 13 minutes at 10:14, 7 minutes at 10:17, and 1 minute 30 seconds at 10:20. We moved the status page to monitoring at 10:24 and resolved the incident at 10:45. At 11:20, the second Gen 2 cluster began running jobs. ### **October 7, 13:27 to 14:06: Elevated wait times for machine jobs** We declared an incident at 13:27 after seeing elevated queueing and wait times. We posted to the status page at 13:31. At 13:31, engineers saw that we were struggling to acquire virtual machines for machine jobs, and that the number of jobs waiting to start was high mainly for machine jobs. At 13:34, wait times were elevated across executors. At 13:48, wait times began to recover as more machines became available, and we resolved the incident at 14:06. ### **October 7, 14:24 to 15:21: Job failures and workflow delays** Shortly after the previous incident was marked as resolved, we declared a new incident at 14:24 due to a sharp drop in the number of jobs submitted to our execution systems. At 14:26, engineers saw that pods of the job configuration service were restarting. Pods that restart leave fewer copies of the service to answer requests, and jobs that could not retrieve their configuration after repeated attempts failed with an infrastructure error. Workflows that were already running waited for updates from jobs that had not started. At 14:35, engineers increased the amount of available memory to the pods and increased the number of pods, which gave the service more copies to share the load. We also increased the amount of memory available Job submissions began recovering, and the number of messages flowing through our workflow system returned to normal levels. At 14:44, we moved the status page to monitoring. Pods in the job configuration service reached their memory limits and restarted. We increased memory allocations and pod counts, and the service recovered. At 15:13, we increased the number of instances in our Gen 2 Docker clusters so we could work through the backlog of waiting jobs quickly. We resolved the incident at 15:21. ### **October 7, 18:49 to 20:05: Elevated wait times for Docker Gen 2 jobs** We declared a new incident at 18:49 after Docker Gen 2 wait times rose. At 18:51, engineers confirmed that a shortage of available instances kept us from acquiring enough of them, which raised wait times. Wait times differed by resource class size. At 19:52, average wait times were lower than an hour earlier as more capacity became available, though brief spikes continued, and we moved the status page to monitoring. We resolved the incident at 20:05. ### **October 8: Linux machine and remote Docker jobs wait up to 50 minutes** We declared an incident at 12:48 after seeing a large number of jobs waiting to start and long wait times for machine and remote Docker jobs. At 12:49, engineers saw that our cloud provider could not supply enough instances. Delays continued to climb as we investigated. At 13:13, we began examining the logic that chooses a different instance type or region when our first choice is unavailable. At 13:24, we suspected that a defect in that logic was stopping us from using other instance types in our primary region, so we disabled our fallback region and confirmed the cause of the defect. Engineers began working on a fix and deployed it at 14:49. The service began requesting other instance types and started machines in large numbers. At 14:58, the queue of jobs waiting for a machine dropped steadily. Average wait times across resource classes continued to drop over the next hour as we worked through the backlog of waiting jobs. At 16:00, the queue of jobs waiting for a machine cleared and the number of waiting jobs returned to normal. We moved the status page to monitoring at 16:09 and resolved the incident at 16:35. ## **Future Prevention and Process Improvement** We are taking the following steps to strengthen the resilience of job start times, including supply diversification, fallback logic, and earlier detection. **We are expanding our compute regions.** In the October 7 and October 8 incidents, our cloud provider could not supply enough instances in the regions we used. We are adding new regions to our fallback logic in order handle spikes for demand that outpace our preferred regions. **We are fixing the logic that picks a different instance type.** On October 8, a defect kept the machine provisioning service from using other instance types in the same region. We deployed a fix during the incident and are now reviewing and refactoring this logic as a whole, with tests, so that a shortage of one instance type moves work to another. **We are expanding the instance types we offer and securing additional capacity.** We are working with our cloud provider to reserve additional capacity in our existing regions, and we are adding instance types that jobs can run on. Industry demand for high-performance compute is growing. We are diversifying where and how we source capacity so our customers are insulated from supply variation. **We added a second Gen 2 Docker cluster so traffic bursts are distributed across two clusters.** We are also adding alerts on scheduling server disk saturation and on how fast the scheduler places jobs, and an alert that fires when a server starts without its fast disks in use. We made the disk setup script fail with an error, and we restored log shipping from these servers so that a similar problem is visible sooner. Customer experience is our top priority, and we commit to continually improving the reliability of our systems to match the trust that our customers place in us. We thank our customers for their patience while our team worked to resolve these incidents. Please reach out to our support team with any questions or concerns.
Resolved
CircleCIIncreased task wait times for Docker Gen2
## **Summary** Between October 1 and October 8, 2026, CircleCI customers experienced five separate incidents in which jobs waited longer than normal to start, or failed to start. Three of the incidents were driven by limited instance availability from our cloud provider. One was caused by a configuration issue in our scheduling infrastructure, and one by memory limits on an internal service, which we raised during the incident. * **October 1, 07:57 to 10:24 UTC:** Docker Gen 2 jobs waited up to 30 minutes to start. A configuration issue prevented the Gen 2 scheduling servers from using their high-performance local disks. During a burst in traffic, the fallback disks saturated and job placement slowed. * **October 7, 13:27 to 14:06 UTC:** Machine jobs waited longer than normal because we could not acquire enough virtual machines from our cloud provider. * **October 7, 14:24 to 15:21 UTC:** Some jobs failed with an infrastructure error, and workflows were delayed. The service that gives each job its configuration at startup ran out of memory and restarted. * **October 7, 18:49 to 20:05 UTC:** Docker Gen 2 jobs waited longer than normal because we could not acquire enough instances from our cloud provider. * **October 8, 12:48 to 16:00 UTC:** Linux machine jobs and remote Docker jobs waited up to 50 minutes to start. Our cloud provider did not have enough of the instance type we request, and a defect in the logic that picks a different instance type kept us from using alternatives. Jobs that waited during these incidents were delayed, not lost. Customers whose jobs failed with an infrastructure error during the October 7 incident from 14:24 to 15:21 UTC can rerun them. The original status pages can be found below: * [October 1 - Delays starting Gen 2 Docker Jobs](https://status.circleci.com/incidents/22mrv46g6n7g) * [October 7 - Delay on starting Machine Job Tasks](https://status.circleci.com/incidents/tj6wdnfjwr64) * [October 7 - Elevated level of infra fails on customer jobs](https://status.circleci.com/incidents/0z31ldjd8m5v) * [October 7 - Increased task wait times for Docker Gen2](https://status.circleci.com/incidents/39rfrjmk5pvd) * [October 8 - Elevated wait times for machine jobs](https://status.circleci.com/incidents/jn756xtsy4xg) ## **Background** Every job that runs on CircleCI needs compute. Where that compute comes from depends on the executor. Docker jobs run on clusters of servers that schedule work onto a fleet of instances. The scheduler is the software on those servers that decides which instance runs each job. Gen 2 Docker jobs ran on one such cluster at the time of the October 1 incident. Gen 1 Docker jobs are spread across several clusters. Machine jobs, including remote Docker jobs, get a dedicated virtual machine. A remote Docker job is handled as a Linux machine job from submission through provisioning. Our machine provisioning service requests each machine from our cloud provider in one of the two regions we normally use for machine jobs. When the provider cannot supply the instance type we request, the service falls back: it requests a different instance type, or a different region, from a list we maintain. A job waits until a machine is available. When any job starts, it asks a job configuration service for its configuration. That service runs as a set of pods, which are small copies of the same program. If too many pods stop, the remaining pods carry more load, and jobs cannot start until the service recovers. These paths depend on three kinds of capacity: the throughput of our scheduling systems, the supply of instances from our cloud provider, and the memory and number of copies of the services that start jobs. A shortfall in any of them makes jobs wait or fail. ## **What Happened** _\(All times UTC\)_ ### **October 1: Docker Gen 2 jobs delayed up to 30 minutes** At 07:44, disk activity on the servers that schedule Gen 2 Docker jobs rose to 90% of capacity and stayed there. At 07:57, the number of Docker jobs waiting to start began to grow, and wait times rose with it. An automated alert paged the on-call engineer at 08:08. We declared an incident at 08:42. At 08:43, engineers identified high disk latency on the scheduling servers. They first believed the disks were too slow for the load, so they prepared a change to a faster disk type. At 09:01, the team agreed on two workstreams: move the servers to faster disks, and add a second Gen 2 cluster so that jobs would split across two clusters. At 09:41, engineers found the root cause. Each scheduling server has fast disks attached directly to it, and the scheduler is meant to keep its working data there. These servers run on a security-hardened operating system image, and on that image the fast disks were not being used. The scheduler wrote its data to slower network-attached disks instead. The script that sets up the fast disks failed without an error, and a gap in logging on these servers hid the failure. Gen 2 traffic had also become increasingly bursty. The slower disks saturated during those bursts, and the scheduler fell behind. Wait times peaked at 09:54, when jobs on the Gen 2 medium resource class waited up to 30 minutes. At 09:55, engineers applied a fix that set up the fast disks correctly. At 10:11, the scheduler began placing jobs faster, which confirmed the servers could keep up with new jobs. Wait times for Gen 2 medium fell to 13 minutes at 10:14, 7 minutes at 10:17, and 1 minute 30 seconds at 10:20. We moved the status page to monitoring at 10:24 and resolved the incident at 10:45. At 11:20, the second Gen 2 cluster began running jobs. ### **October 7, 13:27 to 14:06: Elevated wait times for machine jobs** We declared an incident at 13:27 after seeing elevated queueing and wait times. We posted to the status page at 13:31. At 13:31, engineers saw that we were struggling to acquire virtual machines for machine jobs, and that the number of jobs waiting to start was high mainly for machine jobs. At 13:34, wait times were elevated across executors. At 13:48, wait times began to recover as more machines became available, and we resolved the incident at 14:06. ### **October 7, 14:24 to 15:21: Job failures and workflow delays** Shortly after the previous incident was marked as resolved, we declared a new incident at 14:24 due to a sharp drop in the number of jobs submitted to our execution systems. At 14:26, engineers saw that pods of the job configuration service were restarting. Pods that restart leave fewer copies of the service to answer requests, and jobs that could not retrieve their configuration after repeated attempts failed with an infrastructure error. Workflows that were already running waited for updates from jobs that had not started. At 14:35, engineers increased the amount of available memory to the pods and increased the number of pods, which gave the service more copies to share the load. We also increased the amount of memory available Job submissions began recovering, and the number of messages flowing through our workflow system returned to normal levels. At 14:44, we moved the status page to monitoring. Pods in the job configuration service reached their memory limits and restarted. We increased memory allocations and pod counts, and the service recovered. At 15:13, we increased the number of instances in our Gen 2 Docker clusters so we could work through the backlog of waiting jobs quickly. We resolved the incident at 15:21. ### **October 7, 18:49 to 20:05: Elevated wait times for Docker Gen 2 jobs** We declared a new incident at 18:49 after Docker Gen 2 wait times rose. At 18:51, engineers confirmed that a shortage of available instances kept us from acquiring enough of them, which raised wait times. Wait times differed by resource class size. At 19:52, average wait times were lower than an hour earlier as more capacity became available, though brief spikes continued, and we moved the status page to monitoring. We resolved the incident at 20:05. ### **October 8: Linux machine and remote Docker jobs wait up to 50 minutes** We declared an incident at 12:48 after seeing a large number of jobs waiting to start and long wait times for machine and remote Docker jobs. At 12:49, engineers saw that our cloud provider could not supply enough instances. Delays continued to climb as we investigated. At 13:13, we began examining the logic that chooses a different instance type or region when our first choice is unavailable. At 13:24, we suspected that a defect in that logic was stopping us from using other instance types in our primary region, so we disabled our fallback region and confirmed the cause of the defect. Engineers began working on a fix and deployed it at 14:49. The service began requesting other instance types and started machines in large numbers. At 14:58, the queue of jobs waiting for a machine dropped steadily. Average wait times across resource classes continued to drop over the next hour as we worked through the backlog of waiting jobs. At 16:00, the queue of jobs waiting for a machine cleared and the number of waiting jobs returned to normal. We moved the status page to monitoring at 16:09 and resolved the incident at 16:35. ## **Future Prevention and Process Improvement** We are taking the following steps to strengthen the resilience of job start times, including supply diversification, fallback logic, and earlier detection. **We are expanding our compute regions.** In the October 7 and October 8 incidents, our cloud provider could not supply enough instances in the regions we used. We are adding new regions to our fallback logic in order handle spikes for demand that outpace our preferred regions. **We are fixing the logic that picks a different instance type.** On October 8, a defect kept the machine provisioning service from using other instance types in the same region. We deployed a fix during the incident and are now reviewing and refactoring this logic as a whole, with tests, so that a shortage of one instance type moves work to another. **We are expanding the instance types we offer and securing additional capacity.** We are working with our cloud provider to reserve additional capacity in our existing regions, and we are adding instance types that jobs can run on. Industry demand for high-performance compute is growing. We are diversifying where and how we source capacity so our customers are insulated from supply variation. **We added a second Gen 2 Docker cluster so traffic bursts are distributed across two clusters.** We are also adding alerts on scheduling server disk saturation and on how fast the scheduler places jobs, and an alert that fires when a server starts without its fast disks in use. We made the disk setup script fail with an error, and we restored log shipping from these servers so that a similar problem is visible sooner. Customer experience is our top priority, and we commit to continually improving the reliability of our systems to match the trust that our customers place in us. We thank our customers for their patience while our team worked to resolve these incidents. Please reach out to our support team with any questions or concerns.
Resolved
CircleCIElevated wait times for machine jobs
## **Summary** Between October 1 and October 8, 2026, CircleCI customers experienced five separate incidents in which jobs waited longer than normal to start, or failed to start. Three of the incidents were driven by limited instance availability from our cloud provider. One was caused by a configuration issue in our scheduling infrastructure, and one by memory limits on an internal service, which we raised during the incident. * **October 1, 07:57 to 10:24 UTC:** Docker Gen 2 jobs waited up to 30 minutes to start. A configuration issue prevented the Gen 2 scheduling servers from using their high-performance local disks. During a burst in traffic, the fallback disks saturated and job placement slowed. * **October 7, 13:27 to 14:06 UTC:** Machine jobs waited longer than normal because we could not acquire enough virtual machines from our cloud provider. * **October 7, 14:24 to 15:21 UTC:** Some jobs failed with an infrastructure error, and workflows were delayed. The service that gives each job its configuration at startup ran out of memory and restarted. * **October 7, 18:49 to 20:05 UTC:** Docker Gen 2 jobs waited longer than normal because we could not acquire enough instances from our cloud provider. * **October 8, 12:48 to 16:00 UTC:** Linux machine jobs and remote Docker jobs waited up to 50 minutes to start. Our cloud provider did not have enough of the instance type we request, and a defect in the logic that picks a different instance type kept us from using alternatives. Jobs that waited during these incidents were delayed, not lost. Customers whose jobs failed with an infrastructure error during the October 7 incident from 14:24 to 15:21 UTC can rerun them. The original status pages can be found below: * [October 1 - Delays starting Gen 2 Docker Jobs](https://status.circleci.com/incidents/22mrv46g6n7g) * [October 7 - Delay on starting Machine Job Tasks](https://status.circleci.com/incidents/tj6wdnfjwr64) * [October 7 - Elevated level of infra fails on customer jobs](https://status.circleci.com/incidents/0z31ldjd8m5v) * [October 7 - Increased task wait times for Docker Gen2](https://status.circleci.com/incidents/39rfrjmk5pvd) * [October 8 - Elevated wait times for machine jobs](https://status.circleci.com/incidents/jn756xtsy4xg) ## **Background** Every job that runs on CircleCI needs compute. Where that compute comes from depends on the executor. Docker jobs run on clusters of servers that schedule work onto a fleet of instances. The scheduler is the software on those servers that decides which instance runs each job. Gen 2 Docker jobs ran on one such cluster at the time of the October 1 incident. Gen 1 Docker jobs are spread across several clusters. Machine jobs, including remote Docker jobs, get a dedicated virtual machine. A remote Docker job is handled as a Linux machine job from submission through provisioning. Our machine provisioning service requests each machine from our cloud provider in one of the two regions we normally use for machine jobs. When the provider cannot supply the instance type we request, the service falls back: it requests a different instance type, or a different region, from a list we maintain. A job waits until a machine is available. When any job starts, it asks a job configuration service for its configuration. That service runs as a set of pods, which are small copies of the same program. If too many pods stop, the remaining pods carry more load, and jobs cannot start until the service recovers. These paths depend on three kinds of capacity: the throughput of our scheduling systems, the supply of instances from our cloud provider, and the memory and number of copies of the services that start jobs. A shortfall in any of them makes jobs wait or fail. ## **What Happened** _\(All times UTC\)_ ### **October 1: Docker Gen 2 jobs delayed up to 30 minutes** At 07:44, disk activity on the servers that schedule Gen 2 Docker jobs rose to 90% of capacity and stayed there. At 07:57, the number of Docker jobs waiting to start began to grow, and wait times rose with it. An automated alert paged the on-call engineer at 08:08. We declared an incident at 08:42. At 08:43, engineers identified high disk latency on the scheduling servers. They first believed the disks were too slow for the load, so they prepared a change to a faster disk type. At 09:01, the team agreed on two workstreams: move the servers to faster disks, and add a second Gen 2 cluster so that jobs would split across two clusters. At 09:41, engineers found the root cause. Each scheduling server has fast disks attached directly to it, and the scheduler is meant to keep its working data there. These servers run on a security-hardened operating system image, and on that image the fast disks were not being used. The scheduler wrote its data to slower network-attached disks instead. The script that sets up the fast disks failed without an error, and a gap in logging on these servers hid the failure. Gen 2 traffic had also become increasingly bursty. The slower disks saturated during those bursts, and the scheduler fell behind. Wait times peaked at 09:54, when jobs on the Gen 2 medium resource class waited up to 30 minutes. At 09:55, engineers applied a fix that set up the fast disks correctly. At 10:11, the scheduler began placing jobs faster, which confirmed the servers could keep up with new jobs. Wait times for Gen 2 medium fell to 13 minutes at 10:14, 7 minutes at 10:17, and 1 minute 30 seconds at 10:20. We moved the status page to monitoring at 10:24 and resolved the incident at 10:45. At 11:20, the second Gen 2 cluster began running jobs. ### **October 7, 13:27 to 14:06: Elevated wait times for machine jobs** We declared an incident at 13:27 after seeing elevated queueing and wait times. We posted to the status page at 13:31. At 13:31, engineers saw that we were struggling to acquire virtual machines for machine jobs, and that the number of jobs waiting to start was high mainly for machine jobs. At 13:34, wait times were elevated across executors. At 13:48, wait times began to recover as more machines became available, and we resolved the incident at 14:06. ### **October 7, 14:24 to 15:21: Job failures and workflow delays** Shortly after the previous incident was marked as resolved, we declared a new incident at 14:24 due to a sharp drop in the number of jobs submitted to our execution systems. At 14:26, engineers saw that pods of the job configuration service were restarting. Pods that restart leave fewer copies of the service to answer requests, and jobs that could not retrieve their configuration after repeated attempts failed with an infrastructure error. Workflows that were already running waited for updates from jobs that had not started. At 14:35, engineers increased the amount of available memory to the pods and increased the number of pods, which gave the service more copies to share the load. We also increased the amount of memory available Job submissions began recovering, and the number of messages flowing through our workflow system returned to normal levels. At 14:44, we moved the status page to monitoring. Pods in the job configuration service reached their memory limits and restarted. We increased memory allocations and pod counts, and the service recovered. At 15:13, we increased the number of instances in our Gen 2 Docker clusters so we could work through the backlog of waiting jobs quickly. We resolved the incident at 15:21. ### **October 7, 18:49 to 20:05: Elevated wait times for Docker Gen 2 jobs** We declared a new incident at 18:49 after Docker Gen 2 wait times rose. At 18:51, engineers confirmed that a shortage of available instances kept us from acquiring enough of them, which raised wait times. Wait times differed by resource class size. At 19:52, average wait times were lower than an hour earlier as more capacity became available, though brief spikes continued, and we moved the status page to monitoring. We resolved the incident at 20:05. ### **October 8: Linux machine and remote Docker jobs wait up to 50 minutes** We declared an incident at 12:48 after seeing a large number of jobs waiting to start and long wait times for machine and remote Docker jobs. At 12:49, engineers saw that our cloud provider could not supply enough instances. Delays continued to climb as we investigated. At 13:13, we began examining the logic that chooses a different instance type or region when our first choice is unavailable. At 13:24, we suspected that a defect in that logic was stopping us from using other instance types in our primary region, so we disabled our fallback region and confirmed the cause of the defect. Engineers began working on a fix and deployed it at 14:49. The service began requesting other instance types and started machines in large numbers. At 14:58, the queue of jobs waiting for a machine dropped steadily. Average wait times across resource classes continued to drop over the next hour as we worked through the backlog of waiting jobs. At 16:00, the queue of jobs waiting for a machine cleared and the number of waiting jobs returned to normal. We moved the status page to monitoring at 16:09 and resolved the incident at 16:35. ## **Future Prevention and Process Improvement** We are taking the following steps to strengthen the resilience of job start times, including supply diversification, fallback logic, and earlier detection. **We are expanding our compute regions.** In the October 7 and October 8 incidents, our cloud provider could not supply enough instances in the regions we used. We are adding new regions to our fallback logic in order handle spikes for demand that outpace our preferred regions. **We are fixing the logic that picks a different instance type.** On October 8, a defect kept the machine provisioning service from using other instance types in the same region. We deployed a fix during the incident and are now reviewing and refactoring this logic as a whole, with tests, so that a shortage of one instance type moves work to another. **We are expanding the instance types we offer and securing additional capacity.** We are working with our cloud provider to reserve additional capacity in our existing regions, and we are adding instance types that jobs can run on. Industry demand for high-performance compute is growing. We are diversifying where and how we source capacity so our customers are insulated from supply variation. **We added a second Gen 2 Docker cluster so traffic bursts are distributed across two clusters.** We are also adding alerts on scheduling server disk saturation and on how fast the scheduler places jobs, and an alert that fires when a server starts without its fast disks in use. We made the disk setup script fail with an error, and we restored log shipping from these servers so that a similar problem is visible sooner. Customer experience is our top priority, and we commit to continually improving the reliability of our systems to match the trust that our customers place in us. We thank our customers for their patience while our team worked to resolve these incidents. Please reach out to our support team with any questions or concerns.
Resolved
WorkOSElevated errors across SSO and AuthKit sign-in
From 20:20 to 20:21 UTC on October 9, 2026 some SSO and AuthKit sign-in requests failed with HTTP 500 errors. The failures hit SSO token exchange and AuthKit authenticate requests. A database maintenance operation conflicted with live sign-in traffic. The operation self-corrected and service recovered shortly thereafter with no data loss. End users who saw an error could sign in again by retrying. We are updating the maintenance job to avoid live traffic disruption.
Resolved
TemporalElevated error rates for cross-region traffic between US AWS regions
Between 18:41 and 18:54 UTC on October 9, 2026, cross-region traffic to and from AWS us-east-1, us-east-2, and us-west-2 may have seen increased error rates. The issue has been resolved.
Resolved
ResendDelayed new team provisioning and contact webhook delivery
We have resolved the underlying issue and service has been resumed.
Resolved
BasetenElevated errors on new model builds. Existing models are not affected
This incident has been resolved.
Resolved
DockerHub Registry Authenticated Actions Failing
All functionality is restored on Hub Registry.
Resolved
GitHubActions Job Delays
On October 1, 2026, GitHub Actions experienced two periods of delayed and failed job starts on GitHub-hosted runners, beginning at approximately 14:15 UTC. Across the 14:15–17:56 UTC incident-response window, approximately 0.68% of workflow runs experienced start delays, and 0.196% of jobs failed to start. <br /><br />A routing change at our cloud provider concentrated API traffic on a single backend, causing an increase in rate-limit errors. Incorrect retry behavior in the provider’s inventory service added load to the affected APIs, compounding the rate limiting. This disrupted runner provisioning and reduced available capacity in affected regions. The resulting increase in demand in unaffected regions exposed a concurrency limit in our job-assignment system, delaying jobs even where runners were available. <br /><br />During the response, we increased job-assignment concurrency, scaled runner capacity, and worked with our cloud provider to reduce API load by excluding our runner subscriptions from its inventory service. By approximately 16:15 UTC, jobs were starting and executing as expected. We continued mitigation and monitoring until the incident was resolved at 17:56 UTC. <br /><br />Our cloud provider has since increased resource-management API rate limits for our workloads. To reduce the likelihood and impact of similar incidents, we are also adding more granular rate-limiting alerts and updating our job-assignment system to handle increased demand across fewer available regions.
Resolved
CursorInvestigating service degradation — Cloud Agents
This incident has been resolved.
Resolved
Augment CodeMajor Outage — Services Unavailable
We have successfully brought back all the services. We’ll be doing a detailed postmortem to better understand what caused the issue so that we can prevent this in the future.
Resolved
WorkOSElevated API and AuthKit errors for some requests (Columbus, OH region)
On October 1, 2026, from 11:34 to 11:40 AM EDT, our CDN provider had a network problem at its Columbus, Ohio location. Some requests to the WorkOS API (api.workos.com) and to AuthKit-hosted pages failed with HTTP 500 or 403 errors. Only requests routed through that location were affected. At the peak, about 8% of all API requests failed, including Audit Logs event ingestion, authentication and Vault decrypt calls. WorkOS systems stayed healthy the whole time and the provider fixed the problem by 11:40 AM EDT. Audit Logs event ingestion (POST /audit_logs/events) was the most affected endpoint: up to 21% of event submissions failed at the peak. Events from failed requests were not recorded. If your application did not retry them, resend any audit log events submitted between 11:34 and 11:40 AM EDT.
Resolved
CloudflareCloudflare DNS Resolver 1.1.1.1 and Gateway DNS are having issues in SIN
This incident has been resolved.
Resolved
HarnessSingle Customer: IaCM pipeline executions failing intermittently in Prod 1
## Summary On **Tuesday, 6 October 2026**, IaCM pipelines that use the **OpenTofu plugin** in **Prod1** failed intermittently for a production accountAffected runs injected secrets successfully, then later steps timed out or failed authentication. The same pipeline often succeeded on retry, with no configuration change. Typical failing executions ran for about 90 minutes. Other Harness modules were not broadly impacted. We have no report of customer data loss. Pipelines that finished inside the token lifetime were unaffected by this defect. ## Root Cause The IaCM stages minted authentication tokens that expired after one hour. Policy checks later in a long run then rejected the token, so the step timed out or failed authentication. That one-hour lifetime was fixed. Runs of about 90 minutes crossed the one-hour mark, which is why the failures showed up on longer pipelines and often cleared on a shorter retry. A separate set of plan errors in the same account came from credential setup in the customer workspace and is outside this incident. ## Impact * **Product area:** IaCM pipelines using the IaCM plugins talking to OPA sdk. * **Customer experience:** intermittent timeouts and authentication failures after secret injection, on runs longer than about one hour. * **Environment:** Prod1, one production account * **Data:** No data loss reported. ## Remediation * **Immediate:** Opened a status page for the affected Prod1 account, and shipped an interim IaCM manager change for one class of the observed failures while the plugin fix was built. * **Permanent:** Shipped an OpenTofu plugin hotfix so token expiry is configurable. In production the lifetime was raised to about 6 hours, so a long pipeline does not outlive the token. The hotfix was validated in QA and deployed to Prod1. We apologise for the disruption caused. Please reach out to your Harness support contact if you have any further questions.
Resolved
CloudflareNetwork Performance Issues in Queretaro
This incident has been resolved.
Resolved
BitbucketJob processing and scheduling is degraded affecting multiple Atlassian products
### Summary On October 1, 2026, between 01:00 and 07:25 UTC, degraded job processing and scheduling impacted Atlassian products including Bitbucket Cloud, Confluence, Forge apps, Jira, and Jira Service Management in all commercial cloud regions. This caused delayed or missed background tasks, such as email-to-ticket processing and scheduled pipelines; interactive product use was not affected. The event was triggered when an unexpected volume of scheduled tasks became due simultaneously, overwhelming our shared job scheduling platform. Automated monitoring detected the incident within three minutes. Recovery began around 03:45 UTC as engineering scaled capacity and the platform worked through the backlog, with full resolution reached in 6 hours and 25 minutes. ### IMPACT Most scheduled tasks stopped running at 01:00 UTC and resumed from about 03:45 UTC. Most delayed work had caught up by about 06:00 UTC, and the remaining delays had cleared by 07:25 UTC. Across affected products: - Recurring scheduled tasks skipped or delayed runs due during the incident; skipped runs were not automatically re-run, and tasks resumed at their next scheduled time. - Most one-time scheduled tasks ran late; a small number may not have run. - Automation rules, including scheduled rules, ran normally, except as noted below. Bitbucket Cloud Scheduled pipelines due between 01:00 and about 03:50 UTC did not run. If a scheduled pipeline did not run during the incident, you can wait for the next scheduled execution or trigger it manually. Other pipeline runs were not affected. Confluence Some calendar event reminders due between 01:00 and about 03:45 UTC were not sent. Forge apps Scheduled triggers stopped at 01:00 UTC and resumed from about 04:00 UTC; missed invocations were not replayed. Goals Start of the month email reminders to update goals could be delayed. Jira and Jira Service Management Incoming emails stayed in their mailboxes and were processed automatically after recovery. Scheduled notifications, such as filter subscriptions, were delayed. In some service projects, automation rules and notifications triggered by SLAs or time in status ran late. SLA timer data remained intact. Atlassian Support Until 07:25 UTC, issues with our support tools may have slowed some of our responses. Based on our investigation to date, we have not identified any loss of work items, pages, repositories, or incoming emails. ### ROOT CAUSE Our shared job scheduling platform operates at high scale under normal conditions. One of the platform consumers scheduled an abnormally high volume of tasks to be executed at the same time on October 1st at 01:00 UTC. This order-of-magnitude surge exceeded our architectural and scaling limits, creating a processing backlog across dependent services. Engineering manually increased capacity to handle the abnormal load, enabling the service to process the backlog and return to normal operations. ### REMEDIAL ACTIONS PLAN & NEXT STEPS We know that delayed and missed work affects your productivity. We are prioritizing these actions to help reduce the likelihood and impact of similar incidents: - **Enhance workload isolation:** Implement fair scheduling guardrails so volume surges from a single source do not starve or delay tasks across other products. - **Strengthen surge controls:** Add operational capabilities to quickly throttle or pause scheduled workloads from specific sources during unexpected traffic spikes. - **Improve system scaling and recovery:** Enhance platform capacity and optimize scaling processes to handle extreme demand spikes and recover more rapidly. We apologize to customers whose services were impacted during this incident. Thanks, Atlassian
Resolved
ConfluenceJob processing and scheduling is degraded affecting multiple Atlassian products
### Summary On October 1, 2026, between 01:00 and 07:25 UTC, degraded job processing and scheduling impacted Atlassian products including Bitbucket Cloud, Confluence, Forge apps, Jira, and Jira Service Management in all commercial cloud regions. This caused delayed or missed background tasks, such as email-to-ticket processing and scheduled pipelines; interactive product use was not affected. The event was triggered when an unexpected volume of scheduled tasks became due simultaneously, overwhelming our shared job scheduling platform. Automated monitoring detected the incident within three minutes. Recovery began around 03:45 UTC as engineering scaled capacity and the platform worked through the backlog, with full resolution reached in 6 hours and 25 minutes. ### IMPACT Most scheduled tasks stopped running at 01:00 UTC and resumed from about 03:45 UTC. Most delayed work had caught up by about 06:00 UTC, and the remaining delays had cleared by 07:25 UTC. Across affected products: - Recurring scheduled tasks skipped or delayed runs due during the incident; skipped runs were not automatically re-run, and tasks resumed at their next scheduled time. - Most one-time scheduled tasks ran late; a small number may not have run. - Automation rules, including scheduled rules, ran normally, except as noted below. Bitbucket Cloud Scheduled pipelines due between 01:00 and about 03:50 UTC did not run. If a scheduled pipeline did not run during the incident, you can wait for the next scheduled execution or trigger it manually. Other pipeline runs were not affected. Confluence Some calendar event reminders due between 01:00 and about 03:45 UTC were not sent. Forge apps Scheduled triggers stopped at 01:00 UTC and resumed from about 04:00 UTC; missed invocations were not replayed. Goals Start of the month email reminders to update goals could be delayed. Jira and Jira Service Management Incoming emails stayed in their mailboxes and were processed automatically after recovery. Scheduled notifications, such as filter subscriptions, were delayed. In some service projects, automation rules and notifications triggered by SLAs or time in status ran late. SLA timer data remained intact. Atlassian Support Until 07:25 UTC, issues with our support tools may have slowed some of our responses. Based on our investigation to date, we have not identified any loss of work items, pages, repositories, or incoming emails. ### ROOT CAUSE Our shared job scheduling platform operates at high scale under normal conditions. One of the platform consumers scheduled an abnormally high volume of tasks to be executed at the same time on October 1st at 01:00 UTC. This order-of-magnitude surge exceeded our architectural and scaling limits, creating a processing backlog across dependent services. Engineering manually increased capacity to handle the abnormal load, enabling the service to process the backlog and return to normal operations. ### REMEDIAL ACTIONS PLAN & NEXT STEPS We know that delayed and missed work affects your productivity. We are prioritizing these actions to help reduce the likelihood and impact of similar incidents: - **Enhance workload isolation:** Implement fair scheduling guardrails so volume surges from a single source do not starve or delay tasks across other products. - **Strengthen surge controls:** Add operational capabilities to quickly throttle or pause scheduled workloads from specific sources during unexpected traffic spikes. - **Improve system scaling and recovery:** Enhance platform capacity and optimize scaling processes to handle extreme demand spikes and recover more rapidly. We apologize to customers whose services were impacted during this incident. Thanks, Atlassian
Resolved