PulumiNeo Outage
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
PLN Payments have experienced settlement delays. This is now resolved. Impact start time: 11/09/2026, 23:35 UTC Impact end time: 12/09/2026, 00:05 UTC Client actions: None
Resolved
This incident has been resolved.
Resolved
We are pleased to inform you that the incident affecting our Compute - Instance offering has been resolved. <b>Start time :</b> 10/09/2026 22:13 UTC <b>End time :</b> 11/09/2026 21:00 UTC <b>Root Cause :</b> A service disruption occurred due to an unexpected underlying infrastructure malfunction. We apologize for any inconvenience caused and appreciate your understanding.
Resolved
This issue is now resolved!
Resolved
## Summary Between approximately 7:49 AM and 10:57 AM PDT on September 1, 2026, a network fault in a Google Cloud Platform region disrupted connectivity to infrastructure that powers a portion of Imgix's image and video rendering pipeline. This resulted in an average error rate ~2% of total requests served during that window. Previously cached images and videos continued to be served normally throughout the incident. ## What Went Wrong Google Cloud Platform experienced a network-level fault affecting compute infrastructure in one of its US regions. The fault disrupted connectivity between the GCP services that make up our rendering infrastructure, preventing a share of render requests in that region from completing. Google has confirmed the disruption was caused by an incident that occurred during their network maintenance and is continuing to review the incident. We may update this report once we receive more information. Despite having multi-zonal configurations for our services, traffic was impacted across the whole region resulting in the degradation we experienced. As error rates rose and fell in waves throughout the incident, we performed a phased cutover of traffic to a healthy region to reduce impact, and returned traffic to normal once the issue cleared. ## What We Will Do To Prevent This In The Future We are refining our criteria and procedures for cross-regional traffic shifts to improve our responses in future incidents.
Resolved
We are no longer observing elevated error rates and have identified the root cause. We are marking this incident as resolved while we implement additional fixes behind the scenes.
Resolved
This incident has been resolved.
Resolved
The issue has been resolved.
Resolved
This incident has been resolved.
Resolved
This is resolved
Resolved
This incident has been resolved.
Resolved
Resolved
The metrics processing pipeline has caught up. No metrics were lost, this was only a delay in processing.
Resolved
The issue affecting Claude services has been resolved. Impact occurred from 13:57 to 14:20 UTC
Resolved
This incident has been resolved.
Resolved
Data Policies are now fully synchronized across all edge regions, and policy updates in the Radar/JSC Web Console have been re-enabled. The CSV import feature will remain temporarily disabled while improvements are implemented.
Resolved
We're pleased to inform you that the issue affecting Lumen by Talkwalker data retrieval from our media data provider has been successfully resolved. The Metabase API connection with the media data provider has been fully restored, and data flow (Print, TV, Online, and Social sources) has resumed as normal. Content published during the outage window has been backfilled. Thank you for your patience and cooperation throughout this incident.
Resolved
The issue causing the Windows Desktop App to crash has now been resolved and the app is working as normal.
Resolved
We are no longer seeing issues with Cloud SIEM Processing and normal operation has been restored at this time.
Resolved
This incident has been resolved.
Resolved
All impacted services have now fully recovered.
Resolved
We can confirm that the issue was mitigated at 00:10 UTC on 11 September 2026 and the service has resumed normal operation. Customers and partners can view additional details about the incident by logging in to: https://community.akamai.com/customers/s/feed/0D5a700001T8d9uCAB or reaching out to Akamai Support. We apologize for the impact and thank you for your patience and continued support. We are committed to making continuous improvements to make our systems better and prevent a recurrence of this issue.
Resolved
All impacted services have now fully recovered.
Resolved
This incident has been resolved.
Resolved
We are currently investigating this issue.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
On September 11 2026, between 5:30 UTC and 6:45 UTC, users experienced intermittent reliability issue that affected pull requests, commits view, related Bitbucket pages, and some Pipelines activity. Users may have also seen pages fail to load and timed out. The issue is now resolved and services have been restored to normal operation for all the affected customers.
Resolved
The environment has been validated and all residual issues have been cleared. This incident is now resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved
Resolved
## Service Impact _All times UTC unless stated otherwise._ Customers experienced elevated errors and latency across the Buildkite web interface, REST API, Agent API, and job queue during two periods: 18:43:00–19:21:00 and 20:02:00–20:28:00 on September 9, 2026. During these periods, customers encountered failed API requests, delayed job dispatch, and errors in agent operations including authentication, job acceptance, and artifact uploads. Some jobs impacted by the incident did not recover automatically. These jobs had to be manually cancelled and retried to complete successfully. Between 18:43:00–19:21:00: Agent API had an error rate up to 11.2% with no latency impact. REST API had a 9.7% error rate with no latency impact. Web request had a 2.6% error rate with no latency impact. Job queues were delayed by up to 3 minutes. Between 20:02:00–20:28:00: Agent API had an error rate up to 24.7% with no latency impact. REST API had a 4.9% error rate with no latency impact. Web requests had a 0.8% error rate with no latency impact. Job queues were delayed by up to 10 minutes. During this window, notifications were also delayed up to 1 minute. Core services fully recovered by 20:28:00. ## Incident Summary ### Background Several Buildkite services use a shared Redis cluster for coordination, caching, and agent-facing operations. This is one of the last pieces of non-shard aligned shared infrastructure, and is on our roadmap to address. AWS enforces a connection limit of 65,000 connections per node on this Redis cluster. As part of our ongoing migration from Amazon ECS to Amazon EKS, we reduced the number of Ruby threads in each Puma and Sidekiq process to their framework defaults. In ECS, we historically configured high thread counts, which caused resource contention and increased latency without a corresponding increase in throughput. We’d intentionally timed this work as part of the EKS migration because the migration already required us to redesign how these services are sized and scaled: EKS can respond directly to request queues and worker utilization, allowing us to run more, smaller application containers while preserving total capacity. This gives customers faster, more predictable API responses and background-job processing, particularly during periods of high load. ### What happened We began testing Puma Agent services on EKS in July and migrated traffic incrementally from late July, completing the migration earlier this week. Running more, smaller processes increased total Redis connection demand—a trade-off we had modeled, but not adequately accounted for before completing the migration. Before the incident, our shared services were already using much of the available connection capacity on the impacted Redis cluster, around 60-80% throughout a 24 hour window. At 18:43:10 we started seeing elevated HTTP request errors. At this time, our application reported a burst of Redis connection timeouts, all targeting a single replica in the cluster. Clients with active Redis connections then attempted to reconnect. Failed connections were retried, causing a surge in connection attempts that exhausted that same Redis host’s connection-tracking allowance. This caused the host to reject new connections from the Agent API’s application pods, which led to increased request latency for our Puma Agent service. At 18:43:30, the increased request latency from this network failure caused our application platform to rapidly start more Puma Agent containers to respond to the increase in request latency. Within a minute, we’d doubled our running pod count. Each additional container opened its own pool of Redis connections. At 18:44:00, several Redis nodes hit their limit of 65,000 connections, and latency of requests slowed further triggering more Puma Agent autoscaling. Once the Redis connection limits were reached, applications could no longer reliably connect to Redis. This caused failures across agent authentication, job assignment, artifact uploads, and other API operations. The failures then created a feedback loop: slower requests caused our shard aligned Puma Agent services to autoscale, and those additional containers opened more Redis connections, placing further pressure on Redis. By 18:50:00, our system stabilized at max, running about five times the number of pods we’d started with. This stable state allowed the autoscaling to start bringing pod counts back down. By 19:10:00 the error rate had returned to 0 and and services were no longer degraded. By 19:20:00 our pod counts had autoscaled down to more reasonable but still high numbers, and the Redis cluster was functioning again. At this point customers saw recovery. At 20:02:00, a routine application deployment temporarily introduced additional containers. Since the Redis cluster was already under strain, this quickly increased request latency which added autoscaling on top of the regular deployment surge numbers. This amplified the connection growth again and caused the second period of degraded service, lasting until 20:28:00. The influx of new pods resulted in a very large number of Envoy Gateway configuration updates, which in turn caused some network throttling of the underlying node. This caused packet drops and delays updating our Cilium Operator, which controls configuration for the cluster’s overlay network. This caused resulted some Buildkite Jobs to remain in a stuck state for some customers, requiring manual retries. ### How we responded When we saw that the Redis cluster had reached its capacity to handle incoming connections, we immediately began adding more nodes to the cluster, increasing incoming connection capacity by 33%. When we saw the impact of deployments, we immediately paused further application deployments while we investigated the connection growth. We increased the Puma Agent utilization autoscaling threshold, reducing the rate at which workloads scale in response to brief load increases. We also increased our scale up stabilization window to prevent future runaway autoscaling of our workloads. We reduced the configured maximum surge of our workloads during application deployments, to reduce pod churn impact on connections. Since Redis connections and latency remained stable, we then resumed deployments while continuing to monitor the cluster. ## Changes we're making **Reduce autoscaling sensitivity.** * We increased the utilization threshold that triggers the Puma Agent autoscaling, so short-lived latency increases do not cause rapid increases in application and Redis connection demand. * We increased our scale up stabilization window so that we only add more pods in response to scaling triggers in incremental bursts. This will throttle scaling up based on downstream latency and runaway scaling. **Reduce deployment-related load.** * We reduced the maximum deployment surge for Puma and Sidekiq workloads, limiting the additional capacity and downstream connections introduced during deployments. **Increase immediate Redis capacity by 33%.** * We added more nodes to the Redis cluster to handle more connections, given the shape of our connection load has shifted with our different thread counts in EKS **Reduce connection pressure from high-traffic workloads.** * We moved the two largest Puma Agent sharded deployments back to ECS while we evaluate how the threading changes impact Redis connections and our autoscaling sensitivity. **Bound aggregate Redis connection demand.** * We are reviewing Redis connection-pool sizing, application thread counts, and workload scaling limits so that application growth cannot exceed the available Redis connection capacity.
Resolved
This incident has been resolved.
Resolved
This issue is now resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved and the affected services have been restored.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Some S3 export files delivered between September 9 at 20:32 UTC and September 10 at 04:01 UTC were missing the bucket-owner-full-control permission, which prevented affected customers from reading them. The issue is resolved. Exports moving forward and outside this timeframe are not impacted. No data was lost. For impacted files in the affected timeframe, set your bucket's Object Ownership to Bucket owner enforced (disable ACLs), if compatible with your bucket access requirements. If you have questions or continue to experience issues, please submit a support request.
Resolved
Between 3:44 PM and 4:27 PM PDT on August 19, 2026 and 10:28 AM and 11:21 AM PDT on August 24, 2026, some users may have experienced intermittent errors with some Box features, including Logins, Uploads, Downloads, and Box Sign. The issues occurred after a rapid surge of external Box client connections overwhelmed front-end capacity in the affected region, which caused an increase in server-side error rates and prevented some requests from reaching our back-end services. We restored service by working with our cloud provider and applying operational mitigations until front-end error rates returned to normal and user requests succeeded again. # Analysis This incident highlighted opportunities to improve our detection and correlation of frontend/load balancer health signals with our application telemetry and to strengthen joint monitoring and mitigation processes with our cloud provider during large-scale client connection events. It also underscored the importance of clearer detection thresholds and runbook steps for frontend degradations, so we can escalate and remediate faster without over or under communicating impact to customers. # Corrective Actions Box has initiated the following corrective actions: * Improving monitoring and observability for large-scale client connection events and working with our cloud provider on coordinated mitigation strategies. * Coordinating with our cloud provider to ensure connection rates are within expected thresholds. We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. Sincerely, The Box Team
Resolved