Fireworks AIService Degradation for Qwen 3.8 Max
We've resolved the incident and the affected service is now operational.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
We've resolved the incident and the affected service is now operational.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
HMRC have resolved the issue causing UK customers are experiencing slowness and error pages when using MTD ITSA or UK VAT.
Resolved
This incident has been resolved.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
The issue is resolved and audit logs are delivered as expected again.
Resolved
We can confirm that the issue was mitigated at 03:30 UTC on 28 August 2026, and the service has resumed normal operation. Customers and partners can view additional details about the incident by logging in to: https://community.akamai.com/customers/s/feed/0D5a700001QvOI6CAN or reaching out to Akamai Support. We apologize for the impact and thank you for your patience and continued support. We are committed to making continuous improvements to make our systems better and prevent a recurrence of this issue.
Resolved
Issue resolved.
Resolved
This incident has been resolved.
Resolved
🛠 We have resolved all known negative service impacts.
Resolved
Audio generation has been restored and is operating normally.
Resolved
This incident has been resolved.
Resolved
## Service Impact Customers experienced a site-wide disruption affecting the Buildkite web interface, REST API, Agent API, job queue, and notifications. The disruption began at 22:47 UTC and continued until 23:16 UTC. During this period, all customers were unable to create new builds via the UI or REST API, and job dispatch was delayed. We continued to receive webhooks from source control providers, but processing of those webhooks was delayed. New build processing resumed at 23:10 UTC and the accumulated backlog of work was processed by 23:16 UTC. Some builds that were already in progress at the beginning of the impact period remained temporarily stuck until services recovered. Some customers also experienced errors in the web UI, and on receiving webhooks from SCM providers. This impacted <1.5% of all requests during the impact period. ## Incident Summary **Background** Buildkite services running in one of our production Kubernetes clusters depend on an internal DNS service CoreDNS to locate databases, queues, and other application components. Over the last four months, we have been migrating our production workloads from AWS ECS to this AWS EKS cluster. The cluster has been steadily increasing in size during the course of this migration. Additionally, Buildkite recently moved time-sensitive notification jobs from a general-purpose pool of background workers into a new low-latency worker pool. To ensure sufficient capacity for both pools, we initially configured each with the same high maxReplica count as the original shared pool, with the intention of reviewing and adjusting the limits for both pools downwards at a later date. **Trigger: A sudden increase in demand on CoreDNS** At 22:44 UTC, an application deploy created a surge in application Pod volume, which caused our EKS cluster to scale out. This surge consumed the available headroom on already-deployed Nodes, which limited applications’ capacity to autoscale promptly. Some background workers \(including the aforementioned notification workers\) were also attempting to scale out at this time. The headroom shortage and subsequent cluster autoscaling delayed provisioning of the compute requested by those services. Since there had been no change in the metrics that triggered the services to scale up, those services requested even more Pods. Normally the impact of such runaway autoscaling would be limited by the services’ configured maximums. However, as mentioned previously the maximums for these services had been set higher than usual. The application deployment and runaway autoscaling combined to trigger an unusually high rate of change to applications, network endpoints, and cluster nodes. **Why CoreDNS failed** The cluster's CoreDNS service was running at a fixed size and did not automatically scale with the size or rate of change of the cluster. During post-incident analysis we discovered some bugs and gaps in our monitoring of CoreDNS, particularly around query volume and duration. * We found a defect in a key monitoring query which had masked an upward trend in query duration, correlated with increasing cluster size. * We also found that DNS query volume was not sufficiently monitored, and had been trending upwards as we migrated more workloads into EKS. These defects masked the upward trend in latency on the CoreDNS side; the client-side latency increase was offset by recent performance gains in our applications, and so did not catch our attention. Hence, we had not correctly prioritised our planned implementation of autoscaling for CoreDNS. **What happened when CoreDNS failed** During the impact period CoreDNS’s completed query rate remained stable, but query processing time increased from under 1ms to approximately 780ms. Pending requests accumulated in memory, until all three original CoreDNS pods exceeded their allowed memory limits and were restarted by Kubernetes. Continued demand and DNS retries prevented the service from recovering after restarting. CoreDNS unavailability caused failures across APIs, job dispatch, and notifications for all customers. Retries and delayed work increased the load during recovery. As part of the migration from ECS to EKS, database queries for a subset of customers were routed to PgBouncer instances running in the EKS cluster. During the impact period, these customers’ requests to webhooks and the Web UI received error responses. It was less than 1.5% of all requests during this incident that returned errors. **How we responded** We paused further application deployments, deployed more CoreDNS service replicas, raised the memory available to each CoreDNS replica, and expanded the node pool available to run them. The new set of CoreDNS Pods came into service by 23:12 UTC. After that, DNS errors fell rapidly. Customer-facing services processed the accumulated backlog and recovered fully by 23:16 UTC. ## Changes we're making * **Increase immediate CoreDNS headroom.** We increased CoreDNS from three to twelve pods, raised each pod’s memory limit from 512 MiB to 2 GiB, and expanded the system node pool available to run them. * **Enable automatic CoreDNS capacity scaling.** We will enable EKS-managed CoreDNS autoscaling, with a tested minimum replica count and sufficient capacity to distribute those replicas across nodes and availability zones. * **Detect both rapid degradation and declining headroom.** We will add direct alerts for CoreDNS query duration, query volume, goroutine growth, memory pressure, OOM restarts, and available replicas. We will also monitor longer-term trends as part of capacity planning. * **Apply the same standard to other cluster-critical services.** We will identify and remediate critical services that lack tested capacity, direct alerting, failure-domain distribution, and either safe autoscaling or documented static headroom.
Resolved
## Summary * Starting at **23:42 UTC** on August 23, 2026, several FME customers reported failures loading the FME UI. * FME UI Artifacts served from the CDN expired due to a retention policy, causing FME UI to fail to load. * Any flag request changes through the API, change delivery, and the data pipeline continued to work with no interruption. ## Root Cause * The FME UI is served from a CDN. The UI artifacts got evicted due to a retention policy, causing the UI to fail to load for all users. ## Impact * The FME UI was unable to load for all users across all production environments. ### What was not impacted? * SDK functionality and runtime flag evaluation * Admin API calls * Customer flag configuration data * No data loss occurred ## Remediation * FME UI got restored in the CDN through a deployment * Recovery confirmed across all production environments before closing the incident. ## Action Items * Improve the asset retention policy so that the currently active version is never subject to eviction.
Resolved
**Summary** On 19 August 2026 between 12:35 and 17:29 UTC, the Harness Application Security service experienced a significant disruption affecting both the customer-facing console and the data ingestion pipeline in the SaaS Production and US1 regions. **Root Cause** The internal configuration service that supplies runtime settings to nearly every other component became overloaded and entered a repeated restart cycle. Because so many services depend on it, the effects were broad: console pages such as protection policies, posture views, activity logs, API inventory, and custom policy failed to load or timed out, and downstream processing stalled while waiting for configuration it could not obtain. # **Customer impact** | **Dimension** | **Detail** | | --- | --- | | Console \(UI\) impact | Multiple pages failed to load or timed out, including protection policies, posture event pages and posture views inside dashboards and insight pages, activity log queries, API inventory screens, custom policy, and sensitive-data views and widgets. | | Ingestion impact | Security telemetry processing degraded severely and, in some paths, stopped entirely. Consumer lag grew across normalisation, grouping, anomaly detection, generation, and related processing stages. | | Data loss | A subset of telemetry ingested during the disruption was permanently dropped. | **Mitigation** Several intermediate mitigations additional CPU and memory, relaxed health-check thresholds, a database restart, and a larger connection pool ameliorated the issue. Disabling the new feature in both affected regions restored throughput sharply and durably. The incident was resolved at 17:29 UTC. # **Preventive actions** The following actions are committed and tracked internally to completion. The feature that triggered this incident remains disabled and will not be re-enabled until the work below is complete and validated. | **Action** | | --- | | | | OPtimize the code by tuning parameters such as cache eviction and retention , evaluate cursor-based pagination for bulk rule retrieval as rule counts grow | | Add a purpose-built database index for the service-scoping access pattern | | Remediate pipeline recovery semantics so consumers replay safely after position-marker loss instead of skipping backlog | | Mandate staged rollout for configuration overrides that alter downstream request patterns: low-volume cluster, then mid-volume, then high-volume | | Add backpressure and concurrency protection to the configuration service: circuit breaking, bounded queues, and timeout isolation | | Enhance observability by Instrumenting more detailed metrics |
Resolved
-- Only WHN customers are impacted -- Around 2026-08-26 2:30 PT (16:30 UTC), we were experiencing a delay in processing experiment and feature gate exposure data for Warehouse-Native (WHN) customers. As a result, experiment and feature gate results may show fewer participants than expected for August 25–26, and cumulative metric totals from August 25 onward may appear understated. Live event ingestion and the Exposure Stream are unaffected. Data is delayed, not lost. As of 2026-08-27, 06:53 PT, our team has identified the root cause and actively worked to backfill affected data and restored normal processing. The underlying data has now been reprocessed and delivered. - If your experiments are on a scheduled reload, no action is needed — the next scheduled run recalculates the affected dates automatically and restores the missing participants. Results should be complete after that run. - A reload is required in two cases. If an experiment/gate has already been stopped or concluded, its results will not update on their own and need a full reload. If you reload manually rather than on a schedule, please run a reload before 28 August so the recalculation covers both affected dates; after that a full reload is needed instead of an incremental one.
Resolved
The Apple bounce levels have maintained at pre incident levels during our monitoring phase. Our clean up of the bad suppressions has finished. This incident is resolved.
Resolved
All impacted services have now fully recovered.
Resolved
Service has remained stable in the Toronto (CA) data center. This issue is now resolved. If you are still seeing issues with your website please contact our support team. Thank you for your patience and we do apologize for any disruption to your business.
Resolved
This incident has been resolved and the affected services have been restored.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Email sending on behalf of Lovable apps has returned to normal operation.
Resolved
We have deployed a fix to prod and confirmed that Terminal inside sessions work as expected.
Resolved
This incident has been resolved. Sign-in to Cursor has returned to normal operation.
Resolved
This is now resolved
Resolved
The fix has remained stable across all regions, and the issue has been resolved.
Resolved
We have confirmed that all services are operating as normal.
Resolved
The issue has been resolved and the service is back to normal. Please reach out to our support if you experience any further issues.
Resolved
Our latency is now back to normal. Incident resolved
Resolved
Performance remains stable and prevention steps are being implemented. Find more information here: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22807
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has now been resolved.
Resolved
On August 24th, 2026, between approximately 06:35 and 07:25 UTC, the Copilot service experienced a degradation of the Claude Fable 5 model due to an issue with our upstream provider. Users encountered elevated error rates when using Claude Fable 5, with requests sometimes failing mid-response. No other models were impacted.<br /><br />The issue was resolved by a mitigation put in place by our provider. GitHub is working with our provider to further improve the resiliency of the service to prevent similar incidents in the future.
Resolved
The mitigation is in place and errors rates have returned to normal. We will continue to monitor.
Resolved
Issue was resolved
Resolved
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved