BraintrustGateway 5xxs
The faulty deploy was rolled back and 5xxs were mitigated.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
The faulty deploy was rolled back and 5xxs were mitigated.
Resolved
We are pleased to inform you that the incident affecting our Data & Analytics offering has been resolved. <b>Start time :</b> 25/08/2026 14:00 UTC <b>End time :</b> 27/08/2026 13:02 UTC <b>Root Cause :</b> This incident was caused by a software issue. We apologize for any inconvenience caused and appreciate your understanding.
Resolved
Latency has since decreased, and we are working on adding more alerts on our side to help us identify and respond to similar latency issues earlier.
Resolved
This incident has been resolved.
Resolved
The issue has been fully mitigated, and users should no longer be affected.
Resolved
This incident has been resolved.
Resolved
Between 11:00 and 11:15 UTC users experienced partial degradation with Gemma-4-31B. Service has been restored.
Resolved
Processing of uploaded test execution data has caught up and is back to normal.
Resolved
The Support-A-Creator website has been restored.
Resolved
The issue has been resolved. Our engineers are still monitoring our systems to assure the situation is completely mitigated.
Resolved
Our upstream provider mitigated a genuine distributed HTTP DDoS attack against gitlab.com, but incorrectly blocked a small amount of legitimate traffic. This has now been resolved.
Resolved
Starting at approximating 19:51 UTC, we saw elevated error rates (HTTP 5xx) on requests to the Embeddings Index, Agent Actions and AI Assist services. The issue was identified and resolved at roughly 20:32 UTC.
Resolved
The issue has been resolved.
Resolved
Starting at approximating 6:33 UTC , we saw elevated error rates (HTTP 5xx) on requests to the Embeddings Index service. The issue was identified and resolved at roughly 7:16 UTC.
Resolved
This incident has been resolved.
Resolved
The issue affecting login verification emails and codes has been fully resolved. We’re sorry for the trouble and appreciate your patience while we worked through it. If you're still experiencing issues, please reach out to us at hello@buffer.com.
Resolved
As of 11:26 PM US/Eastern on August 26, 2026, the repair to process affected flow messages from the window of impact has been completed, and flow sending has resumed as expected across all channels. We apologize for the disruption this caused. This incident has been resolved.
Resolved
On August 26, 2026 from 15:02 to 15:45 UTC, Actions jobs failed to start. The following 2 hours until 17:40 UTC, Actions runs were delayed starting by more than 5 minutes as the system caught up with delayed load. This impact was triggered by saturation of writes to the database primary used by the service processing triggers for Actions workflows. The primary was failed over, but the system did not fully recover. The saturation was caused by growing daily peak load combined with an upstream issue in GitHub’s event processing infrastructure, https://www.githubstatus.com/incidents/hcbtzksccj2f, which caused burst amplification of already-high load. Downstream throttles that were later used to recover were set ~10% too high to protect the system. <br /><br />At 15:45 UTC, throttling combined with service restarts recovered the service’s core health. Those throttles were gradually raised between 15:54 and 17:22 to restore full webhook processing for Actions runs. This ramp was deliberately slow to ensure we did not re-overwhelm the system given our original throttling was now known to be incorrectly set. The queue of webhook events was fully burned down at 17:40 UTC. <br /><br />3.7% of larger-runner jobs, along with some scale-set self-hosted jobs, remained stuck in queued or “waiting for runner” state. We deployed a change to force-revoke jobs in this state, and they transitioned to failed at 18:40 UTC, about 50 minutes after incident mitigation. Releasing these jobs also freed hosted concurrency for larger-runner jobs. <br /><br />Customers using concurrency groups saw longer impact due to a separate issue where runners assigned to a subset of jobs disconnected before the force-revoke mitigation was deployed, which prevented runner acquisition from progressing and left jobs in a waiting-for-runner state. This was resolved at 01:00 UTC on August 27. <br /><br />Some runs triggered during the 15:02-15:45 UTC incident window encountered a bug that left them showing as queued even after service recovery. In the backend, these runs had already failed and will automatically move to canceled state 24 hours after creation. As follow-up, we are fixing the root cause of this queued state and improving our ability to bulk-cancel affected runs. <br /><br />Several changes to improve the general scalability of this part of Actions were already complete and deploying to production. Rollout of those changes will be complete within the next 24 hours. Further work to improve scale, resiliency, and more graceful degradation of Actions workflows are in flight. We are also taking a repair item to accelerate clearing of stuck queued or waiting jobs in similar future cases.
Resolved
Mimir writes in the prod-us-central-0 region had elevated error rates for approximately 15 minutes from 1:05 to 1:20 UTC. The issue has been resolved and we are monitoring.
Resolved
This incident has been resolved.
Resolved
The issue has been resolved. A root cause analysis will be shared shortly.
Resolved
Instant trigger processing has returned to normal. We have successfully replayed the affected triggers and continue not to see any impacted Zap runs, so the incident is now resolved. If you have any questions, please reach out to our support team: [https://zapier.com/app/get-help](https://zapier.com/app/get-help)
Resolved
# Service Disruption on August 21 — What Happened On August 21, between 12:12 and 13:40 PDT, some customers experienced failed Chat and Agent requests. A routine software update exposed a dormant bug in how certain conversation histories were processed, causing roughly 12% of conversations to return errors; new conversations were largely unaffected. We rolled back the update within the hour and deployed a permanent fix the same day. Our internal review identified where our safeguards fell short, and our team is already putting in place a combination of agentic, programmatic, and human safeguards to ensure an issue like this cannot reach customers again. We're sorry for the disruption, and we thank the customers who reported the issue. Please reach out to your account contact or support if you have questions.
Resolved
The issue has been resolved
Resolved
This incident has been resolved. Between 12:00 and 12:12 PM PDT on August 26, server-side SDKs establishing new streaming connections experienced elevated error rates. Existing streaming connections were not affected, and polling requests continued to be served with error rates below 1%. Some requests to the LaunchDarkly web application also experienced elevated errors between approximately 11:29 AM and 12:03 PM PDT. The issue was caused by an internal configuration change, which has been reverted. All systems have recovered and are operating normally. No customer data was lost or compromised.
Resolved
The issue is resolved.
Resolved
This incident has been resolved.
Resolved
Between roughly 8/25 6PM to 8/26 2AM Pacific Time, some users were not able to access or use Cosmos due to seeing the 429 - Too Many Requests error. It was because of a bug that got deployed into production that called an endpoint recursively. At around 1:30AM PT, we deployed the revert of the change containing the bug to production, and new Cosmos tabs opened after this revert operated normally. We have also suggested impacted users to do a hard refresh on their browser so that they get the latest fix. Since then we have been monitoring and our metrics show that this error has died down for production usage.
Resolved
Hello. We can confirm that the issues affecting customers who use the Proofpoint integration have been resolved. You should now be able to publish content as normal. Thank you for your patience.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
A pathological query was impacting other query services. We've added protections against the pathological case and scaled up our query services. All services are now functioning as normal
Resolved
At about 8:30am UTC on 8/25/2026 a scheduled server rotation at https://hire.eulever.co/ caused increased latency and some failed page loads until 9:15am UTC. Impacted users would notice a "504 error" during the affected page loads. Service was restored to all users as of 9:15am UTC on 8/25/2026. An full RCA will be posted to this issue within 7 to 10 business days.
Resolved
# Summary On 20 August 2026, beginning at approximately 15:00 UTC, the Harness platform experienced widespread performance degradation across all production environments. Pipeline executions that normally complete in around two minutes took seven to ten minutes. Continuous Delivery, Continuous Integration, pipeline orchestration, and Feature Management & Experimentation were all affected. Google Cloud Platform experienced a multi-product incident in the us-west1 region affecting Bigtable, Compute Engine, Google Kubernetes Engine, and persistent-disk I/O. Harness production infrastructure runs on persistent disks in that region. The degradation raised database operation latency from approximately 2 ms to over 10 ms at the 95th percentile, which in turn caused message-queue processing lag and propagated to every service that depends on timely database access. # Impact This was a degradation, not an outage. Pipelines continued to execute and complete successfully throughout; they were slow rather than failing. No data was lost, and no customer work was dropped as a result of this incident. # **Root cause** Harness production infrastructure in the affected environments runs on Google Cloud Platform persistent disks in the us-west1 region. When that storage layer degraded, the effect propagated through the platform in a predictable chain: **Persistent-disk I/O degradation in us-west1.** Google Cloud Platform experienced a multi-product incident affecting Bigtable, Compute Engine, Google Kubernetes Engine, and persistent-disk performance. This was an infrastructure failure in the provider’s environment, outside Harness’s control. # **Preventive actions** Although Harness cannot prevent a cloud provider infrastructure failure. The actions below are aimed at detecting one faster and being better positioned to act on it. | **Action** | | --- | | Continue routine pre-testing of targeted cross-region database failovers, as performed during this incident, to keep failover readiness verified rather than assumed | | Assess full-stack multi-region failover readiness for future scenarios in which cross-region latency would be unacceptable |
Resolved
The issue preventing some Codex and ChatGPT users from seeing MCP tools, such as use_figma, has been resolved. The impact began at 10:14 UTC, with recovery beginning at 14:03 UTC.
Resolved
This incident has been resolved.
Resolved
Between 14:15 - 14:41 UTC Fin was slow to respond to conversations. We have identified the cause and do not expect recurrence. Apologies for the disruption. There are more details available in the attached root cause analysis document.
Resolved
The issue has been resolved and Shopify store creation is working again.
Resolved