Slowness in Prod1 and Prod2 environment
Postmortem
Minor impact
Updates
Postmortem
### Summary On September 8, 2026, customers in Prod 1 and Prod 2 experienced elevated platform latency and pipeline failures. The issue was caused by a regression in a newly released capability that triggered cascading failures under high load. Because the capability was behind a feature flag, it was quickly disabled, and service was restored after a brief monitoring period. ### Customer Impact * Customers encountered slowness and failures during pipeline execution and UI operations. Some API calls returned errors or timed out. * No data loss or corruption occurred. ### Root Cause The new capability introduced a regression that created contention on a shared backend resource used by multiple Harness components. This saturated the shared platform infrastructure and caused the cascading failures. ### Mitigation * Disabled the capability across all environments * Temporarily increased platform capacity to restore stability ### Next Steps To prevent recurrence, Harness will: 1. **Permanently fix the capability** by profiling and eliminating the sub-optimal code path and query 2. **Improve detection** by enhancing alerting for resource-intensive queries on high-frequency platform paths
Resolved
This incident has been resolved.
Monitoring
A fix has been implemented and we are monitoring the results.
Monitoring
A fix has been implemented and we are monitoring the results.
Identified
We are continuing to work on a fix for this issue.
Identified
The issue has been identified and a fix is being implemented.
Investigating
We are continuing to investigate this issue.
Investigating
We are continuing to investigate the issue.
Investigating
We are continuing to investigate this issue.
Investigating
We are currently investigating this issue.