Resolved incident

Slowness in Prod1 and Prod2 environment

Harness · Minor · Started

Duration
6h 38m
Severity
Minor
Status
Postmortem

Summary

Minor · Postmortem

### Summary On September 8, 2026, customers in Prod 1 and Prod 2 experienced elevated platform latency and pipeline failures. The issue was caused by a regression in a newly released capability that triggered cascading failures under high load. Because the capability was behind a feature flag, it was quickly disabled, and service was restored after a brief monitoring period.  ### Customer Impact * Customers encountered slowness and failures during pipeline execution and UI operations. Some API calls returned errors or timed out. * No data loss or corruption occurred. ### Root Cause The new capability introduced a regression that created contention on a shared backend resource used by multiple Harness components. This saturated the shared platform infrastructure and caused the cascading failures. ### Mitigation * Disabled the capability across all environments * Temporarily increased platform capacity to restore stability ### Next Steps To prevent recurrence, Harness will: 1. **Permanently fix the capability** by profiling and eliminating the sub-optimal code path and query 2. **Improve detection** by enhancing alerting for resource-intensive queries on high-frequency platform paths

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    Harness

    We are currently investigating this issue.

  2. Investigating

    Harness

    We are continuing to investigate this issue.

  3. Investigating

    Harness

    We are continuing to investigate the issue.

  4. Investigating

    Harness

    We are continuing to investigate this issue.

  5. Identified

    Harness

    The issue has been identified and a fix is being implemented.

  6. Identified

    Harness

    We are continuing to work on a fix for this issue.

  7. Monitoring

    Harness

    A fix has been implemented and we are monitoring the results.

  8. Monitoring

    Harness

    A fix has been implemented and we are monitoring the results.

  9. Resolved

    Harness

    This incident has been resolved.

  10. Postmortem

    Harness

    ### Summary On September 8, 2026, customers in Prod 1 and Prod 2 experienced elevated platform latency and pipeline failures. The issue was caused by a regression in a newly released capability that triggered cascading failures under high load. Because the capability was behind a feature flag, it was quickly disabled, and service was restored after a brief monitoring period.  ### Customer Impact * Customers encountered slowness and failures during pipeline execution and UI operations. Some API calls returned errors or timed out. * No data loss or corruption occurred. ### Root Cause The new capability introduced a regression that created contention on a shared backend resource used by multiple Harness components. This saturated the shared platform infrastructure and caused the cascading failures. ### Mitigation * Disabled the capability across all environments * Temporarily increased platform capacity to restore stability ### Next Steps To prevent recurrence, Harness will: 1. **Permanently fix the capability** by profiling and eliminating the sub-optimal code path and query 2. **Improve detection** by enhancing alerting for resource-intensive queries on high-frequency platform paths