Resolved incident

Harness Login Failure in EU1 environment

Harness · Major · Started

Duration
2h 34m
Severity
Major
Status
Postmortem

Summary

Major · Postmortem

### **Summary** During a planned security improvement to rotate service authentication secrets in the EU1 environment, an out of sequence secret value was inadvertently applied to a subset of internal services. This resulted in authentication failures between platform components, causing login failures. The issue was resolved by rolling back the token configuration to the previous known-good version and validating service recovery. ### **Impact** **Affected environment:** EU1 **Customer-visible impact: Login Failures** No customer data was lost or corrupted. ### **Root Cause** As part of a planned security initiative, authentication secrets used for communication between internal platform services were rotated in the EU1 . During the rotation process, an secret value was inadvertently misconfigured for one of the services. There was a newer format of secret and had to be applied in a certain order. This created a mismatch between services that authenticate using the shared token, causing authentication requests to be rejected with HTTP 400 errors. The authentication failures prevented CI components from obtaining log service tokens, disrupted delegate task creation, and prevented log streaming for affected pipeline executions. ### **Resolution** Engineering responded by: * Reverting the token configuration to the previous known-good secret. * Restoring consistent authentication across affected services. ### **Follow-up Actions** To reduce the likelihood of similar incidents in the future, Harness will: * Implement additional pre-rotation and post-rotation validation checks to verify that the correct secret values and formats are applied before changes are activated. * Introduce automated verification of authentication between dependent services immediately following secret rotations. * Strengthen operational safeguards and deployment validation for security credential rotation procedures to detect configuration mismatches before they can impact customer workloads.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    Harness

    Harness login is failing. We are investigating the problem.

  2. Monitoring

    Harness

    A fix has been implemented and we are monitoring the results.

  3. Monitoring

    Harness

    We are continuing to monitor for any further issues.

  4. Resolved

    Harness

    This incident has been resolved.

  5. Postmortem

    Harness

    ### **Summary** During a planned security improvement to rotate service authentication secrets in the EU1 environment, an out of sequence secret value was inadvertently applied to a subset of internal services. This resulted in authentication failures between platform components, causing login failures. The issue was resolved by rolling back the token configuration to the previous known-good version and validating service recovery. ### **Impact** **Affected environment:** EU1 **Customer-visible impact: Login Failures** No customer data was lost or corrupted. ### **Root Cause** As part of a planned security initiative, authentication secrets used for communication between internal platform services were rotated in the EU1 . During the rotation process, an secret value was inadvertently misconfigured for one of the services. There was a newer format of secret and had to be applied in a certain order. This created a mismatch between services that authenticate using the shared token, causing authentication requests to be rejected with HTTP 400 errors. The authentication failures prevented CI components from obtaining log service tokens, disrupted delegate task creation, and prevented log streaming for affected pipeline executions. ### **Resolution** Engineering responded by: * Reverting the token configuration to the previous known-good secret. * Restoring consistent authentication across affected services. ### **Follow-up Actions** To reduce the likelihood of similar incidents in the future, Harness will: * Implement additional pre-rotation and post-rotation validation checks to verify that the correct secret values and formats are applied before changes are activated. * Introduce automated verification of authentication between dependent services immediately following secret rotations. * Strengthen operational safeguards and deployment validation for security credential rotation procedures to detect configuration mismatches before they can impact customer workloads.