IDP Workflows and Backend failures
Postmortem
Minor impact
Updates
Postmortem
# **Customer RCA: Executive Summary** On June 17, 2026, between 9:30 PM and 10:15 PM IST, customers in certain production environments experienced failures when accessing IDP \(Internal Developer Portal\) workflows. The issue was caused by a configuration mismatch in service authentication settings. ## **Impact** * **Affected Service:** IDP workflows * **Duration:** Approximately 45 minutes \(9:30 PM - 10:15 PM IST\) * **Customer Impact:** Users were unable to load IDP workflows during this period. * **Data Loss:** None * **Security Impact:** None ## **Root Cause** During a planned service deployment, a configuration mismatch occurred in the authentication settings used for internal service-to-service communication. Specifically, the authentication credentials configured in the newly deployed service did not match the credentials expected by existing services in prod environments. This mismatch caused authentication failures when services attempted to communicate with each other, resulting in workflow loading failures for customers. ## **Mitigation** The issue was resolved by reverting the configuration change to restore the previous working authentication settings. The engineering team was proactively monitoring logs immediately after deployment and identified error patterns several minutes before the first customer report. ## **Action Items/Next Steps** 1. **Enhanced Pre-Deployment Validation:** Automated validation checks will be introduced to verify that all required configurations, secrets, and service dependencies are correctly provisioned and consistent across services before any production deployment. 2. **Improved Deployment Procedures:** Updated deployment checklists with explicit verification of service authentication settings across services before rollout.
Resolved
This incident has been resolved.
Monitoring
A fix has been implemented and we are monitoring the results.
Identified
IDP Systems' communication was interrupted due to an update. Harness is rolling back changes. Impacted environments - prod0, prod1, prod2
Identified
The issue has been identified and a fix is being implemented.
Investigating
We are currently investigating this issue.