Customers in us-central1 and asia-southeast2 may experience a delay in workflow progress

Resolved

Major impact

Started

Lasted 10h 30m

View official incident ↗

Updates

  1. Resolved

    The environment has been validated and all residual issues have been cleared. This incident is now resolved.

  2. Monitoring

    A configuration change affecting namespaces in the us-central1 and asia-southeast2 regions resulted in some workflow requests not being processed as expected. The change was rolled back and workflow processing resumed. We are continuing to validate the environment and clear any residual issues, and will provide further updates as that work completes.

  3. Monitoring

    We are continuing to monitor for any further issues.

  4. Monitoring

    While a small number of customer environments may still see some issues, mitigation has been in place and we continue to actively monitor.

  5. Monitoring

    We are continuing to monitor for any further issues.

  6. Monitoring

    The issue has been mitigated (as of 21:55 UTC) and we are actively monitoring the previously impacted environments.

  7. Identified

    Some customers with namespaces in the us-central1 and asia-southeast2 regions may experience errors and blocked progress on their workflows. We are currently rolling out a fix.