Resolved incident

Customers in us-central1 and asia-southeast2 may experience a delay in workflow progress

Temporal · Major · Started

Duration
10h 30m
Severity
Major
Status
Resolved

Summary

Major · Resolved

The environment has been validated and all residual issues have been cleared. This incident is now resolved.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Identified

    Temporal

    Some customers with namespaces in the us-central1 and asia-southeast2 regions may experience errors and blocked progress on their workflows. We are currently rolling out a fix.

  2. Monitoring

    Temporal

    The issue has been mitigated (as of 21:55 UTC) and we are actively monitoring the previously impacted environments.

  3. Monitoring

    Temporal

    We are continuing to monitor for any further issues.

  4. Monitoring

    Temporal

    While a small number of customer environments may still see some issues, mitigation has been in place and we continue to actively monitor.

  5. Monitoring

    Temporal

    We are continuing to monitor for any further issues.

  6. Monitoring

    Temporal

    A configuration change affecting namespaces in the us-central1 and asia-southeast2 regions resulted in some workflow requests not being processed as expected. The change was rolled back and workflow processing resumed. We are continuing to validate the environment and clear any residual issues, and will provide further updates as that work completes.

  7. Resolved

    Temporal

    The environment has been validated and all residual issues have been cleared. This incident is now resolved.