Resolved incident
Some customers may be experiencing stuck workflows or activities
Temporal · None · Started
- Duration
- 332h 22m
- Severity
- None
- Status
- Resolved
Summary
None · Resolved
This incident has been resolved.
Event timeline
How this incident unfolded, from the official status page.
Investigating
Temporal
We are investigating a few reports where a workflow or activity is becoming stuck and not making progress when it should. Please report any such issues you see to support.
Investigating
Temporal
We are continuing to investigate this issue. Please report any stuck workflows or activities to support so we can help unblock them as we work on root cause investigation and resolution.
Investigating
Temporal
We are investigating a recent update that could have caused intermittent missed task dispatches, resulting in some tasks being stuck. Impact identified as of now is limited to a handful of workflows that were caught in a rare edge case. We will continue to share more updates as investigation progresses.
Investigating
Temporal
We're rolling back the recent update as a possible mitigation to avoid any further occurrences of this issue. We are however still in the process of investigating root cause and will post updates as we have them.
Identified
Temporal
A bug in the recent update in History pagination caused a rare edge case where an extra task could be added before the response was returned. This incorrectly advanced the read level, causing some tasks to be skipped. We’ve identified the issue, and are currently rolling back the update to stop this from happening again. We will be refreshing all executions in cells where the update was deployed and will monitor these closely. This will progress any stuck workflows, but will not impact any of the closed ones.
Identified
Temporal
Rollback is still in progress and we are continuing to monitor the situation.
Identified
Temporal
The rollback is making progress across impacted cells, and we are continuing to monitor the situation.
Identified
Temporal
Rollback is complete; refresh of open workflows is underway to unblock impacted workflows.
Identified
Temporal
Refreshes are still in progress and we are monitoring the situation.
Identified
Temporal
Workflows will continue to be refreshed overnight, and we are monitoring the situation.
Monitoring
Temporal
Workflows will continue to be refreshed overnight, and we are monitoring the situation.
Monitoring
Temporal
We are continuing to monitor the ongoing refresh process for all open workflows. We are also considering options for refreshing closed workflows. We will continue to share more updates as this progresses.
Monitoring
Temporal
As refresh of closed workflows progresses, some users may see unblocked stuck parent workflows from closed child workflows; this is happening because the missed task for closed child workflows is refreshed and delivered to parent. It could also happen on other stuck workflows that are waiting for task updates which were missed due to this incident.
Monitoring
Temporal
Refresh across open and closed workflows continues to progress smoothly. We're monitoring the situation and will post updates once complete.
Monitoring
Temporal
Refresh across open and closed workflows continues to progress smoothly. Once this is done, the incident will be resolved.
Monitoring
Temporal
Refresh operations across both open and closed workflows are complete in most regions; we continue to make steady progress across the remaining few namespaces in the us-east-1, us-west-2, us-east4, us-west1, us-central1, asia-south1, europe-west3. No additional impact has been observed since mitigation was implemented, and we continue to closely monitor the environment. The next update will be provided once the refresh process has completed.
Monitoring
Temporal
The last few namespaces are still refreshing. We are starting to patch the code, and are monitoring the situation.
Monitoring
Temporal
Refresh across all executions is nearing completion with no additional impact observed. Incident is fully mitigated and resolved.
Monitoring
Temporal
Refresh across all executions is nearing completion with no additional impact observed. Incident is fully mitigated and resolved.
Resolved
Temporal
This incident has been resolved.