Delay in background job processingPartial Service Disruption

Updates

1. Resolved

   8 months ago

   After monitoring the CI pipelines, we are marking the incident as resolved. More details https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562

2. Monitoring

   8 months ago

   Monitoring of CI pipelines shows metrics have returned to normal. We will keep monitoring the fix for the next hour. More details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562

3. Monitoring

   8 months ago

   We are observing improved system metrics and reduced user impact. Mitigations are pending deployment with the goal of preventing future recurrence. This incident will remain in monitoring status as we observe system behavior. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562

4. Identified

   8 months ago

   The primary fix has been merged and deployment is underway. Full rollout typically takes several hours, so we're pursuing additional mitigations and implementation changes in parallel. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562

5. Identified

   8 months ago

   We've identified a potential fix and are working to deploy it. We'll update you once we have a production deployment timeline. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562

6. Investigating

   8 months ago

   Our investigation is ongoing. Users may still observe delays in CI job status updates. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562

7. Investigating

   8 months ago

   We are investigating a delay in background job processing. Users may observe a delay in CI job status updates in the UI. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562