CI Pipeline processing delaysDegraded Performance

Updates

1. Resolved

   8 months ago

   Our team has been monitoring and job processing has returned to normal. This incident is now resolved. Further details available in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21532

2. Monitoring

   8 months ago

   For clarification, it was noted earlier in the incident that we saw GitLab-hosted Runners were primarily impacted. However, the background job processing delay has impact on self-hosted runners as well. Our team has seen a recovery and we are still monitoring for stability.

3. Monitoring

   8 months ago

   The backlog has recovered and we are no longer seeing delays. Our investigation points to a Sidekiq backlog which lead to delays in CI job execution. Our team is currently monitoring. Additional details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21532

4. Investigating

   8 months ago

   Our team is seeing some recovery but are still investigating the root cause. We believe that a previous Sidekiq backlog caused CI jobs to accumulate, leading to delays. More detail in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21532

5. Investigating

   8 months ago

   Our team is still investigating the issue.

6. Investigating

   8 months ago

   No material updates at this time, our team is still investigating the issue.

7. Investigating

   8 months ago

   We're seeing increased job queuing rates that are primarily affecting small CI runners. Our team is still investigating.

8. Investigating

   8 months ago

   Our team is investigating potential delays with background job processing. This could impact CI/CD job processing.