Issue with processing CI jobsDegraded Performance

Resolved

Started

Lasted 1h 48m

View official incident ↗

Updates

  1. Resolved

    The job queue is now cleared and we do not expect any further customer impact. This is considered resolved. For more info, see: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22256

  2. Monitoring

    Error rates have recovered back to normal levels and we started processing jobs again. We expect that the job queue should be processed over the next 30 minutes. For more info, see: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22256

  3. Identified

    The change has been reverted and we are seeing that error rates are recovering. For more info, see: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22256

  4. Identified

    We are still working on rolling back the change that we believe is the root cause. This issue impacts GitLab Hosted Runners and Hosted Runners on Linux. For more info, see: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22256

  5. Identified

    We are still working on rolling back the change that we believe is the root cause. This issue impacts GitLab Hosted Runners and Hosted Runners on Linux. For more info, see: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22256

  6. Identified

    We have identified a recent change as a possible root cause and we are working on rolling back the change. For more info, see: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22256

  7. Investigating

    We are investigating an issue where CI jobs are not processing. For more info, see: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22256