Issues with pipelinesDegraded Performance

Updates

1. Resolved

   3 months ago

   We've applied mitigations that have stabilized the job queue. The issue is now resolved. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22654.

2. Identified

   3 months ago

   We've identified a likely root cause, and the job queue has remained healthy. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22654.

3. Investigating

   3 months ago

   We've disabled a worker, which should let the job queue begin recovering while we continue investigating the root cause. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22654.

4. Investigating

   3 months ago

   We're investigating reports of pipelines failing and becoming stuck. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22654.