CI pipelines processing delaysDegraded Performance

Updates

1. Resolved

   7 months ago

   Pipeline job metrics have returned to a healthy state and jobs are being picked up and processed normally. We consider this incident resolved. Follow the incident: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21835

2. Monitoring

   7 months ago

   We’ve rolled out an infrastructure change and pending job metrics are improving. We’ll continue to monitor. Follow the incident: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21835

3. Identified

   7 months ago

   We are seeing infrastructure capacity errors and we’re inclined to treat this as the likely cause; investigation continues. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21835

4. Investigating

   7 months ago

   We’ve identified some potential causes but haven’t yet confirmed the root cause of the problem. Investigation continues. Track progress: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21835

5. Investigating

   7 months ago

   We are continuing to investigate the issue; there are no material updates to share at this time. Follow https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21835 for details.

6. Investigating

   7 months ago

   We are investigating an issue with pipelines where jobs are not being picked up or they are stuck in pending. See issue for more details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21835