Jobs not being picked up/stuck pendingPartial Service Disruption

Resolved

Started

Lasted 1h 38m

View official incident ↗

Updates

  1. Resolved

    We've continued to monitor the job queues following the additional provisioning of runners to the resource pool and we observe that the queueing is back to normal. This incident is moved to resolved. More details about the incident here: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22145

  2. Monitoring

    We're seeing improvements in job queueing after additional runner allocations to the runner resource pool. We will continue to monitor until the the jobs are getting picked up at normal rates. More details about the incident here: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22145

  3. Identified

    We've identified the root cause of some jobs not getting picked up due to runner pool exhaustion. We're working on getting more runners added to the pool. More details about the incident here: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22145