Delay in CI job processingDegraded Performance

Resolved

Started

Lasted 1h 40m

View official incident ↗

Updates

  1. Resolved

    We are no longer observing delays in CI job execution and the backlog of pending jobs has been worked through. We'll now be marking this as resolved.

  2. Monitoring

    CI job execution delays have improved significantly, and the backlog of pending jobs is shrinking. We're continuing to monitor the situation.

  3. Investigating

    We're still observing elevated CI job queues and are working to address the problem. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22897

  4. Investigating

    We're observing an increase in CI job queues. Users may see jobs stuck in a pending state for longer than usual. Jobs may not fire their associated webhooks or integrations. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22897