CI/CD pipeline processing delaysPartial Service Disruption
Resolved
Updates
Resolved
The configuration change has been deployed and we have seen stability with Sidekiq and pipeline processing. This incident is now resolved.
Identified
We are continuing to monitor but the main backlog drivers are mitigated. Users should expect normal operations. Our team is currently deploying a configuration update to further address this. We'll provide an update once it is deployed.
Identified
Our team has turned off a specific worker to help improve pipeline processing. There has been significant recovery, and our team has identified this as a resource issue. We are monitoring for stability in addition to investigating the root cause and determining a permanent fix.
Investigating
Our team is considering increasing resources to address the issue with the Sidekiq backlog impacting CI/CD pipelines. We're working on a deployment fix to address this, though we are seeing some improvement overall with queue durations.
Investigating
Our team is still investigating this issue. Users might encounter some delays with processing CI/CD pipelines.
Investigating
We are seeing delays in CI/CD pipeline processing. The cause for this problem is under investigation. Please follow https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21484 for more details.