Degraded performance for pipelines on gitlab.comDegraded Performance
Resolved
Updates
Resolved
This issue is now resolved. The CI job queue on Hosted Runners and Environments/Review Apps have remained stable during our monitoring period.
Monitoring
We have applied a fix and are now monitoring before marking as resolved. More details can be found in gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Identified
Background processing remains degraded, affecting Environments and Review Apps, a fix is awaiting deployment. Hosted Runners and CI jobs are healthy. Details in gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Monitoring
We've disabled the worker responsible for the issues. We've merged another fix for that specific worker and are in the process of deploying it to production before re-enabling it. The next update will be posted once that is completed.
Investigating
Background processing remains degraded, however the revert MR has been merged a short time ago. We are monitoring performance to see if the MR results in performance returning to optimal levels. See: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Investigating
Background processing remains degraded, affecting Environments & Review Apps. Hosted Runners and CI jobs are healthy. A revert MR is queued for production, with more changes needed to address background processing. See: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Investigating
Hosted Runners continue to process without delay and are able to retrieve cached CI/CD variables. However some background processing is currently impacted and users may observe environments/review apps URLs not updating. We continue to wait for the fix to reach production. Details in gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Investigating
We have observed that jobs on Hosted Runners are processing without delay and are able to retrieve cached CI/CD variables. We are currently awaiting deployment of a fix to ensure the issue remains mitigated. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Investigating
We have observed some improvement in queue times but are still investigating CI job delays on Hosted Runners and CI variable retrieval. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Investigating
No material update available at this time. Investigation into CI job delays on Hosted Runners and CI variable retrieval is ongoing. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Investigating
Issues with Hosted Runner CI job queue delays and cache errors related to CI/CD variables are still under investigation. Self-hosted Runners are not currently affected. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Investigating
In addition to CI job processing delays, users may also see "Failed to retrieve CI/CD variables from cache" when running new pipelines manually via the UI. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Identified
We are continuing to see delays in CI job processing and are actively investigating mitigation steps. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190.
Monitoring
CI workloads are recovering as expected and should return to normal within the next hour. Some pipelines may still be delayed during this time. We are currently monitoring. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Identified
We have identified the cause and taken action to mitigate re-occurance. We are seeing recovery on CI workloads and it should return to normal within the next hour. Some delays are still expected during this time. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Investigating
Mitigation action has been taken and new pipelines can be triggered. Pipelines are being processed, but delays are still expected for up to 2 hours. Jobs may still be stuck in pending state during this time. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Investigating
The resource limits increase did not sufficiently mitigate impact. Investigation into cause and mitigation options is still in progress. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Investigating
Resource limits have been increased and we are monitoring the impact. Investigation into the root cause is still in progress. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Investigating
Investigation is still underway. An MR to increase resource limits to mitigate impact is currently awaiting merge. See issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190
Investigating
We have received reports of delayed or stuck pipelines, and jobs waiting longer than expected to execute. We are investigating. See issue for more details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21190