Delay in background job processingDegraded Performance
Resolved
Updates
Resolved
The combination of infrastructure and application improvements made has sustained stability during peak hours yesterday. This incident has been moved to resolved following our monitoring. Details: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Monitoring
The team has continued to make improvements to the underlying infrastructure to prevent further impact. We have not seen any recent occurrences of background job delays. We will keep this incident open through peak load times over the next 20 hours to monitor performance metrics. Details: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Monitoring
We observed an increased delay in background job processing and CI job queue times, both of which have now recovered. We're closely monitoring the situation as we work on additional improvements. Details: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Investigating
We are currently investigating performance degradation and our team is currently actively working on a mitigation. More details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Monitoring
The team has continued to make improvements since yesterday's update to further reduce impact during peak load. We are actively monitoring the situation. Details: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Monitoring
Background job processing has recovered. Additional changes have been implemented which helped to reduce impact during peak load. We will continue to monitor this as we move through peak load times tomorrow. Details: http://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Investigating
We are investigating delay in background job processing. After a monitoring period that followed yesterday fixes, and expanding infra resources, we are investigating further improvements, more details in gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Monitoring
Background job processing has recovered as peak load has passed. Additional changes are under review to address this going forward. This incident will remain in monitoring as we deploy these changes and observe metrics through tomorrow's peak. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Identified
Improvements to help address this issue are currently under review. We are still observing a delay in background job processing. Users may still see delays in CI job status updates. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562
Investigating
We are investigating delay in background job processing. We are working on further improvements, more details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21562