Elevated errors in background job processingDegraded Performance

Resolved

Started

Lasted 2h 46m

View official incident ↗

Updates

  1. Resolved

    A recent code change was rolled back and the error rates have subsided since it was merged. Our team has been monitoring and performance has returned to expected levels. This incident is now resolved. https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22335

  2. Identified

    We've identified a recent code change as the likely cause of elevated Sidekiq job execution errors on the low-urgency-cpu-bound shard, which may delay processing of some background jobs. Mitigations are in progress. We'll continue to update as soon as the mitigation is merged and deployed.