500 errors across several REST and GraphQL API endpointsPartial Service Disruption

Resolved

Started

Lasted 12h 22m

View official incident ↗

Updates

  1. Resolved

    Our metrics indicate this is now fixed. We will now be marking this as resolved. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21573

  2. Monitoring

    The fix has reached production and users should not experience 500 errors on API calls. We will be monitoring the fix before marking this incident as Resolved. More details https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21573

  3. Identified

    The fix has now reached Canary environment, users can enable it by clicking "Next" at https://next.gitlab.com/ until the fix reaches production. More details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21573

  4. Identified

    The revert change has been merged, users will still experience 500 errors until the fix reaches production. We estimate it will be fully deployed at the end of EMEA day. More details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21573

  5. Identified

    We have identified a code change that provokes 500 errors for API requests. We are currently reverting the change. Workaround in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21573