Elevated Errors Affecting AI Gateway Health ChecksPartial Service Disruption

Updates

1. Resolved

   4 months ago

   We are seeing error rates return to normal. This issue has been resolved. Details at https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

2. Monitoring

   4 months ago

   We are monitoring the situation to ensure that services have returned to normal. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

3. Monitoring

   4 months ago

   Deployment for the revert has reached production. Manual verification is successful with error rates trending down. We are continuing to monitor. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

4. Identified

   4 months ago

   The revert remains in progress. Additional updates will be shared in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

5. Identified

   4 months ago

   We're continuing to work on reverting the change causing this issue. Details will be shared in: gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

6. Identified

   4 months ago

   We have identified the problematic change causing this behavior and are working to revert it. Details will be shared in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

7. Investigating

   4 months ago

   We're aware of elevated errors affecting AI Gateway health checks. We've identified a likely cause and are actively working on a fix. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519