Degraded performance affecting Duo featuresDegraded Performance

Resolved

Started

Lasted 4h 38m

View official incident ↗

Updates

  1. Resolved

    Latency and service health has stabalized after a period of monitoring. Incident is considered resolved. More details in: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21867

  2. Investigating

    We have increased max concurrency and are monitoring the impact on latency. We'll further investigate segfaults, which may have contributed to this incident. More details in: gitlab.com/gitlab-com/gl-infra/production/-/work_items/21867

  3. Investigating

    The root cause is still under investigation. We are taking steps to address the sustained request saturation in order to mitigate latency issues. More details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21867

  4. Investigating

    We are investigating which model and provider are causing the latency. Apdex for ai-assisted service is not stable yet. More details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21867

  5. Investigating

    We are investigating issues related to our ai-assisted service and ai-gateway. Some users may experience higher latency or errors affecting Duo features. More details here: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21867