Degraded LLMaJ execution in EU

Resolved

Started

Lasted 3h 20m

View official incident ↗

Updates

  1. Resolved

    Steady throughput for the last 30 minutes. Delay < 5min.

  2. Monitoring

    our autoscaling responded slower than expected, but responded nonetheless. the backlog is draining

  3. Identified

    evaluation throughput was overwhelmed by a slow tenant/model workload, with lock-expiry retries multiplying work; the concurrent worker rollout prolonged the incident.

  4. Investigating

    Eval execution in EU region is taking longer than usual