LLM as a Judge execution is delayed

Resolved

Started

Lasted 5h 19m

View official incident ↗

Updates

  1. Resolved

    We have scaled our infra to drain the queue backlog. Evals are executed without delay again.

  2. Monitoring

    We continue to monitor the situation

  3. Identified

    We believe only a single customer is affected and it is caused by LLM's provider rate limits. Since we monitor worst delay across our system this initially looked as a bigger issue.

  4. Investigating

    We are investigating the root cause