Delayed LLM as a judge execution

Resolved

Started

Lasted 3h 43m

View official incident ↗

Updates

  1. Resolved

    The queue backlog has been fully processed for legacy trace and dataset targeted evals. There is no longer a delay for eval executions.

  2. Identified

    We have identified the root issue and are scaling our infrastructure to work through the evaluation backlog.

  3. Investigating

    We currently see a delay of LLM as a judge execution for trace-level evaluations by 30 minutes. We currently look into the issue to make sure we execute evals in time. Observation level evaluations are not impacted as they are on a new code path and more scalable by an order of magnitude. Please migrate to observation level evaluations.