[US] Elevated error rates on APIs affecting non-tracing API routes (a.o. prompts) and UI

Resolved

Started

Lasted 2h 59m

View official incident ↗

Updates

  1. Resolved

    The incident has been resolved by a patch. The underlying issue was an unbounded data query data kept fetching data and forced our containers to be out of memory killed.

  2. Identified

    We've identified the issue and limited the blast radius to tracing specific API calls. The UI and the prompt management features should be fully available again. We're working on a patch for the underlying problem.

  3. Investigating

    We’re seeing elevated error rates impacting US-region APIs (including non-tracing routes like Prompts) and parts of the UI, and are working to restore normal service as quickly as possible.