Resolved incident
Elevated inference errors — US East
LiveKit · None · Started
- Duration
- 1h 38m
- Severity
- None
- Status
- Resolved
Summary
None · Resolved
Between 08:47 and 08:54 UTC, the service that routes LLM, STT and TTS requests through LiveKit Inference in our US East region was unavailable, causing inference requests from agents in that region to fail or time out. Sessions depending on those responses may have degraded or ended prematurely. Replacement capacity came online at 08:54 UTC and request volumes returned fully to normal levels by 09:02 UTC. We have confirmed no further impact since then. The cause was a loss of capacity during automated infrastructure maintenance. We have identified fixes to prevent a recurrence.
Event timeline
How this incident unfolded, from the official status page.
Investigating
LiveKit
We are investigating a period of elevated error rates and timeouts (08:47:00-08:53:35 UTC) for LiveKit Inference requests in our US East region. Service has since recovered and inference requests are completing normally. We are continuing to investigate the cause and confirm the full scope and duration of impact. A further update will follow.
Resolved
LiveKit
Between 08:47 and 08:54 UTC, the service that routes LLM, STT and TTS requests through LiveKit Inference in our US East region was unavailable, causing inference requests from agents in that region to fail or time out. Sessions depending on those responses may have degraded or ended prematurely. Replacement capacity came online at 08:54 UTC and request volumes returned fully to normal levels by 09:02 UTC. We have confirmed no further impact since then. The cause was a loss of capacity during automated infrastructure maintenance. We have identified fixes to prevent a recurrence.