Resolved incident

Investigating delayed session data ingestion on Cloud Dashboard

LiveKit · Minor · Started

Duration
4h 57m
Severity
Minor
Status
Resolved

Summary

Minor · Resolved

This incident has been resolved. Between 19:20 and 00:10 UTC, session data and observability ingestion was delayed by 45-60 min. The root cause has been addressed and data ingestion returned to normal by 00:10 UTC. No data was lost during this window.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Resolved

    LiveKit

    This incident has been resolved. Between 19:20 and 00:10 UTC, session data and observability ingestion was delayed by 45-60 min. The root cause has been addressed and data ingestion returned to normal by 00:10 UTC. No data was lost during this window.

  2. Monitoring

    LiveKit

    Mitigations have been deployed and we are now catching up to realtime.

  3. Identified

    LiveKit

    The amount of data we are ingesting in the sessions view is outpacing our ability to ingest, thus causing a delay in ingestion. We are pursuing a few mitigations right now including moving to a larger database instance to relieve the processing bottleneck.

  4. Identified

    LiveKit

    We are continuing to work on a fix for this issue.

  5. Identified

    LiveKit

    Sessions data ingest remains delayed for about an hour. It's not impacting other dashboards. We've identified the source of the slow ingest and are working on a mitigation. We will update again in 30 mins.

  6. Investigating

    LiveKit

    We are currently investigating delayed session data ingestion on Cloud Dashboard. No services appear to be impacted and we don't expect any data to be lost.