Run traces and metrics delayed

Resolved

Started

Lasted 41m

View official incident ↗

Updates

  1. Resolved

    The fix has been running stable for 30 minutes and all system metrics are healthy. Run traces and metrics ingestion should be near realtime again.

  2. Monitoring

    The team has fixed the issue with our event streams which power run trace data and run metrics ingestion into our database. The dashboard traces and metrics should be loading as expected again. Function execution was unaffected by this incident, but may have been in an earlier incident: https://status.inngest.com/incidents/01KF5JV5EWHAN03N5JGXRAMK6T

  3. Identified

    We have identified an issue causing trace data and metrics for runs to be delayed in the Inngest dashboard. Function execution is running alright. The issue is related to an issue with an internal event stream that ingests run data.