[US] Fast Mode Preview data delayed

Resolved

Started

Lasted 72h 36m

View official incident ↗

Updates

  1. Resolved

    We've completed the catch-up for experiment spans and resolved the problem.

  2. Monitoring

    We added a patched and are making progress again in the experiment dual writes into the new format. While this is ongoing, we recommend to disable "Fast" mode to see your experiment runs within the trace view.

  3. Identified

    Processing delay for fast mode is down to our usual levels again. When in fast mode, a subset of experiment tracing data is currently unavailable via the UI. We continue to investigate the experiments dual write issues.

  4. Investigating

    Processing delay is down to ~50min and declining. We expect full catch-up for non-experiments within the next 20 minutes.

  5. Investigating

    In addition, we're observing extended delays for experiment spans being ingested with SDK versions published before November 2025. We're working on progress on those.

  6. Investigating

    Data written via non-latest SDK versions may arrive with an delay of ~90min in the fast mode right now. We have identified the issue and are currently catching up.