Slow trace data retirival across API and UI in EU

Resolved

Started

Lasted 2h 25m

View official incident ↗

Updates

  1. Resolved

    should be finally resolved

  2. Monitoring

    Looks like some timeout are still occurring. Monitoring.

  3. Resolved

    All replicas look healthy again

  4. Identified

    Slow clickhouse read replicas affecting performance across the service. Draining and replacing them.

  5. Investigating

    Slow traces data serving across API and UI