Fetching traces timing out

Resolved

Started

Lasted 37m

View official incident ↗

Updates

  1. Resolved

    A pathological query was impacting other query services. We've added protections against the pathological case and scaled up our query services. All services are now functioning as normal

  2. Monitoring

    We've identified the query causing problems to our systems and have pushed a mitigation. All systems look recovered but we are monitoring for any other problems.

  3. Investigating

    This is primarily affecting run querying

  4. Investigating

    Currently dealing with some heavy queries causing issues fetching traces. We are scaling out our system which should remediate.

  5. Investigating

    We are currently investigating an issue where some trace loads are timing out