API errors and latencies

Resolved

Started

Lasted 2h 30m

View official incident ↗

Updates

  1. Resolved

    We resolved the underlying issue. A distributed cache roll out for ClickHouse caused the latency and error spikes. The roll out was reverted and we see normal API performance as of now.

  2. Identified

    We identified the issue and work on a fix momentarily.

  3. Investigating

    We scaled our systems but still see elevated errors and latencies. We continue to investigate the underlying issue.

  4. Investigating

    We observed that our ClickHouse based queries have elevated latencies and high error rates. We currently scale the database so resolve the situation.