API requests timing out

Resolved

Started

Lasted < 1m

View official incident ↗

Updates

  1. Resolved

    Between 09:55 and 10:33 UTC on Nov 19, requests to all API endpoints timed out or returned 502 errors in three windows: 09:55-10:03, 10:05-10:14 and 10:22-10:32. One of our internal Redis instances became overloaded and stopped responding, and API requests waited on it. We restarted it at 10:32, and the API has been healthy since. We're shipping fixes so a stall like this fails fast and the load pattern that caused it can't recur.