Elevated percentage of API calls failing

Resolved

Minor impact

Started

Lasted 6h 7m

View official incident ↗

Updates

  1. Resolved

    After a period of extended monitoring, we have confirmed the remediation has completely resolved the issue. Requests and error rates have been normal since 06:27UTC

  2. Identified

    We have identified the cause as a set of stale IPs for an upstream service. We are now validating the approach for remediation.

  3. Investigating

    We are still investigating. During the period, the error rate has remained low as initially reported, approximately ~0.015%.

  4. Investigating

    We are investigating an unexpected but small increase in the percentage of API calls failing since 01:30UTC. Approximately 0.015% of API calls are failing with a HTTP 504 (Gateway timeout) error. We will update this incident as we have more information.