Resolved incident

Feed Realtime - SQS high error rate

Stream · Critical · Started

Duration
6h 28m
Severity
Critical
Status
Resolved

Summary

Critical · Resolved

Millions of requests to the handshake endpoint of our feed realtime system broke the API. This issue has been resolved and a full post mortem will follow.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Resolved

    Stream

    Millions of requests to the handshake endpoint of our feed realtime system broke the API. This issue has been resolved and a full post mortem will follow.

  2. Monitoring

    Stream

    We are continuing to monitor for any further issues.

  3. Monitoring

    Stream

    A fix has been implemented and we are monitoring the results.

  4. Investigating

    Stream

    We are currently investigating an issue with AWS SQS, we are receiving 100% error rate from SQS APIs. Our feeds realtime endpoint is currently unable to push notifications to SQS.