Increased timeouts due to surge in anonymous users

Resolved

Minor impact

Started

Lasted 3h 11m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.

  2. Monitoring

    We are resolving this incident as error rates have remained low and we have prepared additional protections.

  3. Monitoring

    The degradation has been mitigated. We are monitoring to ensure stability.

  4. Investigating

    Error rates have declined again, and we’re preparing targeted traffic controls while monitoring for a recurrence. Logged-in users and authenticated requests have not been affected.

  5. Investigating

    We are continuing to observe high rates of 504 errors for anonymous requests. We are continuing to work on mitigation. Logged-in users and authenticated requests are unaffected.

  6. Investigating

    We have observed a repeated increase in 504 errors for anonymous and unauthenticated traffic, including users who are not logged in and release downloads. We are continuing to investigate and apply mitigations.

  7. Monitoring

    The problem with anonymous traffic receiving 504 status codes has been mitigated. We are continuing to observe for an abundance of caution.

  8. Monitoring

    The degradation has been mitigated. We are monitoring to ensure stability.

  9. Investigating

    We are debugging a spike in anonymous traffic. Some users who are not logged in may see errors on the site. CI jobs and release downloads may fail with a 504 Gateway Timeout status if they make anonymous calls.

  10. Investigating

    We are investigating reports of impacted performance for some GitHub services.