GitHub returning 403 for all Git Sync repositories

Resolved

Minor impact

Started

Lasted 1h 58m

View official incident ↗

Updates

  1. Resolved

    The fix has been deployed to all production clusters and we are observing a continued period of stability. At this time, we are marking the incident as resolved.

  2. Monitoring

    The fix has landed in 60% of production. We are currently monitoring as it lands in the remaining clusters.

  3. Identified

    Since ~21:40 UTC on Nov 3, GitHub started rejecting the default User-Agent our Git Sync client sends (nanogit/0), so nearly every GitHub-backed repo shows as unhealthy ("not authorized") and syncs are being skipped. Our Engineering team has rolled out a fix to development and validated that the fix works. We will roll out the fix to production shortly and will provide another update.