Disruption with some GitHub services

Resolved

Minor impact

Started

Lasted 4h 10m

View official incident ↗

Updates

  1. Resolved

    On April 14, between 00:58 UTC and 06:08 UTC, GitHub Enterprise Cloud customers experienced 500 errors when attempting to access Copilot Insights pages which was caused by an authentication failure in our metrics pipeline. We fully mitigated the issue and validated the fix in production. Approximately 709 users were impacted. The total impact duration was approximately 5 hours and 10 minutes. <br /><br />Our investigation determined the incident was caused by a change in a tenant credential which caused authentication errors to retrieve the required data needed on our Copilot Insights pages. <br /><br />We understand this disruption impacted customers' ability to access the Copilot Insights page. To prevent similar issues and reduce resolution time in the future, we are investing in improved diagnostics tooling to quickly identify the root cause of failures, enhanced monitoring, and alerting to detect issues at a more granular level. <br /><br />GitHub is a critical infrastructure for your work, your teams, and your businesses. We are focused on these remediations and continued reliability improvements for Copilot Insights and related metrics experiences.

  2. Monitoring

    This incident has been resolved. We will continue to monitor to ensure stability. Thank you for your patience and understanding as we addressed this issue.

  3. Monitoring

    The degradation has been mitigated. We are monitoring to ensure stability.

  4. Investigating

    We identified an issue that impacts the Copilot Dashboard on the Insights tab and are working on mitigation. We will continue to keep you updated on progress.

  5. Investigating

    The team continues to investigate issues accessing with Copilot Dashboard on the Insights tab.  We will continue providing updates on the progress towards mitigation.

  6. Investigating

    The Copilot Dashboard on the Insights tab is not accessible and we are continuing to investigate.

  7. Investigating

    Degradation of Service - Insights Page

  8. Investigating

    We are investigating reports of impacted performance for some GitHub services.