500 Errors due to failures in Gitaly service nodesOperational

Resolved

Started

Lasted 19h 6m

View official incident ↗

Updates

  1. Resolved

    Restarting specific nodes improved the situation. We'll mark the incident as resolve while we keep monitoring the situation. More information: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

  2. Monitoring

    Restarting specific nodes improved the situation. We'll keep monitoring the situation for now. More information: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

  3. Identified

    We're still exploring the solution for the git problem. See details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

  4. Identified

    We've identified the problem related to git activities. We're looking into solution. See issue: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

  5. Investigating

    We're observing elevated 500 error rate on GitLab.com again and we're investigating the root cause. See detail issue: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

  6. Monitoring

    Gitaly appears to be working normally at this time. We are continuing to monitor the issue and have mitigation steps should the issue reoccur. We do not anticipate further updates unless we start seeing degraded Gitaly performance again. Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21310

  7. Monitoring

    The nodes have been rebooted and Gitaly appears to be working as expected. We are moving this incident to Monitoring. Our team is working to identify the root cause of these errors while we monitor the node performance. Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21310

  8. Investigating

    We have rebooted the VMs that were generating 500 errors. Once the VMs have rebooted we will validate that Gitaly is responding normally and work to identify the root cause of these issues to prevent re-occurrence. Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21310

  9. Investigating

    We are continuing to investigate the 500 errors on GitLab.com. We are investigating the root cause of the incident. We are preparing to apply mitigations to reduce error rates. Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21310

  10. Investigating

    We have seen an unexpected increase in 500 errors on GitLab.com pages, often related to Merge Requests. We are investigating the root cause of these issues. Further details in the incident issue: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

  11. Monitoring

    We have recovered and are still monitoring. https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303

  12. Identified

    Partial service disruption due to this issue.

  13. Identified

    We see a 1.5% degraded state on two of our Gitaly nodes. Some customer may experience delays while contributing to their projects. We have identified the problem and we are working through steps to resolve this. https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21303