Issues with webhook deliveriesDegraded Performance

Updates

1. Resolved

   3 months ago

   Webhook delivery failures have subsided in logs, and customer reports indicate no further errors. This incident is considered resolved.

2. Monitoring

   3 months ago

   A fix has been deployed and checks have confirmed that unexpected IP addresses are no longer being used.

3. Identified

   3 months ago

   Deployment of the fix is progressing steadily. Another update will be provided once deployed. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22637.

4. Identified

   3 months ago

   We've traced the issue to a recent network change and are preparing a fix. Another update will be provided by 23:00 UTC.

5. Investigating

   3 months ago

   Our investigation is ongoing. We will provide another update by 21:00 UTC.

6. Investigating

   3 months ago

   We are investigating reports of webhook deliveries from GitLab.com originating from IP addresses outside our documented ranges. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22637.