Delayed email delivery for some email service providersPartial Service Disruption

Resolved

Started

Lasted 11h 37m

View official incident ↗

Updates

  1. Resolved

    We have not seen new failures, so we are marking this as solved. More information is available in gitlab.com/gitlab-com/gl-infra/production/-/work_items/23061.

  2. Monitoring

    There were no new bounced emails since 02:29 UTC. We will continue to monitor the logs and provide updates when new information is available.

  3. Investigating

    Our monitoring has not observed any bounced emails since 02:29 UTC. We are seeing early signs of improvement in email delivery, but some messages may continue to be delayed or fail for certain email service providers. The team continues to monitor delivery performance and work with our email delivery provider to understand and address the issue. We will provide another update as more information becomes available.

  4. Investigating

    We have taken additional measures to reduce elevated email volumes and improve platform stability. The team continues to monitor the impact and will provide another update as more information becomes available. The incident is being tracked in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23061

  5. Investigating

    We are continuing to escalate with our email delivery provider, and other affected email service providers directly. The team continues to monitor the impact and will provide another update as more information becomes available. The incident is being tracked in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23061

  6. Investigating

    We are escalating this issue directly with our email delivery provider. The team continues to monitor the impact and will provide another update as more information becomes available. The incident is being tracked in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23061

  7. Investigating

    We’ve detected email delivery delays affecting a small number of email service providers. The team is actively investigating to identify the full scope and impact. The incident is being tracked in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23061