Unreliable or intermittent network connectivity for Linux and Windows builds

Resolved

Major impact

Started

Lasted 20h 46m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Monitoring

    The NAT gateway changes have been fully reverted. Network performance and reliability in your builds should be back to normal. We’ll be monitoring for a short while before closing this incident.

  3. Identified

    At this point, we've reverted half of the NAT gateway updates and a 50% of the Linux and Windows jobs currently processed have better network performance already. We'll provide another update as soon as this process is complete. Thanks for your patience.

  4. Identified

    We've identified an issue with the latest changes to update the NAT gateways. While this change was meant to improve maintainability, we are rolling it back due to the degraded network performance. We're expecting this provides network speed improvements within the next hour. We will provide additional updates at the top of the hour. Thank you

  5. Investigating

    We are continuing to investigate this issue.

  6. Investigating

    We are receiving reports of Linux and Windows builds where network connectivity is unreliable, intermittently working or slow. We are opening this incident to acknowledge the issue and also keep you posted on our work to solve it. Thank you for your patience.