Jobs using IP ranges failing with infrastructure errors

Resolved

Minor impact

Started

Lasted 2h 26m

View official incident ↗

Updates

  1. Resolved

    We have seen recovery for IP range jobs for gen2 docker jobs. We appreciate your patience while we worked through the incident.

  2. Monitoring

    We are continuing to monitor the rollout for recovery.

  3. Monitoring

    A fix has been fully deployed, we are now monitoring for recovery.

  4. Identified

    The fix for IP range docker jobs are rolling out across the fleet. We will monitor for recovery in infra fails.

  5. Identified

    We are continuing to rollout a fix for IP range docker jobs. Customers may continue to retry jobs that use IP ranges that experience infra fails. We will provide an update within 20 minutes on the fix.

  6. Identified

    Customers running Docker executor jobs with the IP ranges feature enabled are experiencing a higher rate of infrastructure failures. A fix is being rolled out. Customers who see an infrastructure failure on one of these jobs can rerun the failed job. We will provide another update as we have more information to share.