Mac jobs failing to start

Event timeline

How this incident unfolded, from the official status page.

1. Identified

   5 months ago

   CircleCI

   We have identified a network outage that is preventing Mac jobs from starting. This include m4pro.medium and m4pro.large resource types. We are working with our infrastructure provider to restore service. We will update this incident as we work to resolve the issue.

2. Identified

   5 months ago

   CircleCI

   We are continuing to work on resolving the networking issue for Mac jobs.

3. Identified

   5 months ago

   CircleCI

   We are still seeing failures with Mac jobs starting due to high network latency and packet loss. We will continue to update as we have more information on a resolution.

4. Identified

   5 months ago

   CircleCI

   We are continuing to investigate failures with Mac jobs starting due to high network latency and packet loss. Our engineers remain engaged with our infrastructure provider to restore service. We will provide another update within the hour, or sooner if circumstances change. We thank you for your patience while our engineers work to resolve this.

5. Identified

   5 months ago

   CircleCI

   Mac jobs are continuing to experience failures. Our infrastructure provider is reporting they are experiencing a DDoS attack causing high network latency and packet loss for our Mac fleet. We are working to bring up additional capacity to restore service. We will continue to update as more information is available.

6. Monitoring

   5 months ago

   CircleCI

   We are seeing Mac jobs starting successfully and improved network connectivity to the Mac fleet. We will continue to monitor the recovery.

7. Monitoring

   5 months ago

   CircleCI

   We have processed the backlog of Mac jobs and job start times are recovering to normal parameters.

8. Resolved

   5 months ago

   CircleCI

   Mac jobs have now fully recovered. Thank you for your patience during this incident.