Mac jobs failing to start

Resolved

Minor impact

Started

Lasted 6h 29m

View official incident ↗

Updates

  1. Resolved

    Mac jobs have now fully recovered. Thank you for your patience during this incident.

  2. Monitoring

    We have processed the backlog of Mac jobs and job start times are recovering to normal parameters.

  3. Monitoring

    We are seeing Mac jobs starting successfully and improved network connectivity to the Mac fleet. We will continue to monitor the recovery.

  4. Identified

    Mac jobs are continuing to experience failures. Our infrastructure provider is reporting they are experiencing a DDoS attack causing high network latency and packet loss for our Mac fleet. We are working to bring up additional capacity to restore service. We will continue to update as more information is available.

  5. Identified

    We are continuing to investigate failures with Mac jobs starting due to high network latency and packet loss. Our engineers remain engaged with our infrastructure provider to restore service. We will provide another update within the hour, or sooner if circumstances change. We thank you for your patience while our engineers work to resolve this.

  6. Identified

    We are still seeing failures with Mac jobs starting due to high network latency and packet loss. We will continue to update as we have more information on a resolution.

  7. Identified

    We are continuing to work on resolving the networking issue for Mac jobs.

  8. Identified

    We have identified a network outage that is preventing Mac jobs from starting. This include m4pro.medium and m4pro.large resource types. We are working with our infrastructure provider to restore service. We will update this incident as we work to resolve the issue.