Slow booting Linux builds for both open source and private repositories

Resolved

Minor impact

Started

Lasted 11h 29m

View official incident ↗

Updates

  1. Resolved

    Backlog on travis-ci.org has cleared has of 2019-07-03 2:14 UTC. Thanks again for your continued patience during this incident.

  2. Monitoring

    Backlog on travis-ci.com has cleared around 21:55 UTC. We are monitoring travis-ci.org's backlog closely.

  3. Identified

    Sorry for the lack of updates. It's been challenging to remain below the API rate-limit and the backlogs haven't been decreasing at a steady pace as previously mentioned. We now confirm that the backlog on .com is decreasing at a comfortable pace while the backlog on .org is over its peak. Thanks for hanging in there with us.

  4. Identified

    We’ve made progress in being able to stay under the API rate-limit and things are looking better on travis-ci.com. A higher backlog of jobs remains on travis-ci.org. We’ll keep you posted on the status of each backlog on a timely basis.

  5. Investigating

    We are currently hitting the API rate-limit when using our Cloud Provider’s API that is preventing us from creating/deleting VMs. We are looking into getting the system back under the API rate-limit threshold. Thank you for your patience.

  6. Investigating

    We are receiving reports of slow booting Linux builds. We are currently looking into it.