Buildkite service disruption

Event timeline

How this incident unfolded, from the official status page.

1. Investigating

   2 months ago

   BuildKite

   We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

2. Investigating

   2 months ago

   BuildKite

   We’re investigating elevated latency in background worker processing between approximately 12:39am and 1:02am UTC. Processing has since caught up and was near real time as of 1:13am UTC. We’re continuing to monitor the service and are checking remaining retries. Some builds or hosted-agent-related operations may have experienced delays during this period. We’ll provide another update as soon as we have confirmed the service is stable and have more information about the cause. We apologise for the disruption.

3. Monitoring

   2 months ago

   BuildKite

   We have confirmed that all background workers have caught up as of 1:13am UTC. Hosted Agents dispatch has returned to normal. We'll continue to to monitor the service and are checking remaining retries.

4. Resolved

   2 months ago

   BuildKite

   This incident has been resolved.