Unexpected agent disconnection for some customers

Event timeline

How this incident unfolded, from the official status page.

1. Monitoring

   about 2 months ago

   BuildKite

   At 06:04 UTC we deployed a defective request routing change to `agent-edge.buildkite.com`, which caused some customers' agents to abruptly disconnect. At 06:09 UTC we noticed the problem, and immediately reverted the change. We began to observe recovery at 06:10 UTC, and are continuing to monitor.

2. Monitoring

   about 2 months ago

   BuildKite

   Agent connectivity is recovered, but some agents are not reconnecting automatically after the earlier forced disconnects. We recommend restarting agent service to make agents reconnect.

3. Resolved

   about 2 months ago

   BuildKite

   This incident has been resolved.