Unexpected agent disconnection for some customers
Resolved
Minor impact
Updates
Resolved
This incident has been resolved.
Monitoring
Agent connectivity is recovered, but some agents are not reconnecting automatically after the earlier forced disconnects. We recommend restarting agent service to make agents reconnect.
Monitoring
At 06:04 UTC we deployed a defective request routing change to `agent-edge.buildkite.com`, which caused some customers' agents to abruptly disconnect. At 06:09 UTC we noticed the problem, and immediately reverted the change. We began to observe recovery at 06:10 UTC, and are continuing to monitor.