Resolved incident
Code services are degraded, git clone is failing
Harness · Minor · Started
- Duration
- 21m
- Severity
- Minor
- Status
- Postmortem
Summary
Minor · Postmortem
# Summary Between June 19 and July 17, 2026, the built-in Git Clone step — and any pipeline step using the drone-git clone plugin — failed on ARM64 Kubernetes build infrastructure with the error exec /usr/local/bin/clone: exec format error. AMD64 \(Intel/AMD\) builds, Windows builds, and the VM containerless binary path were not affected. The root cause was a defect in our internal image publishing process that caused ARM64-tagged drone-git images to actually contain AMD64 binaries. We identified and mitigated the issue the same day it was reported by reverting the drone-git image to the last known-good version. No customer action or configuration change was required. # Root Cause On June 19, 2026, a security remediation restructured how the drone-git image is built. AMD64 builds were updated correctly, but the ARM64 build pipeline didn't build ARM64 files directly — it adapted the AMD64 build file via text substitution and compiled it on ARM64 infrastructure. The June 19 change altered the AMD64 file so that substitution silently no-op'd instead of failing, so the pipeline published an image tagged ARM64 whose Git Clone and Git LFS binaries were still compiled for AMD64. # Impact * Affected: The built-in Git Clone step, and any pipeline step using the drone-git clone plugin, running on ARM64 Kubernetes build infrastructure, across all accounts, between July 6 and July 17, 2026. * Symptom: Builds failed at the Git Clone step with exec /usr/local/bin/clone: exec format error. * Not affected: AMD64 \(Intel/AMD\) Kubernetes and VM builds, Windows builds, the VM containerless execution path, and our hardened image variant. # Mitigation We reverted the drone-git image version used across all affected services to the last known-good release. This fully resolved the ARM64 execution failures; no customer configuration changes were required. # Next Steps To prevent such issues from happening again. * Rebuild the ARM64 image publishing pipeline to build our dedicated ARM64 build files directly, rather than adapting the AMD64 build files. * Enhance automated post-publish validation to every image release: verify binary architecture matches the image tag, and run a functional smoke test before an image is considered releasable. * Expand automated test coverage to include ARM64 Kubernetes build scenarios. * Remove the affected intermediate image versions from circulation once the corrected release is validated.
Event timeline
How this incident unfolded, from the official status page.
Investigating
Harness
We are currently investigating this issue.
Resolved
Harness
This incident has been resolved.
Postmortem
Harness
# Summary Between June 19 and July 17, 2026, the built-in Git Clone step — and any pipeline step using the drone-git clone plugin — failed on ARM64 Kubernetes build infrastructure with the error exec /usr/local/bin/clone: exec format error. AMD64 \(Intel/AMD\) builds, Windows builds, and the VM containerless binary path were not affected. The root cause was a defect in our internal image publishing process that caused ARM64-tagged drone-git images to actually contain AMD64 binaries. We identified and mitigated the issue the same day it was reported by reverting the drone-git image to the last known-good version. No customer action or configuration change was required. # Root Cause On June 19, 2026, a security remediation restructured how the drone-git image is built. AMD64 builds were updated correctly, but the ARM64 build pipeline didn't build ARM64 files directly — it adapted the AMD64 build file via text substitution and compiled it on ARM64 infrastructure. The June 19 change altered the AMD64 file so that substitution silently no-op'd instead of failing, so the pipeline published an image tagged ARM64 whose Git Clone and Git LFS binaries were still compiled for AMD64. # Impact * Affected: The built-in Git Clone step, and any pipeline step using the drone-git clone plugin, running on ARM64 Kubernetes build infrastructure, across all accounts, between July 6 and July 17, 2026. * Symptom: Builds failed at the Git Clone step with exec /usr/local/bin/clone: exec format error. * Not affected: AMD64 \(Intel/AMD\) Kubernetes and VM builds, Windows builds, the VM containerless execution path, and our hardened image variant. # Mitigation We reverted the drone-git image version used across all affected services to the last known-good release. This fully resolved the ARM64 execution failures; no customer configuration changes were required. # Next Steps To prevent such issues from happening again. * Rebuild the ARM64 image publishing pipeline to build our dedicated ARM64 build files directly, rather than adapting the AMD64 build files. * Enhance automated post-publish validation to every image release: verify binary architecture matches the image tag, and run a functional smoke test before an image is considered releasable. * Expand automated test coverage to include ARM64 Kubernetes build scenarios. * Remove the affected intermediate image versions from circulation once the corrected release is validated.