Resolved incident
Major Service Outage - Website, API and Pages
Bitly · Critical · Started
- Duration
- 38m
- Severity
- Critical
- Status
- Postmortem
Summary
Critical · Postmortem
On `2024-08-14` We experienced an outage affecting the Bitly API from `15:03Z` to `15:10Z` and affecting the Bitly website from `15:03Z` to `15:31Z`. The root cause was a failure in our deployment pipeline which caused the incorrect release to be deployed. This change was immediately detected and rolled back but the failure triggered an unknown dependency loop which required a manual rollback to restore some systems. We have already taken steps to resolve some root causes and are evaluating ways to improve our ability to rollback more quickly in similar situations.
Event timeline
How this incident unfolded, from the official status page.
Investigating
Bitly
Hi community. We’re currently investigating issues impacting our website and API. We apologize if this is causing trouble for you. We’ll be sure to update you as soon as we know more!
Monitoring
Bitly
We have implemented a fix for the issue that was affecting our Website, API, and Pages and we are seeing full recovery. We will continue monitoring and report back with any updates.
Resolved
Bitly
This incident has been resolved.
Postmortem
Bitly
On `2024-08-14` We experienced an outage affecting the Bitly API from `15:03Z` to `15:10Z` and affecting the Bitly website from `15:03Z` to `15:31Z`. The root cause was a failure in our deployment pipeline which caused the incorrect release to be deployed. This change was immediately detected and rolled back but the failure triggered an unknown dependency loop which required a manual rollback to restore some systems. We have already taken steps to resolve some root causes and are evaluating ways to improve our ability to rollback more quickly in similar situations.