Major Service Outage - Website, API and Pages

Postmortem

Critical impact

Started

Lasted 38m

View official incident ↗

Updates

  1. Postmortem

    On `2024-08-14` We experienced an outage affecting the Bitly API from `15:03Z` to `15:10Z` and affecting the Bitly website from `15:03Z` to `15:31Z`. The root cause was a failure in our deployment pipeline which caused the incorrect release to be deployed. This change was immediately detected and rolled back but the failure triggered an unknown dependency loop which required a manual rollback to restore some systems. We have already taken steps to resolve some root causes and are evaluating ways to improve our ability to rollback more quickly in similar situations.

  2. Resolved

    This incident has been resolved.

  3. Monitoring

    We have implemented a fix for the issue that was affecting our Website, API, and Pages and we are seeing full recovery. We will continue monitoring and report back with any updates.

  4. Investigating

    Hi community. We’re currently investigating issues impacting our website and API. We apologize if this is causing trouble for you. We’ll be sure to update you as soon as we know more!