Resolved incident
Problems with Link-In-Bio
Bitly · Major · Started
- Duration
- 29m
- Severity
- Major
- Status
- Postmortem
Summary
Major · Postmortem
On `2024-09-16` We experienced an outage affecting the Bitly Pages product from `19:03Z` to `21:11Z`. The root cause was an added dependency which impacted client-side rendering of Pages. This issue was not immediately detected by automated tests or monitors. From `20:27Z` to `21:11Z`, our oncall team activated a recovery system allowing Pages served on the `bit.ly` domain to recover. Some custom domains continued to see impact. The code change contributing to the outage was identified and rolled back allowing for full recovery of Page rendering. We have already taken steps in our automation to detect similar rendering problems before they can affect our customers. We apologize for the inconvenience this outage may have caused and understand the impact to our customers. We place a high value on the reliability of our services and are evaluating ways to improve our ability to mitigate this type of issue. We also understand the impact and potential confusion caused by the lack of messaging during this outage. We value highly and will work to improve our ability to communicate about incidents in a timely fashion.
Event timeline
How this incident unfolded, from the official status page.
Investigating
Bitly
Hi community. We are aware of a service outage specifically impacting our Link In Bio product and are currently investigating a fix.
Monitoring
Bitly
A fix has been identified and we are monitoring the results.
Resolved
Bitly
The validation of the fix has completed. Bitly Pages are fully operational again.
Postmortem
Bitly
On `2024-09-16` We experienced an outage affecting the Bitly Pages product from `19:03Z` to `21:11Z`. The root cause was an added dependency which impacted client-side rendering of Pages. This issue was not immediately detected by automated tests or monitors. From `20:27Z` to `21:11Z`, our oncall team activated a recovery system allowing Pages served on the `bit.ly` domain to recover. Some custom domains continued to see impact. The code change contributing to the outage was identified and rolled back allowing for full recovery of Page rendering. We have already taken steps in our automation to detect similar rendering problems before they can affect our customers. We apologize for the inconvenience this outage may have caused and understand the impact to our customers. We place a high value on the reliability of our services and are evaluating ways to improve our ability to mitigate this type of issue. We also understand the impact and potential confusion caused by the lack of messaging during this outage. We value highly and will work to improve our ability to communicate about incidents in a timely fashion.