Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Sep 30, 2024

Sep 28, 2024

Sep 27, 2024

  1. HotjarSome users are experiencing issues logging in.

    We’ve validated the deployed fix has worked, and we’re closing this incident. Thanks so much for bearing with us while we investigated and resolved this! We’re sorry for the trouble as well. If you don’t think the incident is resolved, or you’re still seeing problems on your end please raise a ticket through our Help Center. Thanks!

    Resolved

Sep 26, 2024

  1. BitlyProblems with Link-In-Bio

    On `2024-09-16` We experienced an outage affecting the Bitly Pages product from `19:03Z` to `21:11Z`. The root cause was an added dependency which impacted client-side rendering of Pages. This issue was not immediately detected by automated tests or monitors. From `20:27Z` to `21:11Z`, our oncall team activated a recovery system allowing Pages served on the `bit.ly` domain to recover. Some custom domains continued to see impact. The code change contributing to the outage was identified and rolled back allowing for full recovery of Page rendering. We have already taken steps in our automation to detect similar rendering problems before they can affect our customers. We apologize for the inconvenience this outage may have caused and understand the impact to our customers. We place a high value on the reliability of our services and are evaluating ways to improve our ability to mitigate this type of issue. We also understand the impact and potential confusion caused by the lack of messaging during this outage. We value highly and will work to improve our ability to communicate about incidents in a timely fashion.

    Resolved

Sep 25, 2024

  1. TwitchIdentified Web Issues

    🛠 We have resolved all known issues causing the Twitch website to fail to load for some users.

    Resolved

  2. AblyInaccurate day and month stats aggregates since 8th September

    The remediation is complete; app and account day and month aggregates should now be correct again. Apologies again for the inconvenience. (For nearly all customers at least; for customers on enterprise dedicated clusters it might take a few more hours before the month aggregrate for the current month shows correctly).

    Resolved

Sep 24, 2024

  1. KeeperKeeper API website load time and API errors in US region

    On Tues, Sept 24 at 6:06AM PDT, the DevOps Engineering team started receiving notifications regarding API slowness and HTTP 504 errors coming from the Keeper AWS infrastructure. After an investigation, it was determined that our NGINX instances were throwing the errors due to an overload of network traffic. It was then determined that the network traffic increase was caused by an update to the Keeper Desktop application which was published on Monday, the day before. The DevOps team scaled the AWS infrastructure several times over the period of 60 minutes to address the issue. The issue was fully stabilized and resolved by 7:20AM PDT. After a full analysis of the product update process, the engineering team has decided to change the update mechanism of the Keeper Desktop application to utilize a new distributed content delivery network which does not impact the other Keeper AWS infrastructure. This change will be implemented over the next 2 weeks and in preparation of the next Keeper Desktop application update.

    Resolved

  2. imgixElevated rendering errors

    ## What happened? On September 12, 2024, imgix deployed a change to optimize image delivery for Safari browsers. This update inadvertently impacted a small number of browsers by delivering image formats they could not display. imgix identified the issue on September 14 and rolled back the changes to prevent further impact. ## How were customers impacted? Between September 12 and 14, 2024, customers using certain older browsers may have experienced issues displaying images. These users may have encountered image errors, while modern browsers, including Safari, continued to display images as expected. ## What went wrong during the incident? An update meant to optimize image delivery for Safari browsers was applied too broadly. As a result, some browsers not fully compatible with the AVIF image format served these files, causing display issues and affecting approximately 0.02% of renders. ## What will imgix do to prevent this in the future? To prevent similar issues, imgix will implement the following measures: * **More Precise Targeting**: Future updates will be more carefully targeted to specific browsers, ensuring only those that can handle advanced image formats like AVIF receive them. * **Enhanced Compatibility Checks**: We are improving browser compatibility checks to ensure the correct image formats are delivered based on the browser's capabilities. * **Continued Monitoring**: We will closely monitor image delivery performance to detect any compatibility issues early and ensure all users have a seamless experience. These actions will help us deliver a more reliable and optimized experience for all of our customers.

    Resolved

Sep 23, 2024

Sep 20, 2024

Sep 19, 2024

  1. MapboxNavigation API: Missing live-traffic data

    Navigation services experienced an outage affecting our live traffic data. As a result, users might experience issues with navigation, including inaccuracies in ETA, routing recommendations, and the display of real-time traffic conditions such as congestion and incidents for approximately 1 hour. The issue has been resolved.

    Resolved

  2. CrowdinThis is an example incident

    When your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.

    Resolved

Sep 18, 2024

  1. AsanaErrors accessing Asana

    Incident: An experimental flag was enabled that changed initial page loading behavior. With the flag enabled, users were inadvertently directed to load a code version that was not servable by all groups of servers. Subsequent requests would fail when requesting this incompatible version, including requests to report the page load errors.  ‌ Impact:  From 16:52 to 18:59 UTC, 10% of users were unable to load the Asana application in the desktop app or web browser. Existing sessions, as well as mobile and API traffic, were not impacted. ‌ Moving forward: We reverted the configuration change and identified gaps in our monitoring. We are adding monitoring and alerting for version mismatches during page load in order to more quickly identify similar regressions in the future. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

    Resolved

  2. UdemyLogin feature is degraded

    We have resolved the stability issues with the login feature on our site and performance is back to normal. We appreciate your patience.

    Resolved

  3. CodecovReports failing

    From ~8:30 UTC - ~12:00 UTC Codecov failed to process reports. These reports will need to be uploaded again.

    Resolved

Older incidents