Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Jul 14, 2024

Jul 13, 2024

Jul 12, 2024

Jul 11, 2024

Jul 10, 2024

  1. TrelloSome users may experience delays in receiving email notifications

    Between 12:00am 9th July to 08:00am 10th July, we experienced email deliverability issues for some recipient domains for Confluence, Jira Work Management, Jira Service Management, Jira, Trello, Atlassian Bitbucket, and Jira Product Discovery. The issue has been resolved and future emails will flow normally.

    Resolved

Jul 9, 2024

  1. AsanaPage load failures

    Incident: An off-peak routine update of our service handling new page loads caused us to deploy code that depended on an artifact that wasn't yet available. The net effect was that all traffic attempting to do a fresh page load was broken for 48 minutes from June 28 5:52 pm PT to June 28 6:39 pm PT. Impact: Asana was unavailable to new page loads, but existing sessions continued to work as normal. Moving forward: This incident involved interaction across teams and systems. Our postmortem of this event identified operational and tooling changes to reduce the likelihood of incidents -- and decrease the time to resolution. We have an ongoing [5 Whys](https://wavelength.asana.com/workstyle-ask-5-whys-to-get-to-the-root-of-any-problem/) analysis for this incident.

    Resolved

Jul 8, 2024

  1. TypeformTypeform.com - Some Public Pages Unavailable

    We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

Jul 4, 2024

  1. TrelloSome products are hard down

    Between 03-07-2024 20:08 UTC to 03-07-2024 20:31 UTC, we experienced downtime for Trello. The issue has been resolved and the service is operating normally.

    Resolved

  2. OpsgenieSome products are hard down

    Between 03-07-2024 20:08 UTC to 03-07-2024 20:31 UTC, we experienced downtime for Opsgenie. The issue has been resolved and the service is operating normally.

    Resolved

Jul 3, 2024

  1. AsanaSlow loads and degraded mobile app usability

    **Incident**: On June 28, 2024 between 0334  and 0356 UTC, network maintenance was performed to remap internal endpoints, with unforecasted negative effects on communication between internal services. This caused a large amount of requests to Asana during the maintenance period to either drop or time out. **Impact**: App usability was degraded for US-based users, resulting in load times of up to a minute. Users of the mobile app in our US, European, Australian, and Japanese data centers were also affected to a lesser extent by elevated error rates. **Moving forward**: We've identified process gaps during internal retrospective. Moving forward, we'll perform internal endpoint remappings more incrementally to avoid app disruption, and will apply a more thorough approval process for similar maintenance procedures. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

    Resolved

  2. UdemySite is degraded

    We have resolved the stability issue on our site, and performance is back to normal. We appreciate your patience.

    Resolved

Jun 30, 2024

Jun 28, 2024

Jun 27, 2024

Jun 25, 2024

  1. TypeformTypeform Platform - Users are unable to load forms

    We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

Jun 22, 2024

Jun 21, 2024

  1. imgixElevated rendering errors

    ## What happened? On June 17, 2024, at 00:00 UTC, imgix experienced an extreme spike in requests to our render stack. This unexpected surge caused a failure in our auto-scaling infrastructure, leading to an inability to manage all incoming traffic effectively. A fix was implemented at 00:38 UTC, and the issue was resolved by 01:06 UTC. ## How were customers impacted? Between 00:00 and 01:06 UTC, customers may have experienced failures when requesting new renders. However, previously cached assets served successfully during this time. ## What went wrong during the incident? The incident was triggered by a significant increase in requests, which our automated systems did not properly handle. Although the system started to auto-scale as expected, the unexpected surge caused issues with the health checks used for auto-scaling. The combination of extra traffic and health check failure led to an inability to render new images that required manual intervention to resolve. ## What will imgix do to prevent this in the future? To avoid similar incidents in the future, imgix is taking the following actions: 1. **Health Check Enhancement:** We have investigated and implemented updated health checks to support increased traffic volumes. 2. **Rate Limiting:** Further rate limits will be applied to manage traffic spikes and minimize their impact. 3. **Traffic Routing:** Traffic will be rerouted as necessary to distribute the load and reduce the risk of system overloads. 4. **Automated Alerts Improvement:** We will enhance our automated alert systems to respond more effectively to traffic surges and potential issues, including health check failures. By addressing these areas, we aim to further improve our system's resilience and ensure a smoother customer experience during periods of high demand.

    Resolved

  2. TypeformTypeform Follow ups - Form notifications not being delivered

    We are glad to inform you that the pending notifications have been delivered and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

  3. BitlyLink-In-Bio Intermittent Errors

    From `17:56Z` through `19:01Z` on `2024-06-18` Bitly experienced a major outage affecting Bitly Pages functionality. Preliminary root cause was a change to a CDN configuration. The misconfiguration caused all Bitly Pages to result in a 500 page during the outage. All other Bitly services remained operational during the incident.

    Resolved

Older incidents