RedditComment Processing is Delayed
We're confident things are as normal as can be, back to posting!
Resolved
Outages and recoveries from the services you depend on.
40 incidents
We're confident things are as normal as can be, back to posting!
Resolved
Batch transcription with Summarization, Topic Detection and Auto Chapters failed between July 13, 01:00 AM and July 13, 09:00 AM. The services are back online and operational.
Resolved
This maintenance is complete.
Resolved
This incident has been resolved.
Resolved
Coursera is unavailable for some learners; we apologize for the inconvenience and are working to resolve this issue.
Resolved
At 7/10/2024 9:15 PM PT there is a minor outage affecting a small percentage of non-English speech to text jobs that caused the jobs to fail and/or incurred a delay in the jobs' turn around times. All operations were resumed back to normal by 10:10 PM PT.
Resolved
Cloud Functions invocation performance was degraded from 12:15 PM PDT to 12:30 PM PDT due to an infrastructure failure. This issue is remediated.
Resolved
Between 12:00am 9th July to 08:00am 10th July, we experienced email deliverability issues for some recipient domains for Confluence, Jira Work Management, Jira Service Management, Jira, Trello, Atlassian Bitbucket, and Jira Product Discovery. The issue has been resolved and future emails will flow normally.
Resolved
Issues affecting the Base Command service have been resolved and the service is operating normally.
Resolved
This incident has been resolved.
Resolved
Incident: An off-peak routine update of our service handling new page loads caused us to deploy code that depended on an artifact that wasn't yet available. The net effect was that all traffic attempting to do a fresh page load was broken for 48 minutes from June 28 5:52 pm PT to June 28 6:39 pm PT. Impact: Asana was unavailable to new page loads, but existing sessions continued to work as normal. Moving forward: This incident involved interaction across teams and systems. Our postmortem of this event identified operational and tooling changes to reduce the likelihood of incidents -- and decrease the time to resolution. We have an ongoing [5 Whys](https://wavelength.asana.com/workstyle-ask-5-whys-to-get-to-the-root-of-any-problem/) analysis for this incident.
Resolved
This incident has been resolved - all billing/payment related functionality should be restored.
Resolved
All remaining document updates have now been indexed and are appearing in search as normal.
Resolved
This incident has been resolved.
Resolved
We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We restored the backup overnight which resolved the issue.
Resolved
The issue has been resolved.
Resolved
Between 03-07-2024 20:08 UTC to 03-07-2024 20:31 UTC, we experienced downtime for Trello. The issue has been resolved and the service is operating normally.
Resolved
Between 03-07-2024 20:08 UTC to 03-07-2024 20:31 UTC, we experienced downtime for Opsgenie. The issue has been resolved and the service is operating normally.
Resolved
This incident has been resolved.
Resolved
Issues affecting the Private Registry service have been resolved and the service is operating normally.
Resolved
Issues affecting the NGC service have been resolved and the service is operating normally.
Resolved
**Incident**: On June 28, 2024 between 0334 and 0356 UTC, network maintenance was performed to remap internal endpoints, with unforecasted negative effects on communication between internal services. This caused a large amount of requests to Asana during the maintenance period to either drop or time out. **Impact**: App usability was degraded for US-based users, resulting in load times of up to a minute. Users of the mobile app in our US, European, Australian, and Japanese data centers were also affected to a lesser extent by elevated error rates. **Moving forward**: We've identified process gaps during internal retrospective. Moving forward, we'll perform internal endpoint remappings more incrementally to avoid app disruption, and will apply a more thorough approval process for similar maintenance procedures. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Resolved
We have resolved the stability issue on our site, and performance is back to normal. We appreciate your patience.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Comment processing has now fully caught up. Thank you for your patience!
Resolved
This issue has been resolved.
Resolved
The issues with the Jira powerup have been resolved!
Resolved
This incident has been resolved.
Resolved
We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.
Resolved
This incident has been resolved.
Resolved
## What happened? On June 17, 2024, at 00:00 UTC, imgix experienced an extreme spike in requests to our render stack. This unexpected surge caused a failure in our auto-scaling infrastructure, leading to an inability to manage all incoming traffic effectively. A fix was implemented at 00:38 UTC, and the issue was resolved by 01:06 UTC. ## How were customers impacted? Between 00:00 and 01:06 UTC, customers may have experienced failures when requesting new renders. However, previously cached assets served successfully during this time. ## What went wrong during the incident? The incident was triggered by a significant increase in requests, which our automated systems did not properly handle. Although the system started to auto-scale as expected, the unexpected surge caused issues with the health checks used for auto-scaling. The combination of extra traffic and health check failure led to an inability to render new images that required manual intervention to resolve. ## What will imgix do to prevent this in the future? To avoid similar incidents in the future, imgix is taking the following actions: 1. **Health Check Enhancement:** We have investigated and implemented updated health checks to support increased traffic volumes. 2. **Rate Limiting:** Further rate limits will be applied to manage traffic spikes and minimize their impact. 3. **Traffic Routing:** Traffic will be rerouted as necessary to distribute the load and reduce the risk of system overloads. 4. **Automated Alerts Improvement:** We will enhance our automated alert systems to respond more effectively to traffic surges and potential issues, including health check failures. By addressing these areas, we aim to further improve our system's resilience and ensure a smoother customer experience during periods of high demand.
Resolved
We are glad to inform you that the pending notifications have been delivered and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.
Resolved
The backlog has fully recovered.
Resolved
From `17:56Z` through `19:01Z` on `2024-06-18` Bitly experienced a major outage affecting Bitly Pages functionality. Preliminary root cause was a change to a CDN configuration. The misconfiguration caused all Bitly Pages to result in a 500 page during the outage. All other Bitly services remained operational during the incident.
Resolved