1PasswordDegraded Performance When Sharing Items
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We identified a configuration change to our edge networking that affected our ability to route traffic for Public Statuspages that use a custom domain. This occurred from 02:55 to 03:13 UTC, at which point the issue was rectified. During this period, the ability to view these status pages was unavailable, but the ability to manage them remained functional, ensuring that no updates or notifications were lost.
Resolved
This incident has been resolved.
Resolved
Between 3:46pm EST and 7:29pm EST on 23rd Oct 2025, customers may have noticed an issue causing web view & unsubscribe tags to redirect to incorrect Template previews or broken pages. This may have affected some customers' ability to view or test templates as expected. The Engineering team has identified the cause and implemented the fix. We are seeing links redirect as expected.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
**Incident:** AWS infrastructure had [an outage affecting core services](https://aws.amazon.com/message/101925/), which our application depends on, in the Virginia \(us-east-1\) region. After partial recovery within AWS, we made configuration changes and saw recovery for our traffic. We continued to observe while AWS worked through the incident on their side. The remaining impact on Asana's side is covered in [https://status.asana.com/incidents/lvtt14d4d80g](https://status.asana.com/incidents/lvtt14d4d80g) . **Impact:** We saw global downtime from 2025-10-20 06:52 UTC through 09:27 UTC, 2 hours and 35 minutes. This affected all regions because we store some global state and shared application configuration in the Virginia region. **Moving forward:** We are working on ways to reduce the impact of single region outages, including for this region, to limit the impact and improve options for recovery. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Resolved
**Incident:** AWS infrastructure had [an outage affecting core services](https://aws.amazon.com/message/101925/), which our application depends on, in the Virginia \(us-east-1\) region. The initial impact to our services is covered by [https://status.asana.com/incidents/zfdtyvckl5k2](https://status.asana.com/incidents/zfdtyvckl5k2); this covers the second major impact to our services. **Impact:** Starting at 2025-10-20 14:58 UTC, our application was down for approximately 10% of our users. By 15:08 UTC the application was completely down for all users We saw some improvement in internal error rates around 17:06 UTC resulting in intermittent availability. We did not see full infrastructure recovery until approximately 19:50 UTC and our traffic fully recovered by 20:29 UTC. This affected all regions because we store some global state and shared application configuration in the Virginia region. **Moving forward:** We are working on ways to reduce the impact of single region outages, including for this region, to limit the impact and improve options for recovery. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Resolved
## **Root Cause Analysis: Increased Latency and Message Delivery Failures – US2 Cluster** **Incident Date:** October 19, 2025 **Duration:** 00:44 UTC – 08:17 UTC **Status:** Resolved ### **Summary** On October 19, 2025, customers using Pusher experienced increased latency and message delivery failures. These issues primarily affected the US2 cluster, with intermittent impact also observed in the MT1 and US3 clusters. The incident resulted in delayed or undelivered messages for many applications. Latency stabilized at 08:17 UTC after mitigation actions were completed. ### **Impact** Between 00:44 UTC and 08:17 UTC, multiple customers experienced: * **Delayed or failed message delivery** across affected clusters * **Degraded performance** in connection establishment and publishing The most significant and prolonged impact occurred in the **US2 cluster**, while **MT1** and **US3** clusters saw elevated latency for a shorter period before stabilizing. ### **Root Cause** The primary cause of the incident was IP address saturation within the subnet assigned to the public Pusher clusters. When traffic levels increased, the **US2 cluster** was unable to scale out further because the available IP addresses in its subnet were fully utilized. This IP scaling limitation prevented the creation of additional instances needed to handle the load. Secondary factors included temporary **network saturation** and **capacity limits** at our cloud provider, which amplified the latency in the early stages of the incident. ### **Detection and Response** The issue was first detected through a combination of **customer reports** and **internal monitoring alerts** showing elevated response times and connection errors. The timeline of actions was as follows: * **00:44 UTC** – Monitoring alerted the team to increased latency across MT1, US2, and US3 clusters. * **03:11 UTC** – Engineers identified subnet capacity as a contributing factor; mitigations began. * **04:03 UTC** – All clusters were scaled out to distribute traffic; latency began to improve in MT1 and US3 clusters. * **08:17 UTC** – Manual intervention allowed US2 to scale successfully, restoring normal latency. ### **Resolution** To restore service, the engineering team: * Scaled out **MT1** and **US3** clusters to handle increased traffic loads * Monitored all clusters to confirm sustained stability Once additional capacity was provisioned and high loads normalized, latency levels returned to normal and remained stable. ### **Preventative Actions** To prevent recurrence, Pusher has initiated the following actions: * **Rate Limits:** We will re-evaluate how rate limits are implemented to better mitigate content from neighboring customers on the shared clusters. * **Subnet Expansion:** Re-evaluating and increasing the size of subnets assigned to shared clusters to ensure sufficient IP availability for future scaling events. * **Load Balancer Enhancements:** We will implement load balance sharding in order to better distribute connections. * **Capacity Planning Improvements:** Enhancing internal monitoring and alerting for subnet and IP utilization thresholds. ### **Next Steps and Commitment** We recognize that message latency and delivery reliability are critical to our customers’ applications. Our team is continuing a full review of cluster capacity management and provider configuration to improve resilience under high traffic conditions. We apologize for the disruption this incident caused and appreciate your patience while we worked to resolve it. Ensuring reliability and transparency remains our highest priority.
Resolved
## **Root Cause Analysis: Elevated API Errors and Outage in AP4 Cluster** **Incident Date:** October 20, 2025 **Status:** Resolved ### **Summary** Between **October 20 and October 21, 2025**, customers using the **AP4 cluster** experienced elevated API errors, latency, and message publishing failures. The issue primarily affected the **Channels API**, preventing customers from publishing new messages and leading to degraded real-time functionality for end-users. During system recovery and while implementing mitigations from a previous incident on Oct 18th, a **misconfigured Redis container** in the AP4 cluster failed to start correctly, preventing caching operations needed for API requests. This misconfiguration went undetected by proactive monitoring, delaying full recovery until October 21 at 17:43 UTC. ### **Impact** Throughout the incident period, customers in the **AP4 cluster** experienced: * **High API error rates** when attempting to publish messages through the Channels API * **Failed or delayed message delivery** for connected clients * **Temporary downtime** for end-customer applications relying on real-time messages Other clusters remained operational, though some minor latency was observed in isolated regions due to dependencies on shared services. ### **Root Cause** This incident resulted from **a chain of events** involving both external and internal factors: 1. \*\*Major AWS Outage \(October 20\)\*\*A large-scale **AWS outage in the US-East region** disrupted multiple dependent systems, impacting several Pusher clusters. 2. **Misconfigured Redis Container \(October 21\)** As systems in the AP4 cluster attempted to scale during recovery, one of the backend **Redis cache containers** failed to start due to a **misconfigured environment variable**. This prevented Redis from initializing properly, resulting in API operations failing or timing out. 3. **Monitoring Gap** Existing monitoring did not capture the **Redis startup failure** because the specific failure mode occurred after initialization checks had passed. This delayed internal detection until API error rates increased and customer impact was observed. 4. **Delayed Customer Communication** Initial updates to customers were delayed while the team triaged the issue and verified the failure pattern, prolonging the time before external notification. ### **Detection and Response** The issue was detected through a combination of **monitoring alerts** showing elevated error rates and **customer reports** of publishing failures. **Timeline of Events:** * **October 20** – AWS outage began, affecting multiple Pusher clusters leading to increased delays and errors. **October 20, evening UTC** – Pusher clusters began recovery as AWS services were restored. * **October 21, 15:06 UTC** – Internal monitoring detected elevated API errors in AP4; engineers began investigation. Incident was unrelated to prior AWS outage. * **October 21, 15:09 UTC** – Root cause identified as a failed Redis caching container. * **October 21, 15:27 UTC** – Restoration of Redis connections underway. * **October 21, 15:29 UTC** – Fix implemented; cluster began gradual recovery. * **October 21, 17:39 UTC** – Full stabilization confirmed across AP4 nodes. * **October 21, 17:43 UTC** – Incident marked resolved after sustained recovery. ### **Resolution** To restore full functionality, the engineering team: * Corrected the **Redis container configuration** preventing startup * Restarted and validated cache services across all AP4 nodes * Confirmed API endpoints were fully operational and message publishing resumed * Monitored latency and error metrics to confirm sustained stability ### **Preventative Actions** To reduce recurrence risk and improve detection and response, Pusher is implementing the following: * **Enhanced Redis Monitoring:** Extending monitoring coverage to detect Redis startup and post-init failures. * **Customer Communication Enhancements:** Improving internal escalation and communication processes to ensure faster external updates. ### **Next Steps and Commitment** We recognize the importance of reliable API performance for our customers. Our teams are conducting a full review of caching dependencies and configuration management across all clusters to prevent similar incidents. We sincerely apologize for the disruption caused by this event and appreciate your patience as we worked through a complex multi-day recovery scenario. Pusher remains committed to transparency, reliability, and continuous improvement in service resilience.
Resolved
We have resolved the stability issues with the Udemy Business management feature on our site, and performance is back to normal. We appreciate your patience.
Resolved
This incident has been resolved.
Resolved
We have identified and fixed an issue with authorization in the us2 region of Batch SaaS
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We have continued to see good performance for requests using Slam-1 and are now marking this issue as resolved.
Resolved
Braze Engineers have confirmed outbound message latency and service performance is now operating normally and within normal operational bounds. Engineers will continue to monitor service health across all clusters. This incident has been resolved.
Resolved
At 07:05 UTC on October 22, 2025, Honeycomb Platform On-Call received a page indicating that our [End to End tests](https://www.honeycomb.io/blog/how-honeycomb-uses-honeycomb-part-3-end-to-end-failures) that run every 5 minutes had failed. The responding engineer logged in to find multiple systemic failures, involving querying, SLO evaluation, and noted the lack of ability to log in to the AWS console. 6 minutes later at 07:11 UTC, AWS declared an outage affecting `us-east-1`. This outage would not be declared resolved until 22:53 UTC, nearly sixteen hours later. Even at the time of AWS-indicated resolution, services continued to be impacted by throttling imposed to permit system recovery. During the duration of the incident, we remained able to ingest events; **no events that were received by our ingest hosts were dropped due to system failures**. Querying was affected between approximately 13:30 and 14:06 UTC, 15:00 and 16:00 UTC, 16:10 UTC and 16:45 UTC, 17:12 UTC and 17:21 UTC, and from 18:40 to 19:06 UTC, which also created delays in evaluations of Triggers and SLOs during those periods. AWS described the trigger of the outage as being DNS resolution for internal DynamoDB endpoints. Once this DNS resolution issue was handled, issues affecting dependent systems within AWS became visible. The most notable symptoms for Honeycomb were errors while launching instances and failed Lambda invocations. The Lambda failures increased the difficulty of seeing exactly which systems were impacted since under normal operations Honeycomb relies on Lambda to power the query functionality, and EC2 scheduling issues reduced our capacity to keep up with SLO evaluation; however, we were able to maintain the EC2 capacity we already had in order to ensure ingest continued to function. We also had degraded performance on ancillary systems like Service Maps due to reduced overall capacity. Multiple providers we rely on also experienced significant hardship with the AWS outage, limiting our ability to make on-the-fly changes to the running system. Ordinarily, if particular features are inaccessible or otherwise in a problematic state, we have levers to de-activate the features and ensure the stability of the key parts of Honeycomb. We were able to mitigate some aspects of the failure without this, but several of these were unavailable to us in this case - for example, to allow querying to continue in a limited capacity while Lambda was inaccessible. Because of the duration and severity of the incident, we plan on conducting a full incident review with further discussions where we will re-evaluate our Disaster Recovery story, reliance on external vendors, and regional dependence.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We've re-enabled email login with a CAPTCHA check to prevent scaled abuse. Thanks for your patience.
Resolved
A new version of the Media Library Widget has been published with a fix for this issue. If you're still seeing errors with the thumbnails/previews, please clear your browser cache to ensure you're using the newest version of the widget code.
Resolved
This incident has been resolved.
Resolved
The EU API has been restored.
Resolved
This incident has been resolved.
Resolved
Our teams have identified and fixed the root cause of the issue that caused delay in executing Automation rules. This was caused due to scaling issues following on from the recent AWS outage. This issue is marked as resolved.
Resolved
Scaling is stable again.
Resolved
This incident has been resolved.
Resolved
The issue affecting publishing, webhook delivery, and form Actions between 22:00 and 22:50 UTC has been fully resolved. All systems are now operating normally, and hosted websites remain fully operational. Some users may need to republish their sites to ensure recent changes are reflected. We appreciate your patience and understanding while our team worked to restore full functionality.
Resolved
This incident has been resolved.
Resolved
Many services from the upstream provider AWS were down and impacted us.
Resolved
Monitors are clear and we’re resolving this incident.
Resolved
This incident has been resolved.
Resolved
On October 20 2025, Loops was impacted by AWS’s outage in the US-EAST-1 region. We observed a variety of misbehaviors during that time, some more serious than others. Overall, the impact to Loops fell within the wider window provided by AWS \(06:49 UTC to 22:53 UTC\). During the outage, we experienced temporary but notable impact to, * Transactional email sending. * Webhook delivery. * Incoming request routing to our SMTP service. **Transactional email** Between 7:02 UTC and 9:18 UTC, transactional email sending was impacted. During this window the Loops API may have accepted transactional requests but failed to send them. Impacted messages were not resent later as we do not store the raw payloads. Timeline: * 7:02 UTC: We began experiencing elevated error rates related to transactional email sending, manifesting as increasing rates of HTTP 500 responses from AWS APIs. * 9:18 UTC: AWS API error rates subside. Transactional sending returns to normal. **Webhooks** Between 9:30 UTC and 10:31 UTC webhook delivery was impacted due to the inability to provision sufficient EC2 capacity. Timeline: * 9:30 UTC: Error rates between the Loops API and our backend webhooks service rise. * 10:01 UTC: The issue is identified as insufficient capacity for the webhooks service due to EC2 provisioning scarcity. * 10:31 UTC: Error rates subside as we change our autoscaling configurations to accept a wider range of EC2 instance types and sizes. **SMTP availability** Between approximately 19:33 UTC and 21:35 UTC an incorrect DNS entry was returned in the results for the network load balancer \(NLB\) used by our SMTP service. During this window, one of three IP addresses returned by DNS responses were unable to accept connections. Attempts to connect to the incorrect IP address manifested as timeouts. Timeline: * ~19:33 UTC: An IP address returned NLB DNS requests stops accepting connections. DNS requests that select the impacted IP address fail to connect and time out. * ~21:35 UTC: The DNS for the NLB corrects itself and begins returning correct responses.
Resolved
All services have been restored and are operating normally. Monitoring is complete, and this incident has been resolved. We appreciate your patience throughout this incident.
Resolved
Airtable's systems are back to normal, and this incident is resolved.
Resolved