Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Oct 30, 2025

  1. ChargebeeAWS Outage Impacting Chargebee Services

    # **Incident Overview:** **On October 20, 2025**, a major outage in the [**AWS US-EAST-1**](https://aws.amazon.com/message/101925/) region impacted multiple global services, including components critical to our infrastructure. Between **06:46 AM UTC and 09:30 AM UTC**, we experienced major failures in our systems primarily due to **DNS resolution** issues and **EC2 instance scaling** impairments within AWS. This resulted in degraded performance and limited availability of our services during that window. # **Root Cause and Timelines:** **06:46 AM – 09:30 AM UTC, 20th October \(Major Impact Window\)** * We began observing increased error rates and DNS lookup failures from **06:48 AM** UTC. Critical AWS services such as DynamoDB, SQS, STS, EC2, and Lambda were fully degraded. * The AWS outage - rooted in a DNS resolution failure of EC2 services - prevented us from provisioning new instances and caused some of our application servers to become unhealthy. Since we couldn’t provision new instances, to ensure uptime, we started shedding load on existing instances by moving low-priority workloads to a separate instance. * By **09:30 AM** UTC, core services were partially stabilized, though intermittent DNS errors persisted. # **Recovery:** **After 09:30 AM UTC, 20 October** * We noticed some application servers went unhealthy, and we restarted some of the servers as **autoscaling/new instance provisioning** was not available. * We continued to observe random **DNS lookup failures** from AWS, averaging about **2k errors** per hour. **Job Rescheduling and Completion:** * To avoid further load on the system during the AWS recovery phase, we paused rescheduling of failed jobs until AWS fully declared that their services were fully operational, esp their auto scaling functionality. * AWS gradually restored autoscaling and EC2 instances by **09:30 PM** UTC on **20th October**. * By **11:00 PM** UTC on 20th October, system stability had largely been restored. Some upstream third-party services were still not completely **recovered**. * All **critical jobs** were rescheduled in a staggered way so as not to overload the system and completed by **02:45 PM** UTC on 21st October. * All non-critical jobs were rescheduled, and the entire activity was completed by **02:15 PM** UTC on **22nd October.** We have treated this downtime as a key learning opportunity and established a dedicated internal team to enhance our tooling and processes for faster recovery.

    Resolved

Oct 29, 2025

  1. PlivoIntermittent Message Failures

    We experienced intermittent SMS message failures between 13:50 UTC and 13:59 UTC on 29th Oct 2025 due to an underlying infrastructure failure. The issue was auto-recovered, and all systems have since stabilized. Message processing has fully recovered, and the platform is operating normally. Incident Start Time: 29th Oct 2025 13:50 UTC Incident End Time: 29th Oct 2025 13:59 UTC

    Resolved

  2. Atlassian StatuspagePartial outage while accessing statuspage

    On October 29th, between 9:50 UTC and 10:04 UTC, some Statuspage customers using custom domains may have experienced downtime. Once we identified the root cause of the issue, we took swift action to resolve it. We have verified that the service is now fully restored and operational. Our team is closely monitoring the situation to ensure continued stability. We will conduct an internal review of this incident and implement necessary improvements to our systems to prevent or reduce the likelihood of similar occurrences in the future.

    Resolved

Oct 28, 2025

  1. Convexelevated latency/errors for a subset of customers

    This incident has been resolved. The specific cause was due to a maintenance operation from a downstream provider which resulted in a database cluster switching to using an unoptimized table format. We'll be working to ensure the underlying cause is resolved.

    Resolved

  2. WarpAgent mode requests failing with 401: "is_eligible_for_reverse_team_trial" does not exist

    Agent mode requests are failing with this error message I'm sorry, I couldn't complete that request. Request failed with error: ErrorStatus(401, "{\"error\":\"Unauthorized: User not in context: ERROR: column \\\"is_eligible_for_reverse_team_trial\\\" does not exist (SQLSTATE 42703)\"}") Update (15:40 UTC): A bad server release has been rolled back, and the incident is resolved

    Resolved

Oct 27, 2025

  1. UdemyCritical component on our site is degraded

    Status: Resolved We have resolved the stability issues affecting the login feature on our site. Performance is back to normal. This issue impacted both Udemy Business and Udemy.com users attempting to log in with passwordless code authentication, resulting in “403 Forbidden” errors. We appreciate your patience while our engineering team worked to restore normal functionality.

    Resolved

  2. 1PasswordIntermittent performance issues with 1Password.com

    # Incident Postmortem - 1Password Cloud Services Degraded * **Date of Incident:** 2025-10-20 * **Time of Incident \(UTC\):** 07:26:00 - 20:55:00 * **Service\(s\) Affected:** [1Password.com](http://1password.com/) website, Sign in, Access to passwords and other items * **Impact Duration:** 13 hours, 29 minutes ## Summary On October 20, 2025 at 07:26:00 UTC, [1Password.com](http://1password.com/) faced intermittent latency, authentication failures, and degraded service availability due to a major outage at [AWS in the us-east-1 region](https://aws.amazon.com/message/101925/). This was not a security incident and no customer data was affected. As a result, the 1Password server-side application experienced degradation or intermittent failures, affecting up to 50% of traffic in the US region. Complete service restoration occurred in conjunction with AWS’s final mitigations around 18:30 UTC. ## Impact on Customers All US customers accessing 1Password cloud services experienced intermittent latency, authentication failures, and degraded availability on [1Password.com](http://1password.com/). * **File Share:** Sharing of passwords via links could intermittently fail * **Login:** Users logging into vaults experienced timeout errors and slow responses * **Web Access:** Users accessing their vault through the web interface experienced timeout errors and slow responses * **API Access:** CLI users and API requests received timeout errors and slow responses ## What Happened? At 07:11:00 UTC, AWS began experiencing DNS resolution failures in the us-east-1 region, initially affecting DynamoDB and rapidly cascading to multiple AWS services.  1Password monitoring detected impact at 07:26:00 UTC when monitoring alerts fired for inability to scale up clusters, and an incident was declared. 1Password immediately deployed mitigations inside our infrastructure to ensure there was adequate compute capacity to serve our US-based users, which included pausing deployments and scaling down any services not critical to key functionality for our users. **Timeline of Events \(UTC\):** * 06:55:05 - 1Password monitoring triggers warning for unavailable Pods in Deployment \(caused by inability to obtain AWS IAM credentials\) * 07:03:06 - 1Password monitoring alerts for 5xx errors on auth start endpoint \(caused by inability to obtain AWS IAM credentials\) - pages authentication team, but alert recovers within minutes * 07:26:00 - 1Password monitoring alerts for inability to scale clusters, engineers begin investigating, Incident declared * 07:26:41- AWS confirms elevated error rates across multiple services * 07:49:06 - 1Password monitoring alerts for 5xx errors on auth start endpoint \(caused by inability to obtain AWS IAM credentials\) * 07:51:09 - AWS identifies DNS as the root cause, begins mitigation * 08:02:13 - 1Password suspends auto-scaling tooling to retain existing capacity * 09:27:33 - AWS reports significant recovery signs * 10:35:37 - AWS declares DNS issue fully mitigated, services recovering * 14:14:00-15:43:00 - AWS announced full recovery across all services; throttles EC2 launches * 16:42:49 - 1Password tooling and users start reporting 503s and inability to login due to volume of traffic * 16:50:00 - 1Password services restarted to reset and flush connections, prioritizing post-recovery traffic. * 20:53:00 - AWS resolves their incident * 20:55:00 - 1Password engineers overscale deployments for stability and overnight observation * Oct 21, 2025 - Incident resolved after confirmation of complete upstream recovery ## How Was It Resolved? * **Mitigation Steps:** 1Password paused deployments and auto-management of cluster capacity to ensure enough capacity was available to serve users through peak access times. As demand outstripped available capacity, 1Password engineering reset the circuit breaker to allow additional connections to the service. * **Resolution Steps:** AWS announced system restoration and a reduction in throttling of EC2 API calls. To ensure sufficient capacity for peak traffic, 1Password engineers updated the required number of pods for core services the following business day and resumed auto-management of cluster capacity tooling. The following day, 1Password engineers resumed verification of the health of the systems, deployments, and auto-scaling of the services. * **Verification of Resolution:** Engineers observed monitoring systems and cluster management tooling logs to ensure system health. ## **Root Cause Analysis** * **Root Cause Analysis:** The failures in AWS's internal network affected multiple AWS product APIs. This disruption directly impacted 1Password’s ability to scale up infrastructure, deploy applications, and retrieve configuration data. * **Contributing Factors \(if any\):** * Third-party incident response services and paging services were affected by the AWS incident, which complicated communications. * Upstream customer IDPs were affected by the AWS outage, and returned errors that resulted in authentication failures. ## What We Are Doing to Prevent Future Incidents * **Improve Incident Response:** Create additional backup protocols for when our incident response tooling is unavailable. * **Improve multi-service outage response:** Create strong break-glass runbooks in the event of a multi-service cloud provider outage. ## Next Steps and Communication No action is required from our customers at this time. We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding. Sincerely, The 1Password Team

    Resolved

  3. SpeechmaticsPartial portal outage

    This incident has been resolved. Sign-ups and access of billing details in the portal should now be fully operational.

    Resolved

  4. TypeformTypeform Platform - Unavailable for all users

    At 09:00 CET, our service experienced an unexpected outage due to an issue with AWS. We understand how much you rely on our service, and we sincerely apologize for any disruption this may have caused. Our team actively worked with AWS to investigate the root cause and ensure that measures were put in place to prevent similar issues in the future. The core issue was fixed by 10:35 AM UTC on October 20th, with most services recovering shortly after. Some residual impacts took a few more hours to resolve completely, and the final remaining issue, where the summary page was not updating with new responses, was fully resolved as of October 23rd, around 12:30 PM UTC. Thank you for your patience during the incident. Your continued support means a lot to us, and we remain committed to providing you with a reliable service. REF: ZD 902803 | INC-119

    Resolved

Oct 26, 2025

  1. AppsFlyerAn issue with processing clicks and impressions

    Our teams reprocessed engagements from approximately 10:25 AM UTC to 11:15 AM UTC, and those engagements are now available for new conversions. Thank you for your patience, and please contact us for any related questions.

    Resolved

Oct 25, 2025

Oct 24, 2025

Older incidents