Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Oct 6, 2025

  1. AsanaScript-based actions on tasks were failing

    One of our infrastructure services that we use to run the customer provided script was failing to start up for some workspaces. This was an unintended consequence of us rolling out improved security controls. We've mitigated this by deploying a previously working version for the service and will continue to investigate why this happened.

    Resolved

  2. Lightning AITemporary Service Interruption - Workloads restarted

    Between 14:00 and 14:30 UTC, some customer workloads experienced a temporary interruption due to an unexpected platform issue. During this time, deployments may have paused or restarted. All systems are now fully operational, and no further impact is expected. We’re continuing to monitor closely and reviewing the cause to prevent similar incidents in the future.

    Resolved

  3. Help ScoutWeb App Performance Issues

    Our team has identified & resolved the issue causing degraded performance & slow load times in the web app this morning. Things should be back to normal again. We're closing this out. Thank you for your patience while we worked on this!

    Resolved

Oct 5, 2025

Oct 4, 2025

Oct 3, 2025

  1. 1PasswordDegraded performance when accessing 1Password

    # Incident Postmortem - Degraded performance when accessing 1Password **Date of Incident:** 2025-09-26 **Time of Incident:** 4:20pm UTC - 5:39pm UTC **Service\(s\) Affected:** SSO, Web Sign In, Sign Up, Web Interface, CLI **Impact Duration:** ~60 minutes ## Summary On September 26, 2025 at 4:20 UTC 1Password’s web interface and APIs experienced degraded performance for all customers in the US region. This was not a result of a security incident and customer data was not affected. ## Impact on Customers During the duration of the incident: * **Web interface, Administration:** Customers experienced delays when accessing the 1Password web interface. * **Single Sign-on \(SSO\), Multi-factor Authentication \(MFA\):** Users with SSO or MFA enabled experienced delays, and in some cases failures to login. * **Command Line Interface \(CLI\):** CLI users faced increased latency and timeouts when attempting to access our web APIs. * **Browser Extension:** Users requiring web interface authentication experienced delays or failures. * **Number of Affected Customers \(approximate\):** ~30% * **Geographic Regions Affected:** [1password.com](http://1password.com) \(US/Global\) ## What Happened? At 4:20PM UTC and 5 PM UTC There were traffic bursts which caused extra load on one of our caches. This cache was under-provisioned to handle that spike of activity, which resulted in it exhausting available CPU. This caused cascading errors/latency which manifested in slow and failed requests. * **Timeline of Events \(UTC\):** * 2025-09-26 4:20pm: Spike in customer traffic began * 2025-09-26 4:29pm: Automated monitoring detects increased errors and latency * 2025-09-26 4:35pm: The team activates our incident protocol and begins investigation * 2025-09-26 4:58pm: The team decides to restart application servers * 2025-09-26 5:00pm: The servers have been restarted, service is still degraded, as a second traffic burst begins * 2025-09-26 5:18pm: Service starts to improve * 2025-09-26 5:25pm: The team detects increased load for the second time * 2025-09-26 5:33pm: The team restarts application servers again * 2025-09-26 5:39pm: Service is back to normal, team continues to investigate * 2025-09-26 7:26pm: Team has found the issue, and proceeds to upgrade cache instance size * 2025-09-26 7:49pm: Cache upgrade completed successfully * 2025-09-26 7:50pm: Team continues to monitor, performance has returned to nominal levels * 2025-09-26 8:24pm: Incident is marked as resolved * **Root Cause Analysis:** A code library installed in July introduced latency issues for cache connections. Authentication operations weren't properly rate-limited, allowing large traffic influxes. During peak traffic periods, the cache infrastructure was operating near maximum CPU capacity. The incident occurred when a burst of authentication traffic pushed the cache CPU utilization to 100%. The increased latency and CPU usage together directly caused the incident. * **Contributing Factors:** * Latency increase due to cache library version upgrade * Inadequate rate limiting allowed traffic bursts to go unchecked * Cache instance size is under-provisioned ## How Was It Resolved? * **Mitigation Steps:** Restarting application servers temporarily mitigated the latency and errors, but the problems returned when traffic spiked again. * **Resolution Steps:** Increasing the instance size for the cache resolved the issue. * **Verification of Resolution:** The incident team tested the upgrade in a staging deployment before executing it in production. They then monitored metrics to confirm the system returned to normal levels. ## What We Are Doing to Prevent Future Incidents * **Improve capacity planning for cache:** We will ensure our internal infrastructure is properly sized to handle current traffic volumes and accommodate future growth. We'll implement regular resource evaluations to maintain adequate capacity as our traffic increases. We will also implement proactive alerting systems that notify our teams when resource utilization approaches critical thresholds. * **Update library to a more performant version:** We will upgrade our caching library to the latest stable version to eliminate the current latency issues. * **Improve rate limiting for operations that triggered the traffic burst:** Enhancing our rate limiting system will significantly improve our ability to handle future traffic bursts. * **Timeline for Implementation:** Observability improvements have already been implemented, and we will complete the remaining work by the end of Q1, 2026. ## Next Steps and Communication No action is required from our customers at this time. ‌ We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding. Sincerely, The 1Password Team

    Resolved

  2. Convexproject creation and dashboard unavailable

    Project management was unavailable for approximately 15 mins after a database cluster was removed from circulation following a series of upgrades we're performing. Projects were otherwise running and serving traffic successfully during this period.

    Resolved

  3. CrusoeVM Creation Failure

    Update: The issue is now resolved. The engineering team has implemented a fix and confirmed that normal provisioning operations have been restored.

    Resolved

Oct 2, 2025

  1. RadarElevated API Errors

    We experienced elevated error rates between 15:38 and 15:43 UTC, but services are back to normal. The issue was related to a Cloudflare incident (https://www.cloudflarestatus.com/incidents/nz9nlcyz4qy4).

    Resolved

  2. JamfJamf Protect Reporting and Forwarding Degradation

    Between Wed, Sept 10, 22:05 UTC and Fri, Sept 19, 22:05 UTC, customers may have experienced unreported service degradation for Jamf Protect's reporting and forwarding capabilities via JamfCloud. The impacts to the degraded service were not fully understood until now. We're improving monitoring to better detect and surface hidden impacts going forward. During this time window, affected customers may have experienced a lower volume of reported events and telemetry. On-device protections including Threat Prevention, Advanced Threat Controls, Device Controls, and Tamper Prevention remained effective during this time. The underlying issue was resolved, and the service has been stable since Fri, Sept 19, 22:05 UTC.

    Resolved

  3. Webflow[RESOLVED] Reports of downtime on some hosted sites

    This incident has been resolved. A small number of hosted sites were briefly affected due to an upstream provider issue, but services have remained stable following remediation. We’ll continue to monitor overall system health, and no further impact is expected. Thank you for your patience while we worked to restore full functionality.

    Resolved

  4. Lightning AIRegional outage

    Login was temporarily not functioning in some regions, active workloads not impacted Started: 10:51 AM EST Resolved: 11:02 AM EST

    Resolved

Oct 1, 2025

Older incidents