Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Sep 11, 2026

  1. Box[Medium] Issue with Uploads

    We recently addressed issues affecting file uploads and other write operations within Box. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future. Between 02:03 PM and 02:24 PM PDT on August 24, 2026, some users may have experienced difficulties while working in Box. During this time, users saw elevated error rates and reduced availability for uploads and other write-oriented actions. The issue occurred when a background user-cleanup process generated a sudden surge of database update activity that was amplified by an unbounded retry loop, which overloaded a database shard and exhausted connection capacity. We restored normal service by applying a rate limit to the cleanup traffic and stopping the offending cleanup workload. After traffic subsided, we removed the temporary limits once systems were stable. In addition, we are making changes to limit background cleanup traffic and improve retry behavior to reduce the chance of recurrence. # Analysis Following this incident, we are now improving how similar types of background jobs are managed and retried, and making adjustments to better prioritize customer-facing operations over these tasks. Specifically, during this issue, rapid retry cycles amplified service friction, highlighting areas where additional safeguards can further protect against similar issues occurring again in the future. We are also reviewing rollout and feature-flag practices to avoid accidental large-scale job generation. # Corrective Actions Box has initiated the following corrective actions: * Implementing bounded exponential-backoff retries for background cleanup jobs. * Enforcing throttling and monitoring on background cleanup traffic so it cannot overwhelm write-path resources. * Introducing priority-aware protections so low-priority background work is shed before it affects customer-facing operations. We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. Sincerely, The Box Team

    Resolved

Sep 10, 2026

  1. CircleCIIssues loading pipelines for a small number of projects

    This incident has been resolved. A small number of customers who created pipelines between 18:53 UTC and 19:24 UTC that contained config using deprecated config syntax will have seen their build-type jobs fail to start. Customers whose jobs or pipelines failed may rerun them.

    Resolved

  2. StripeElevated Interac errors

    There was an issue causing elevated error rates and response times with card payments through Interac. The issue is now resolved.

    Resolved

  3. TrueLayerEUR Payouts Delayed Settlement

    Between 20:59 UTC and 21:12 UTC on 10 Sep 2026, our Payouts product experienced a temporary window of degraded performance. During this timeframe, clients may have observed delays in EUR payout settlements. Impact start time: 10/09/2026, 20:59 UTC Impact end time: 10/09/2026, 21:12 UTC Client actions: None

    Resolved

  4. HashiCorpHashicorp Linux packages GPG key rotation

    Our Engineering team has resolved the issue. These systems should now be operating normally. If you continue to experience any problems, please open a ticket with our support team.

    Resolved

  5. TelnyxAPI Gateway errors in dc2-gce

    The issue affecting API Gateway requests in our dc2-gce location has been resolved, and service has returned to normal.

    Resolved

  6. LiveKitInvestigating reports of elevated egress & connector API errors

    On August 20, 2026, customers in our US East region experienced approximately 10 minutes of 503 errors on the LiveKit Ingress API and failures launching media track egress, between 20:52 and 21:01 UTC. All other regions were unaffected. ‌ **Root Cause** An internal routing service lost one of its two instances, and a concurrent infrastructure issue with our cloud provider was preventing new compute nodes from provisioning in US East, so the replacement couldn't start. At 20:51 UTC, an automated maintenance process removed that last instance, making the service fully unavailable. It recovered at 21:01 UTC once the evicted instance freed capacity for a replacement. ‌ **Timeline \(UTC\)** * Aug 20, 20:52 - Automated maintenance removes last instance in the pool; customer-facing errors begin * Aug 20, 21:01 - Service restored * Aug 21, 03:00 - Cloud provider resolves the underlying infrastructure issue; region fully healthy ‌ **Mitigations & Follow-ups** * The service is restored with two instances guaranteed to run on separate physical nodes. * We updated the automated maintenance process to avoid evicting the last healthy instance of a service. * We added disruption protection for the affected service and are continuing broader resilience improvements. * Our cloud provider resolved the infrastructure issue that had prevented new nodes from scaling up.

    Resolved

  7. JiraDisrupted JIRA availability in US East

    ### Summary On September 3, 2026, between 14:52 and 17:02 UTC, Atlassian customers using Jira and Jira Service Management in the US-East region experienced intermittent errors and degraded access to core Jira experiences, including Rovo, viewing issues, boards, navigation, search-backed experiences, workflow-related actions. The event was triggered by a progressive rollout of a caching change in Atlassian's authorization service. The change increased cache traffic and caused saturation in the cache layer used for authorization checks, which then resulted in timeouts and service errors for some customers in the region. The incident was detected by our monitoring within approximately 12 minutes. We mitigated by deploying the affected cache service and disabling the feature flags associated with the new cache path. The total time to resolution was 2 hours and 10 minutes. ### IMPACT The overall impact occurred on September 3, 2026 between 14:52 and 17:02 UTC. During this period, some customers in the US-East region using Jira and Jira Service Management experienced intermittent failures or degraded performance when accessing experiences that depend on authorization checks. These included issue view, Rovo, board view, navigation, search-backed experiences, API, and workflow-related actions. ### ROOT CAUSE The issue was caused by a progressive rollout of a new cache path in Atlassian’s authorization service. The change was intended to improve scalability for customers with large authorization datasets by changing how some authorization information was cached. During the rollout, both the existing and new cache paths were active for consistency checks, which increased the number of cache entries and cache operations. In the US-East region, where traffic volume was highest, this additional load saturated parts of the shared cache layer. As cache requests timed out, the authorization service made more fallback requests to upstream systems, further increasing load and causing circuit breakers and timeouts. Dependent Jira experiences that require authorization checks then returned intermittent errors or failed to load for affected users. The initial rollout monitoring did not surface the risk early enough because lower-traffic regions did not show the same level of degradation and some alerts were not scoped or routed in a way that highlighted the regional cache saturation before customer impact grew. ### REMEDIAL ACTIONS PLAN & NEXT STEPS We know that outages impact customer productivity, and we apologize to customers who were affected during this incident. We are prioritizing the following improvement actions to help reduce the likelihood and impact of similar incidents: - **Enhance deployment safeguards:** Improve rollout practices for high-risk cache and authorization-service changes by using smaller, region-aware rollout increments and validating behavior in higher-traffic regions before broader enablement. - **Implement graceful degradation:** Evaluate graceful-degradation options for authorization checks so dependent experiences can continue working, if a cache dependency is unavailable. - **Strengthen regional telemetry and capacity monitoring:** Strengthen cache capacity and health monitoring, including regional detectors for cache CPU, memory pressure, hit rate, eviction rate, timeout rate, and connection behavior. - **Expand production-scale load testing:** Review cache design and capacity planning for authorization data so changes that increase key cardinality or cache operations are load-tested against representative production-scale traffic before rollout. - **Harden client resilience:** Review cache client timeout and reconnect behavior to reduce the risk of connection storms when cache nodes become slow or saturated. We will continue to validate these actions through the incident review process and track them to completion with the responsible engineering teams. Thanks, Atlassian Customer Support

    Resolved

  8. LovableMap and location search issues in some published projects

    Map and location search issues in some published projects have been resolved. We have pushed a fix so that projects using location search are guided to the working setup, and if your project is still affected you can ask the Lovable agent to fix it and it will apply the update for you. All systems are operating normally.

    Resolved

  9. LambdaIssue with Cloud Services API Keys

    Severity Level: High This incident has been resolved. The issue preventing users from viewing and creating Cloud Service API keys has been fixed, and API key functionality has been fully restored. We have continued monitoring since the fix was deployed and confirmed the service is operating normally. No action is required on your part. Existing API keys were not affected by this incident. Thank you for your patience and understanding during this time.

    Resolved

Older incidents