Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Aug 7, 2026

  1. OVHcloud - Public Cloud[BHS][Compute - Instance] - Compute Instance Incident Notification

    We are pleased to inform you that the incident affecting our Compute - Instance offering has been resolved. <b>Start time :</b> 07/08/2026 14:14 UTC <b>End time :</b> 07/08/2026 15:57 UTC <b>Root Cause :</b> A service disruption occurred due to an unexpected infrastructure malfunction. We apologize for any inconvenience caused and appreciate your understanding.

    Resolved

  2. TypeformTypeform Platform - Unavailable

    We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. If you're still experiencing an issue accessing your Typeform account or viewing a published form, please reach out to support at https://www.typeform.com/help/contact/ Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

  3. HarnessThe Prod3 & Prod1 environment is experiencing intermittent outages. We are currently investigating the issue.

    # Summary During a recent production deployment, a defect in our internal deployment tooling caused two critical services   to run with incorrect, non-production configuration values in our production environment This led to a related set of four distinct symptoms: incorrect configuration behavior, intermittent login/access failures, a filestore access issue affecting one customer environment, and delayed pipeline status updates in the UI. ‌ We have identified and are implementing a permanent fix for the underlying configuration defect, and have already put in place resource and capacity changes that resolve the UI delay symptom. ‌ At no point during this incident were pipeline executions themselves lost, corrupted, or left in a stuck state. Where execution behavior was affected, it was limited to delays in status visibility, not in the underlying processing. # Incident Details ## Incorrect Production Configuration Values Applied Our engineering team confirmed a defect in the Service Manager deployment pipeline that caused certain production services to be deployed using configuration values intended for a different environment, rather than the correct production configuration. ‌ **Root Cause** ‌ The service responsible for fetching configuration overrides during deployment queries an internal API that returns a maximum of 1,000 results per request. The total number of services in the environment recently grew beyond that limit. As a result, any service beyond the first 1,000 returned was not included in the response, and the deployment pipeline silently fell back to default configuration values for those services. This is a confirmed pagination defect in the deployment tooling, not an issue with the configuration values themselves. ‌ **Resolution** ‌ Engineering has confirmed the mechanism and is implementing a permanent fix to remove this limit-related gap in the deployment pipeline. ## Intermittent Login / Access Failures During the Service Manager deployment referenced above, some users experienced intermittent login or access failures. Under normal operation, previously running instances should continue serving traffic without interruption while a new deployment is in progress. In this incident, that fallback behavior did not occur as expected, contributing to access failures during the deployment window. ## Filestore Access Issue A filestore access issue was identified that was specific to the Prod-3 environment and affected a single customer's environment. **Root Cause** This is related to an IAM / storage-bucket permission configuration on Service Manager, potentially triggered by rollback activity.  ## Delayed Pipeline Execution Status Updates in UI Some users observed that the pipeline execution graph in the UI was slow to refresh and did not reflect the latest status promptly. Importantly, this was a visibility delay only: there was no impact to actual pipeline executions, and no executions were stuck or failed as a result of this issue. **Root Cause** The pipeline execution graph relies on a message stream \(the orchestration log\) to receive status updates. During the incident window, consumer processing of this stream fell behind \(high consumer lag\), which delayed how quickly status updates reached the UI. This was caused by the fact that the underlying database was in the middle of a planned scaling operation at the same time, and a traffic spike during that window further exacerbated the delay. Users experienced this as apparent pipeline slowness, even though the underlying executions were running normally. **Resolution** We have increased resource capacity for the affected components to maintain more than 50% spare headroom going forward, reducing sensitivity to similar load spikes. This change has been implemented and is currently being validated as part of longer-term hardening for this part of the platform. # Impact Summary * Service Manager and License Manager ran with incorrect configuration values in the Prod-1 and Prod-3 environments. * Some users experienced intermittent login or access failures during the affected deployment window. * One customer environment in Prod-3 experienced a filestore access issue. * Users across affected environments saw delayed pipeline execution status updates in the UI; underlying pipeline executions continued to run correctly and were not lost, stuck, or corrupted. # Preventive Actions The following corrective and preventive actions have been identified. ‌ | **Corrective / Preventive Action** | | --- | | Correct the pagination limit in the configuration-lookup service so that all services are returned and evaluated, regardless of total count. | | Add safeguards so that a service which cannot retrieve its configuration fails safely \(e.g. alerts and blocks the deployment\) rather than silently falling back to non-production defaults. | | Increase Postgres and messaging-pipeline resource headroom \(target: greater than 50% spare capacity\) to reduce sensitivity to concurrent load and scaling events. | ‌ _We recognize the impact this incident had across multiple areas of the platform and appreciate your patience as we work through a complete resolution._

    Resolved

  4. BigCommerceIntermittent Availability Issues

    Commerce engineers were alerted to an increase in error rates for a subset of stores starting at 17:49 UTC. Action was taken and error rates returned to normal at 18:05 UTC. All systems are fully operational.

    Resolved

  5. ChargebeeSelf-Serve Portal Intermittent Issues

    Some users may have experienced intermittent issues managing subscriptions, invoices, or payment methods in the Self-Serve Portal. Service has been fully restored. We apologize for any inconvenience caused.

    Resolved

  6. CartesiaAgents Telephony failing in the US

    This incident has been resolved. We apologize for the inconvenience and will provide further details once we've concluded our internal investigation.

    Resolved

  7. JiraPerformance and access related issues with Jira

    On August 7th - 2026, between 10:27 UTC to 11:49 UTC, some users in the Central European region may have experienced service access issues. The issue has now been resolved and the service is operating normally for all the customers.

    Resolved

  8. TypeformTypeform Platform - Unavailable

    We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

  9. TypeformTypeform Platform - Unavailable

    We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

  10. BlaxelElevated errors in us-was-1 region

    This incident has been resolved, and service is operating normally again. Thank you for your patience. Some issues with Workload Identity may persist. If you encounter an error, the current workaround is to recreate the sandbox.

    Resolved

  11. LovableChatGPT Connector Authorization Failures

    The issue preventing ChatGPT from connecting to Lovable has been fixed, and connections are working normally again. If you still see an error, please try connecting again.

    Resolved

  12. BufferIssue with approval notification emails

    The issue with approval required notification emails has been fully resolved, and it’s functioning as expected again. If you encounter any further problems, please don’t hesitate to reach out to our team.

    Resolved

  13. LiveKitElevated error rate on LiveKit Inference (google/gemma-4-31b-it)

    Today there was an issue with one of our GPU providers that caused it run at about 50% capacity. Requests above capacity return 503s, which we route to another, fallback provider. This fallback provider was not able to rise to the occasion and also returned 503s. In the very near future we will be onboarding additional providers to avoid these kinds of issues.

    Resolved

  14. JiraDegraded performance of JIRA

    On August 7, 2026, JIRA, JWM, and JSM users may have experienced performance degradation when viewing issues and boards. The issue has now been resolved, and the service is operating normally for all affected customers.

    Resolved

  15. SlackTrouble Using Search Bar For Some Admins

    We've completed the rollback of the recent change that was the trigger of the issue. The Search bar and Admin dashboard is restored for all users. We apologize for the inconvenience caused to you and your business.

    Resolved

  16. Box[Medium] Issues with Box AI

    _Notice and disclaimer: Box is providing this preliminary information subject to further review and analysis. To the best of our knowledge, this is the current state and we may update as more information is confirmed._ On May 31, 2026 between 5:00pm PDT and 6:41pm PDT, some customers may have experienced errors or degraded performance when using Box AI features, including failures to generate AI responses or slower-than-expected behavior. The issue affected only AI-related capabilities and did not impact general file access. The disruption was caused by an outage with a third-party AI provider that led to failed AI requests. Service was restored once the provider issue was resolved and normal AI request processing resumed. We are reviewing our third-party dependencies and operational practices to reduce the risk of similar interruptions in the future. We are conducting a full engineering postmortem and our overview is subject to change with further analysis and findings. In the event our continuing investigation yields further substantive findings, we may publish additional analysis after the conclusion of the full engineering postmortem. We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. Sincerely, The Box Team

    Resolved

Aug 6, 2026

  1. Box[Medium] Issue with Multiple Box Services

    We recently addressed issues affecting the Box API, All Files, and Box AI. We would like to take the opportunity to further explain what happened and the steps we have taken to prevent reoccurrence. Between 12:06 PM PDT and 12:39 PM PDT on June 17, 2026, some users may have experienced difficulties while working in Box. During this time, some customers were unable to access content, browse folders, or complete uploads and other file operations, and error rates for affected features were elevated. **What Happened** The issue occurred after a recent rollout of a new traffic-routing change for the All Files experience reached full traffic volume. This routing change directed file-browsing requests through a newer service path that, under full production load, generated a significantly higher volume of calls to an underlying data-access component than had been observed during earlier, partial rollouts. The increased call volume saturated a capacity-limited thread pool within our file service. Once that threshold was crossed, automated health checks detected the degraded state and began restarting affected service instances — which, combined with retry traffic from dependent services, amplified the load and caused a cascading degradation affecting the API, All Files browsing, and Box AI. **How We Restored Service** We restored service by: * Disabling the recent traffic-routing rollout, returning file-browsing requests to the prior service path * Increasing the capacity of the affected file service to absorb the load * Reverting a separate, unrelated deployment as a precautionary measure Monitoring confirmed that availability and error rates returned to normal by 12:39 PM PDT, approximately 33 minutes after impact began. **Analysis** This incident highlighted opportunities to evaluate how a new traffic-routing change is validated at production scale and how dependent components behave when exposed to that traffic pattern for the first time. Specifically, the new routing path generated a more resource-intensive class of data queries than the partial rollout had revealed, and the affected service lacked sufficient capacity headroom to absorb the full traffic volume. It also showed the value of rapid rollback capability and coordinated monitoring across feature rollouts and service capacity limits. **Corrective Actions** Box has initiated the following corrective actions: * **Increase and harden service capacity and autoscaling:** We have increased the capacity of the affected file service and are improving autoscaling thresholds so the service can respond dynamically to traffic spikes. * **Improve pre-rollout validation:** We are strengthening load testing and validation of traffic-routing changes under realistic, production-like conditions before full deployment. * **Tighten rollout safeguards and monitoring:** We are improving rollout gates, alerting, and automated rollback capabilities to enable faster detection and response when changes cause elevated error rates. * **Decouple health probes from capacity saturation:** We are updating how service health checks interact with capacity limits to avoid crash-loop scenarios that can amplify failures during high-load events. We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope this provides clarity on what occurred and the steps we are taking. We would be happy to answer any questions you may still have regarding this matter. Sincerely, The Box Team

    Resolved

  2. Box[Minor] Issues with Uploads

    We recently addressed issues affecting file uploads in Box. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future. Between 4:47 AM PDT and 5:28 AM PDT on June 24, 2026, some users may have experienced difficulties uploading larger files to Box. During this time, chunked uploads of large files may have failed or timed out, while smaller file uploads continued to work normally. The issue occurred when an internal system that temporarily tracks the progress of large, chunked uploads experienced an unexpected failover and lost the in-progress upload information it was holding. The issue resolved once those connections were reset. In addition, we are hardening how our upload service detects and recovers from this kind of interruption to prevent similar issues in the future. # Analysis Large file uploads in Box are split into multiple parts and rely on a shared internal caching system to track the progress of each upload while it is in flight. During this incident, that caching system underwent an unexpected failover, and the newly promoted instance did not contain all of the in-progress upload state. As a result, affected sessions could not continue and returned errors. Recovery was further delayed because the upload service continued to use a connection that was no longer valid. Standard file uploads do not depend on this system and were not affected. ## Corrective Actions Box has initiated the following corrective actions: * Adding dedicated monitoring and alerting for this caching system so that similar failures are detected and corrected more quickly. * Improving how the upload service detects a lost connection and reconnects automatically, so that recovery is faster and does not require manual intervention. * Strengthening the resilience of large-upload session storage so that a failure of this caching system has reduced impact on in-progress uploads. We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. Sincerely, The Box Team

    Resolved

  3. DigitalOceanFunctions on App Platform

    Our Engineering team has confirmed that the issue affecting Functions on App Platform has been fully resolved. Functions builds, deploys, and requests to Functions endpoints are all working as expected. The fix implemented earlier was successful, and we are no longer seeing any problems affecting Functions. However, if you continue to experience any issues, please don't hesitate to raise a support ticket for further investigation.

    Resolved

Older incidents