CloudwaysRackspace Webmail Access Issue
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Our DDOS protection mitigated an attack but also resulted in false positives causing end users to see intermittent 404s. We have adjusted the sensitivity of these rules to prevent further false positives.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
JFrog engineers have applied the fix and all devices are verified to be reporting status accurately.
Resolved
We believe this issue to be resolved. Chat latencies and error rates are back within normal range. Team will investigate into root cause for the spike.
Resolved
This incident has now been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Our team investigated and resolved an issue that prevented some of our customers' Docs sites from loading. All sites should now be loading as expected. If you're still experiencing issues, please contact our support team so we can investigate further.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Docs is back to fully operational.
Resolved
We recently addressed issues affecting Box services**.** We would like to take the opportunity to further explain these issues and the steps we have taken to prevent them from happening in the future. On December 16, 2025 between 4:25 PM PST and 7:23 PM PST, some users may have experienced slowness and temporary unavailability while working in Box. The issue occurred when we attempted to roll back a diagnostic change that was suspected of causing a minor degradation. The rollback unintentionally resulted in decreased capacity in our middleware service, leading to a degraded site experience. It also introduced a distributed deadlock condition that prolonged our incident response and mitigation efforts. Ultimately, we were able to resolve the deadlock and return to normal operations by temporarily throttling inbound traffic. We have implemented measures to increase system capacity and minimize the possibility of a similar deadlock occurring in the future. We are also working on adding safeguards to avoid the rollback failure that served as the proximate cause of this incident. **Analysis** Between 11:11 AM PST and 1:38 PM PST, there was a production deployment of code that was aimed at gathering diagnostic information from our middleware service, in our ongoing efforts to improve system performance and resilience. At 3:30 PM PST, we observed a brief, mostly imperceptible degradation and decided to roll back the change as a precaution. Due to failure in the rollback procedure, we observed a significantly reduced capacity in our middleware service that eventually led to a distributed deadlock condition between critical databases. Box has made significant investments in reliability and mitigation measures. In this case, the issue was regional in scope and cross-region recovery capabilities are still in development. We therefore completed recovery by remediating the impacted region directly rather than executing a regional failover. We restored middleware service capacity by 5:21 PM PST, but the deadlock condition persisted. After several lower-impact attempts to dislodge the deadlock were unsuccessful, we took a broader mitigation step and throttled all traffic at 6:54 PM PST. This resulted in a 26-minute full outage, after which systems stabilized and gradually recovered, with the site returning to full functionality at 7:23 PM PST. **Corrective Actions** Box has initiated the following corrective actions: * Complete the planned capacity increase to reduce the chance of similar deadlocks and shorten recovery time if they occur * Strengthen rollback safeguards so changes can be safely and reliably reverted, preventing a repeat of the rollback failure that triggered this incident Thank you again for being a valued Box customer. We are continuously working to improve Box and want to make sure we are delivering the best in class product and user experience. Should you have any additional questions please do not hesitate to contact us. Sincerely, The Box Team
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
On Friday, Dec 19, at 14:50 UTC \(6:50am PST\), customers backed in our US-West-2 data center experienced dramatically increased API latency, resulting in the website failing to load and messages being queued in the backend. This continued until 18:10 UTC \(10:10am PST\). During this time no messages were lost, though there may have been a significant delay for messages to appear in customer inboxes. All queued messages were delivered by 21:00 \(1:00pm PST\). Customers based in Front’s EU-West-1 and US-West-1 datacenters may have experienced some delays during this time, as some systems are interdependent, but this impact was intermittent and uncommon. The root cause of this issue was the failure of a caching system. There are several database systems that support the Front application, which are supported by caching to improve performance. A recent change increased the size of some objects in the cache layer. This is not inherently wrong, and did not have any immediate impact. On Friday the 19th the caching layer in US-West-2 crossed a new threshold of data volume which triggered a large number of evictions, particularly of other data that is necessary for most application activity. Besides putting additional load on the databases, there was simply not enough room in the cache for all the data we needed to store there. This caused a high amount of thrashing that significantly increased latency for all systems.
Resolved
This incident has been resolved.
Resolved
From 8:15 PM to 8:50 PM UTC, some users experienced partial control plane unavailability.
Resolved
Backfilling of missing execution logs was successfully finished. All systems are fully operational and all executions logs should be visible in the system.
Resolved
Between 20:30:20 UTC on December 18th and 01:57:55 UTC on December 19th, some customers in the US region may have experienced false alerts and false loss of signal alert notifications. We have investigated and resolved this service interruption and impacted services have returned to normal operations.
Resolved
A fix has been rolled out and we have fixed the issues with impacted video posts.
Resolved
The issue has been fixed on Retool Cloud and all systems are operating normally. Self-hosted customers can remove the `prompt` parameter as a workaround until the patch is available.
Resolved
**Incident Date:** June 17, 2025 **Issue:** Profile Flyover Latency and Navigation Failure ### **Impact Summary** On June 17, 2025, users experienced an issue where opening a profile flyover with a scheduled or completed interview triggered an error that prevented further navigation within the application. This was introduced by a deployment and resolved within 30 minutes via a rollback. ### **Root Cause** A code change introduced a query that relied on a parameter incorrectly assumed to always be present resulting in a navigation-blocking error. ### **Preventive Actions** 1. **Enhanced Testing Practices** * Additional acceptance test coverage expanded to include profile flyovers with scheduled or completed interview scenarios for similar code changes in the future 2. **Improved Error Monitoring** * We are evaluating enhancements to our error monitoring to more effectively detect spikes in specific error types.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident is now resolved. Thank you for your patience and understanding.
Resolved
An infra routing changed caused elevated backend errors that may have caused resource queries to timeout or fail
Resolved
Our logs and monitoring verify that the issue is now resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
A fix has been applied and the issue regarding "Device Not Ready" (DNR) errors in the eu-iceland1-a region is now resolved. All impacted services and Virtual Machines have returned to normal operations.
Resolved
This incident is now resolved and closed. During the impact window, log visibility and observability for affected workflows were partially reduced. We apologize for any inconvenience this may have caused.
Resolved