37signalsInvestigating slowness on copies, moves and chat messages in Basecamp 3
We've cleared the cause that was affecting our database load and background queues are again back to normal levels. Everything should be fine again.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
We've cleared the cause that was affecting our database load and background queues are again back to normal levels. Everything should be fine again.
Resolved
We’ve deployed a fix and everything’s back to normal now.
Resolved
This incident has been resolved.
Resolved
Highrise was down for 3 minutes due to a database slowdown.
Resolved
The incident has been resolved and SMS delivery from Twilio to Rain network subscribers in South Africa and Orange network subscribers in Niger is operating normally.
Resolved
The incident has been resolved and SMS delivery from Twilio to Rain network subscribers in South Africa and Orange network subscribers in Niger is operating normally.
Resolved
The incident has been resolved and MMS delivery from a subset of Twilio Short Codes to Verizon and T-Mobile network subscribers in the United States is operating normally.
Resolved
The incident has been resolved and MMS delivery from a subset of Twilio Short Codes to Verizon and T-Mobile network subscribers in the United States is operating normally.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This issue has been resolved and our error rates have returned back to normal. During this incident users would have observed unauthorized/5xx failures across multiple operations on the project operations and dashboard functions. We have since rolled out a fix and all functionality is restored.
Resolved
This incident has been resolved
Resolved
All impacted services have now fully recovered.
Resolved
All impacted services have now fully recovered.
Resolved
This incident has been resolved.
Resolved
From 00:00–03:00 UTC October 6th, we saw elevated errors on Swish payments as a result of scheduled maintenance. This is now resolved.
Resolved
This incident has been resolved.
Resolved
The incident has been resolved and SMS delivery from Twilio to Orange network subscribers in Guinea is operating normally.
Resolved
The incident has been resolved and SMS delivery from Twilio to Orange network subscribers in Guinea is operating normally.
Resolved
The incident has been resolved and SMS delivery from a subset of Twilio Long Codes to AT&T network subscribers in the United States is operating normally.
Resolved
The incident has been resolved and SMS delivery from a subset of Twilio Long Codes to AT&T network subscribers in the United States is operating normally.
Resolved
All impacted services have now fully recovered.
Resolved
All impacted services have now fully recovered.
Resolved
The issue affecting Notion AI for some customers has been resolved.
Resolved
All impacted services have now fully recovered.
Resolved
All impacted services have now fully recovered.
Resolved
This incident has been resolved.
Resolved
We're seeing recovery for function and sandbox execution, worker networking has been restored
Resolved
The incident has been resolved and SMS delivery from a subset of Twilio Short Codes to Brisanet network subscribers in Brazil is operating normally.
Resolved
The incident has been resolved and SMS delivery from a subset of Twilio Short Codes to Brisanet network subscribers in Brazil is operating normally.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
incident resolved, merchants can disconnect and reconnect their quickbooks now
Resolved
This incident has been resolved.
Resolved
All impacted services have now fully recovered.
Resolved
The primary database was under high CPU load from 1:11PM-1:29PM PT. The impact of this is that our website and API were partially unavailable \(more precisely, at 80-90% availability\) during that time. ### Scope What was affected: This affected most services including the website, builds, submissions, CI/CD jobs, and publishing updates. What wasn’t affected: Serving updates to end users maintained 100.000% availability. App performance measurements sent to Observe were also unaffected. ### Root cause analysis An internal orchestrator service used to update the state of CI/CD jobs, including build jobs and store submission jobs, was updated to retry upon application-level failures. We are investigating more thoroughly and currently believe this led to a cascade of retries that caused more failures once the database was under more load than it could handle. ### Remediation The commit that changed retry behavior was rolled back and we are clearing the queue of retries. For a more robust solution, the internal orchestrator must gracefully handle backpressure from the application server and database, and to apply more backoff.
Resolved
The us-south-2 power maintenance is now complete. ----------------------------------------------------------------------------- Again, in the event you’re unsure as to whether your service is still impacted by this, please open a support ticket and we’ll get back to you as soon as possible. • https://support.lambdalabs.com/hc/en-us/requests/new To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button.
Resolved