issues with project creation and long-idle compute starts in us-east-2
Resolved
Updates
Resolved
The issue has been fully resolved. All affected components are operating normally again.
Investigating
Investigating: (14:23) We are aware of an issue affecting new project / branch creation and waking up idle endpoints that have been inactive for an extended period. We're investigating the scope, root cause, and potential mitigations. Identified: (14:32) We have identified the underlying cause of this incident and are actively working to implement a resolution. Our team is committed to resolving this issue as quickly as possible to minimize any further impact. Identified: (15:09) We have mitigated the issue for the vast majority of users. We're continuing to investigate a small rate of branch creation failures. Monitoring: (16:44) After the initial mitigation, there were ~20 projects stuck scheduling onto storage nodes. We have now repaired their state, and all impact for all users should now be resolved.