Updates
Resolved
Our Engineering team has confirmed that the increased error rates in the SFO3 region are resolved. From 02:53 UTC to 04:32 UTC, users may have experienced errors, timeouts, or delays when creating or managing Droplets, Volumes, Load Balancers, Reserved IPs, Cloud Firewalls, Managed Databases, and Managed Kubernetes clusters through the API or Cloud Control Panel. Some users also experienced connectivity issues with Load Balancers and with newly created or restarted Droplets. The impact was caused by a network routing issue that disrupted connectivity to one of our core database systems after a load balancer node became unavailable. Services have been stable since mitigation. Operations that failed during the incident may need to be retried. We apologize for the inconvenience. If you continue to experience issues, please open a support ticket from your DigitalOcean account.
Monitoring
Our Engineering team has mitigated the issue, and services in the SFO3 region are currently operating normally. The team is continuing to monitor the situation to ensure stability given the scope of the impact. We will continue monitoring and share another update once the situation is fully resolved. In the meantime, if you encounter any further errors or timeouts, please retry the affected operations. If the issue persists, please submit a ticket through your Cloud Support page so our team can assist you further: https://cloudsupport.digitalocean.com/s/createticket
Monitoring
Our Engineering team has mitigated the increased error rates impacting multiple services in the SFO3 region and is monitoring the situation. The impact was caused by a network routing issue that disrupted connectivity to one of our core database systems after a load balancer node became unavailable. As of 04:32 UTC, users should no longer experience errors or timeouts when creating or managing resources in SFO3. Operations that failed during the incident may need to be retried. We will post an update as soon as the issue is fully resolved.
Investigating
As of 02:53 UTC, our Engineering team is investigating increased error rates impacting multiple services in the SFO3 region. At this point, users with resources in SFO3 may experience errors, timeouts, or delays when creating or managing Droplets, Volumes, Load Balancers, Reserved IPs, Cloud Firewalls, Managed Databases, and Managed Kubernetes clusters through the API or Cloud Control Panel. Some users may also experience connectivity issues with Load Balancers and with newly created or restarted Droplets. We apologize for the inconvenience and will share an update once we have more information.