Resolved incident

Kubernetes Deployments in NYC1

DigitalOcean · Minor · Started

Duration
28h 56m
Severity
Minor
Status
Resolved

Summary

Minor · Resolved

The issue affecting Kubernetes deployments in the NYC1 region has been resolved. Our investigation found intermittent DNS timeouts affecting a small number of DOKS clusters, with affected worker nodes running on shared-CPU Droplets. This is a documented limitation for latency-sensitive cluster DNS workloads such as CoreDNS. The affected clusters are currently functional. To reduce the risk of recurrence, we recommend running CoreDNS on non-shared/dedicated CPU node pools and using sufficient CoreDNS replicas. If you continue to see DNS failures or NodeNotReady events, please open a support ticket so we can investigate that cluster specifically.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    DigitalOcean

    Our Engineering team is investigating an issue affecting Kubernetes deployments in NYC1. Users may see intermittent DNS failures and NodeNotReady events from application workloads during this time. We apologize for the inconvenience, we'll share new information on this page as soon as it is available.

  2. Monitoring

    DigitalOcean

    The issue affecting Kubernetes deployments in NYC1 has subsided. Workloads should now be functioning normally. Our Engineering team is continuing to monitor the affected systems to confirm full resolution. We'll update this page if any further action is needed. We apologize for the inconvenience this may have caused.

  3. Resolved

    DigitalOcean

    The issue affecting Kubernetes deployments in the NYC1 region has been resolved. Our investigation found intermittent DNS timeouts affecting a small number of DOKS clusters, with affected worker nodes running on shared-CPU Droplets. This is a documented limitation for latency-sensitive cluster DNS workloads such as CoreDNS. The affected clusters are currently functional. To reduce the risk of recurrence, we recommend running CoreDNS on non-shared/dedicated CPU node pools and using sufficient CoreDNS replicas. If you continue to see DNS failures or NodeNotReady events, please open a support ticket so we can investigate that cluster specifically.