unbound-io-affecting-k8s-node

Resolved

Started

Lasted 30h 13m

View official incident ↗

Updates

  1. Resolved

    A high IO demand from a tenant caused around 200 projects on the same k8s node to get stuck, preventing new connections for a duration of about two hours. The k8s node itself became not manageable. We have isolated the tenant and we're working on a solution to prevent this from happening in the future.

  2. Investigating

    Investigating: (07:18) Update: 🛠️ **Affected Component**: [Database Connectivity](https://app.incident.io/neondb/catalog/01HDK7B6RMJ2DA7AXN0MSHXQFC/01HDK7CYY3NEWAZPEBFVN4FVDW), [Console and API Requests](https://app.incident.io/neondb/catalog/01HDK7B6RMJ2DA7AXN0MSHXQFC/01HDK7C3W54S3AB18AS4GMDVP3) 🚀 **Impact**: Investigating We are currently experiencing technical issues that may affect our services. Our team is diligently working to resolve these issues and restore normal operations. We understand the inconvenience this may cause and appreciate your patience. Stay tuned for further updates. Monitoring: (08:31) A high IO demand from a tenant caused around 200 projects on the same k8s node to get stuck, preventing new connections for a duration of about two hours. The k8s node itself became not manageable. As a bandaid, we rescheduled the noisy compute, applied cgroups IO limiting and terminated the unresponsive node. We are monitoring the situation and working on a stable solution.