Intermittent volume request timeouts

Resolved

Started

Lasted 487h 18m

View official incident ↗

Updates

  1. Resolved

    The infrastructure improvements have resolved the issue.

  2. Monitoring

    We've improved the underlying storage infrastructure, and are monitoring to ensure that the performance has returned to normal.

  3. Investigating

    We continue to investigate intermittent volume request failures affecting persistent volumes. Our investigation has identified instability in the underlying NFS storage layer as the likely source of the issue. We are seeing elevated timeout, retransmission, and connection reset rates between our infrastructure and the storage service backing persistent volumes. We have engaged GCP support and are actively testing mitigations, including reducing connection pressure on the storage system and introducing additional caching to decrease request volume. We will provide another update as we learn more or have a mitigation deployed.

  4. Investigating

    We’re investigating intermittent timeouts affecting volume requests.