AWS Cluster DNS Changes Degraded Performance

Resolved

Major impact

Started

Lasted 6h 58m

View official incident ↗

Updates

  1. Resolved

    DNS updates for ACME are confirmed healthy. Cluster updates are now proceeding as expected.

  2. Monitoring

    We are continuing to monitor for any further issues.

  3. Monitoring

    We've identified a bad piece of metadata causing DNS issues. We've since patched this metadata and are seeing recovery in ACME updates for cluster topology changes. We will continue to monitor recovery.

  4. Investigating

    We are still seeing cases where new cluster update requests are failing to go through due to ACME related DNS issues. We are actively investigating this issue.

  5. Identified

    Mitigation is in progress; recovery for DNS operations and downstream cluster topology changes has begun.

  6. Identified

    DNS changes for AWS clusters are currently degraded. Cluster topology modifications may be delayed or not able to complete successfully. We've identified the issue and are pushing out a patch.

  7. Identified

    DNS changes for AWS clusters are currently degraded. We've identified the issue and are pushing out a patch.