Resolved incident

AWS Cluster DNS Changes Degraded Performance

MongoDB Cloud · Major · Started

Duration
6h 58m
Severity
Major
Status
Resolved

Summary

Major · Resolved

DNS updates for ACME are confirmed healthy. Cluster updates are now proceeding as expected.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Identified

    MongoDB Cloud

    DNS changes for AWS clusters are currently degraded. We've identified the issue and are pushing out a patch.

  2. Identified

    MongoDB Cloud

    DNS changes for AWS clusters are currently degraded. Cluster topology modifications may be delayed or not able to complete successfully. We've identified the issue and are pushing out a patch.

  3. Identified

    MongoDB Cloud

    Mitigation is in progress; recovery for DNS operations and downstream cluster topology changes has begun.

  4. Investigating

    MongoDB Cloud

    We are still seeing cases where new cluster update requests are failing to go through due to ACME related DNS issues. We are actively investigating this issue.

  5. Monitoring

    MongoDB Cloud

    We've identified a bad piece of metadata causing DNS issues. We've since patched this metadata and are seeing recovery in ACME updates for cluster topology changes. We will continue to monitor recovery.

  6. Monitoring

    MongoDB Cloud

    We are continuing to monitor for any further issues.

  7. Resolved

    MongoDB Cloud

    DNS updates for ACME are confirmed healthy. Cluster updates are now proceeding as expected.