CDC ClickPipes stuck in maintenance state

Resolved

Started

Lasted 2h 43m

View official incident ↗

Updates

  1. Resolved

    The CDC ClickPipes upgrade rollout remains on hold while we confirm we can safely proceed without triggering the underlying pod networking failure mode that triggered the incident.

  2. Monitoring

    All services have recovered; we will continue monitoring the affected nodes and consolidating our findings. Please reach out to our support team to request access to the RCA, when available.

  3. Identified

    Most services have fully recovered. Continuing to roll out the fix to remaining 4 services (all in eu-central-1, us-east-2).

  4. Identified

    Services in eu-central-1 have recovered. We're in the process of manually recovering ~20 services in other regions.

  5. Investigating

    As part of our weekly upgrade process, CDC ClickPipes in eu-central-1 became stuck in maintenance state. The team is working on a mitigation.