An issue with unidling ClickHouse instances in Azure regions

Resolved

Started

Lasted 1h 43m

View official incident ↗

Updates

  1. Resolved

    All services stuck in waking in all azure regions are started now.

  2. Monitoring

    Azure eastus2 is fully operational again (all services stuck in waking started).

  3. Monitoring

    Azure region westus3 is fully operational again. All services previously stuck in waking up have started.

  4. Monitoring

    A fix is deployed. Services waking up now (from idle or starting from stopped) will start as normal. Services that are currently stuck in waking will come up soon.

  5. Identified

    We have spotted an issue where idled ClickHouse instances in Azure regions can't be woken up. Team is already working on a fix. We expect the fix to be rolled out soon.