AWS ap-south-1 degraded operations

Resolved

Started

Lasted 1h 33m

View official incident ↗

Updates

  1. Resolved

    The issue is fully resolved. The latest data in ControlPlane UI monitoring dashboards was backfilled and should be available. The issue seems to be related to networking issues on underlying VMs. Team is investigating the root cause to make sure it won't happen again or in other regions.

  2. Monitoring

    User dashboards and autoscaling operations should be recovering now. Old data is already available, last hours are being backfilled.

  3. Monitoring

    We are seeing signs of recovery, and closely monitoring.

  4. Identified

    The team has identified the issue and working on the fix. ClickHouse instances are expected to be reachable and working (except autoscaling functionality).

  5. Investigating

    We are investigating issue in AWS ap-south-1 region where: - Admin operations on instances might be delayed or not progressing - Billing metrics not updated - Some dashboards in Control Plane UI not loading or loading slowly