Newly created instances in AWS us-east-1 are stuck provisioning or have wrong configuration

Resolved

Started

Lasted 2h 36m

View official incident ↗

Updates

  1. Resolved

    All systems operational. We apologize for the disruption this issue caused. We are committed to the reliability of all ClickHouse Cloud services and appreciate your understanding as we continue to improve our systems and operations. If you have questions or need information on a specific ClickHouse service, please contact support "contact support": https://clickhouse.cloud/support.

  2. Monitoring

    Configuration for all affected instances was corrected. We are monitoring the health status of these instances.

  3. Identified

    We have fixed the root cause of the issue. Newly created instances have correct configuration now and are expected to provision successfully. Instances that currently are stuck or have wrong configuration are being corrected by the ClickHouse team. We are closely monitoring creation of new instances and remediation for existing stuck instances.

  4. Identified

    We have identified the possible cause of the issue and applying remediation for stuck/misconfigured instances. We do not recommend creating new instances in us-east-1 until the incident is resolved.

  5. Investigating

    We are investigating the issue where instances created in AWS us-east-1 have wrong configuration, causing provisioning to stuck or to have wrong number of replicas. This affects only a small number of instances created in last ~25 hours