Problem with CSE APIs, CSE UI, Data Indexing, Metrics Alerting, Metrics Collection and Tracing Collection in North America 1 (US1)

Resolved

Major impact

Started

Lasted 2h 38m

View official incident ↗

Updates

  1. Resolved

    We are no longer seeing issues with Metrics Alerting and Metrics Ingestion and normal operation has been restored at this time.

  2. Monitoring

    Data Indexing, Tracing Ingestion, CSE UI, and CSE APIs have been restored. Metrics Ingestion and Metrics Alerting are recovering but not yet fully operational. Collection is not impacted — raw data continues to be ingested — however, there may be delays before it is available to search.

  3. Identified

    We have identified a failure of a cloud provider's availability zone as the source of the issue affecting CSE APIs, CSE UI, Data Indexing, Metrics Alerting, Metrics Collection and Tracing Collection. The symptoms are: latency in processing and displaying log data, latency in processing and displaying metrics data, latency in processing and displaying tracing data, UI slowness, latency or errors communicating with API. We are working on a fix for the problem and switching traffic off of the affected infrastructure.