Resolved incident

Customers in GCP us-east4 may experience elevated API latencies

Temporal · Minor · Started

Duration
155h 42m
Severity
Minor
Status
Resolved

Summary

Minor · Resolved

The latencies were caused by issues with a downstream service which resolved after we scaled up. We've been monitoring for errors and latencies and have not seen any further issues. This issue is now considered resolved.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    Temporal

    Customers in GCP us-east4 may experience elevated API latencies.

  2. Investigating

    Temporal

    We are continuing to investigate the elevated API latencies in GCP us-east4 and will provide more details as they become available.

  3. Identified

    Temporal

    We are currently working to remediate the elevated API latencies that were observed in GCP us-east4.

  4. Identified

    Temporal

    We are continuing to work on the investigation and remediation of the elevated API latencies in GCP us-east4. We will provide further updates within one hour.

  5. Monitoring

    Temporal

    We have rolled out a change and have paused further investigation overnight pending validation under daytime traffic.

  6. Monitoring

    Temporal

    We are continuing to monitor for any further issues.

  7. Monitoring

    Temporal

    We are continuing to monitor for any further issues.

  8. Resolved

    Temporal

    The latencies were caused by issues with a downstream service which resolved after we scaled up. We've been monitoring for errors and latencies and have not seen any further issues. This issue is now considered resolved.