Increased network latency in North America

Resolved

Started

Lasted 9h 56m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Monitoring

    Our upstream paths have been fixed. We are monitoring the results.

  3. Identified

    We are working with our upstream network provider to address periodic loss of connectivity over transit in ord

  4. Identified

    We are seeing a recurrence of elevated latency across some hosts in ORD impacting the MPG control plane and a subset of clusters there. We are working to address this.

  5. Monitoring

    The network between our regions has been performing well for the majority of traffic. We're still continuing to monitor a few impacted routes in North America that may be seeing elevated latency and packet loss.

  6. Monitoring

    Impacted NA backbones have been sidestepped where possible, and we're continuing to monitor network health.

  7. Monitoring

    Managed Postgres in ORD has returned to normal operation. We continue to see slightly elevated latencies and loss over transits in North America. We are working with our upstream network providers to improve performance.

  8. Investigating

    We are investigating elevated network instability across some hosts in ORD. Apps and managed postgres clusters on impacted hosts may see elevated latency or networking errors at this time.