Resolved incident

Degraded networking to AWS IPs

Fly.io · Unknown · Started

Duration
6h 5m
Severity
Unknown
Status
Resolved

Summary

Unknown · Resolved

This incident has been resolved, connectivity from AWS to all of our hosts is back to normal.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    Fly.io

    We are investigating reports of elevated networking issues between our network and AWS IPs. We are working to determine the source of the issue.

  2. Investigating

    Fly.io

    Our initial investigations indicate connection errors are occuring the return path from AWS IPs. We are escalating the issue with AWS and continuing to investigate. Customers seeing issues are encouraged to also contact AWS support or their AWS hosted vendors support with details of the connection errors and their fly machine IPs.

  3. Identified

    Fly.io

    We are continuing to see connectivity issues from AWS IPs to a subset of our hosts. From our investigation these connections appear to be being blocked from the AWS side. We are continuing to work with our upstream networking partners and AWS on remediation. In the meantime impacted customers are recommended to allocate app-scoped static egress IPs using fly ips allocate-egress.

  4. Identified

    Fly.io

    We are continuing to work with our upstream networking partners and AWS on remediation. Impacted customers are recommended to allocate app-scoped static egress IPs using fly ips allocate-egress. Customers seeing issues are encouraged to also contact AWS support or their AWS hosted vendors support with details of the connection errors and their fly machine IPs.

  5. Identified

    Fly.io

    AWS has implemented changes to unblock connections from their network from reaching our machines. We are seeing connectivity starting to resume on a host-by-host basis as these changes take effect.

  6. Identified

    Fly.io

    Connectivity from AWS to Fly.io on impacted hosts is continuing to recover. Approximately 80% of the hosts that were seeing connectivity issues are fully recovered at this time.

  7. Resolved

    Fly.io

    This incident has been resolved, connectivity from AWS to all of our hosts is back to normal.