Elevated GraphQL API Latency and Dashboard Error Rates

Updates

1. Resolved

   10 months ago

   This incident has been resolved. We have seen API latency normalize and remain normal since ~19:00 UTC.

2. Monitoring

   10 months ago

   API error rates have normalized, however users may still see elevated latency reaching some GraphQL endpoints. Latency continues to trend in the right direction, we continue to monitor for full recovery .

3. Identified

   10 months ago

   We have deployed an initial fix and are seeing improvements. GraphQL error rates and latency remain elevated over the baseline at this time. We are continuing to keep a close eye on recovery.

4. Identified

   10 months ago

   The issue has been identified and a fix is being implemented.

5. Investigating

   10 months ago

   We are continuing to investigate elevated latency and error rates on our GraphQL API endpoints. Users may see errors on parts of the platform that use these APIs. This includes Flyctl actions such as deploys, as well as the fly.io dashboard.

6. Investigating

   10 months ago

   We are investigating elevated API Latency and Error rates on the platform. Users may see delays or errors creating apps, as well as on some dashboard pages. Machines API actions appear unimpacted at this time