High number of 5XX on the Machines API and dashboard

Resolved

Started

Lasted 9h 59m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Monitoring

    We are still working on fixing degraded Managed Postgres clusters.

  3. Monitoring

    Some Managed Postgres v1 clusters are degraded. We are working on fixing them. Managed Postgres v2 is unaffected.

  4. Monitoring

    A fix has been implemented and we are monitoring the results.

  5. Identified

    We've identified an internal service providing authentication to our Machines API has failed, our team is currently looking at our options for restoring this service. Existing Machines/Apps will continue to run as normal. Thank you for your patience.

  6. Identified

    We've identified an internal service providing authentication to our Machines API has failed, our team is currently looking at our options for restoring this service. Existing Machines/Apps will continue to run as normal. Thank you for your patience.

  7. Investigating

    We are continuing to investigate this issue.

  8. Investigating

    Existing machines are unaffected. We are investigating the issue.