Setting secrets and creating apps is degraded

Resolved

Started

Lasted 2h 17m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Monitoring

    While the secret storage service was in a read-only state, app creation requests queued up, due to the retry logic and insufficient request concurrency limits in our GraphQL API. This prevented our GraphQL API from serving any other requests. We have scaled up the GraphQL API and are continuing to monitor the situation.

  3. Monitoring

    A fix has been implemented and we are monitoring the results.

  4. Identified

    An ongoing data migration in our secret storage service is causing degraded Machines API functionality.