High rate of errors on control plane

Resolved

Started

Lasted 1h 51m

View official incident ↗

Updates

  1. Resolved

    The root cause appeared to be an issue with an upstream dependency (database connection pooler), which we have replaced. We will be writing a full post-mortem with more details.

  2. Resolved

    We are no longer seeing issues

  3. Monitoring

    DB upgrades have completed, and we have seen error rates go down. Continuing to monitor.

  4. Investigating

    We are upgrade DB resources, which may result in brief downtime

  5. Investigating

    We are repairing the DB which may result in brief downtime.

  6. Investigating

    We're observing a high rate of errors on the production control plane which is causing errors on braintrust.dev and data plane operations.