High rate of errors on control plane

Resolved

Started

Lasted 1h 6m

View official incident ↗

Updates

  1. Resolved

    The issue was caused by our firewall provider unexpectedly blocking traffic despite safeguards we had in place to prevent this behavior. We are actively working with the vendor on additional mitigation measures to reduce the likelihood of this occurring again.

  2. Resolved

    We are no longer seeing issues.

  3. Monitoring

    We are actively monitoring the fix.

  4. Identified

    We have identified the cause and are working with an upstream provider on a fix.

  5. Investigating

    We're observing a high rate of errors on the production control plane which is causing errors on braintrust.dev and data plane operations.