Authentication errors in prod-us-west-0

Resolved

Minor impact

Started

Lasted 2h 49m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Monitoring

    The fix has been implemented, and we are monitoring the results.

  3. Identified

    We have noticed improvements across services. We will now be applying a more permanent fix. Impacted users may experience a disruption in the auth-api service of up to 15min while we apply a DB patch that will improve performance.

  4. Identified

    The issue has been identified, and we have applied mitigations on our end. We are currently observing the progress.

  5. Investigating

    We are investigating increased authentication errors affecting Grafana Cloud in the prod-us-west-0 region. Customers may experience intermittent failures for requests that require authentication. Our engineering team has identified database performance degradation affecting the authentication service and is investigating the underlying cause. We will provide another update as more information becomes available.