Increased timeout errors in alert rule evaluations (AWS Germany, prod-eu-west-2)

Resolved

Minor impact

Started

Lasted 1h 38m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Monitoring

    Remediation measures have been applied, and we are monitoring the recovery.

  3. Identified

    The root cause of the issue has been indentified and we are working on the remediation steps.

  4. Investigating

    Since approximately 20:30 UTC on October 20, we have observed an increase in alert rule evaluation errors affecting rules that query Grafana Cloud Prometheus (Mimir) data sources in Cluster AWS Germany. Our engineering team is actively investigating and working on remediation.