Resolved incident
Elevated query errors in US region
Axiom · Unknown · Started
- Duration
- 13h 49m
- Severity
- Unknown
- Status
- Resolved
Summary
Unknown · Resolved
Query performance in the US region has been fully restored since 19:15 UTC. Between 18:05 and 19:15 UTC, some queries and dashboards returned errors and some monitor runs were missed or delayed. Ingest was not affected and no data was lost. We'll share a follow-up describing what we're changing to prevent this.
Event timeline
How this incident unfolded, from the official status page.
Investigating
Axiom
We've observed elevated error rates and slower responses for some queries in the US region. You may see failed or partially loaded dashboards, query errors (HTTP 500), or failed queries from the API and monitors. Ingest is not affected, and no data has been lost. Failed queries can be retried. We'll post another update within 30 minutes or as soon as we know more.
Monitoring
Axiom
We've rolled back the change behind the US query errors. Since 19:15 UTC, query error rates have been back to normal. We're monitoring and will post another update by 20:00 UTC.
Resolved
Axiom
Query performance in the US region has been fully restored since 19:15 UTC. Between 18:05 and 19:15 UTC, some queries and dashboards returned errors and some monitor runs were missed or delayed. Ingest was not affected and no data was lost. We'll share a follow-up describing what we're changing to prevent this.