Resolved incident
Elevated levels of 429 errors on Cosmos for ~7 hours
Augment Code · Major · Started
- Duration
- < 1m
- Severity
- Major
- Status
- Resolved
Summary
Major · Resolved
Between roughly 8/25 6PM to 8/26 2AM Pacific Time, some users were not able to access or use Cosmos due to seeing the 429 - Too Many Requests error. It was because of a bug that got deployed into production that called an endpoint recursively. At around 1:30AM PT, we deployed the revert of the change containing the bug to production, and new Cosmos tabs opened after this revert operated normally. We have also suggested impacted users to do a hard refresh on their browser so that they get the latest fix. Since then we have been monitoring and our metrics show that this error has died down for production usage.
Event timeline
How this incident unfolded, from the official status page.
Resolved
Augment Code
Between roughly 8/25 6PM to 8/26 2AM Pacific Time, some users were not able to access or use Cosmos due to seeing the 429 - Too Many Requests error. It was because of a bug that got deployed into production that called an endpoint recursively. At around 1:30AM PT, we deployed the revert of the change containing the bug to production, and new Cosmos tabs opened after this revert operated normally. We have also suggested impacted users to do a hard refresh on their browser so that they get the latest fix. Since then we have been monitoring and our metrics show that this error has died down for production usage.