Forge Intermittent Web Page Unavailability
Resolved
Major impact
Updates
Resolved
We are seeing a return to nominal performance for the last 30 mins along with corresponding internal metrics, and will mark this incident as resolved.
Monitoring
We are seeing stability return back to normal for the affected services, and are continuing to monitor uptime.
Monitoring
We have deployed the comprehensive patch, and are seeing improvements on our internal metrics and external error rates. We will continue monitoring the results to validate the impact.
Identified
We have not seen a significant improvement from our initial mitigations, but we are finalizing testing on the more comprehensive patch that will be rolled out shortly pending verification. We will update at that time, and monitor the incident from then on.
Identified
We are deploying a temporary configuration change around the integration rate limits to mitigate the impact while we are working on a larger architectural change to resolve the larger root cause. We will monitor for the impact of the mitigating fix and will update when we are able to push out the more comprehensive patch.
Identified
We have identified the technical root cause and are looking into a patch to resolve the rate limit errors that are blocking user flows.
Investigating
We are investigating an issue where the forge site web pages and the inference API are intermittently unavailable due to underlying rate limit issues.