NVIDIA NGCCloud Functions Invocation Performance is Degraded for Some Customers
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
## **Incident RCA: Job Site Outage** **Incident Date:** October 3, 2025 **Start Time:** 4:51 PM ET **End Time:** 6:00 PM ET ### **Impact Summary** Between 4:51 PM and 6:00 PM ET on October 3, 2025, users experienced degraded performance and eventual unavailability across multiple job boards. ### **Root Cause** The outage was triggered by an unexpected pattern of incoming requests that, while not unusually high in volume, each initiated resource-intensive queries. This combination of request frequency and prolonged query execution overwhelmed backend systems, leading to degraded performance and service unavailability. The traffic resembled automated crawling behavior, which our systems were not optimized to handle at that level of query complexity and concurrency. ### **Resolution** To reduce server load and restore service availability, the impacted job site was temporarily disabled. This action immediately alleviated pressure on backend systems and allowed all job boards to recover. The site remained offline over the weekend and was re-enabled on October 6, 2025, after Engineering confirmed that the irregular traffic had subsided. ### **Preventive Actions** To prevent similar incidents in the future, the following measures have been implemented: * **Enhanced Caching:** Improvements to our job site caching mechanisms now limit the ability of similar request patterns to generate excessive server queries. * **Updated Crawling Guidelines:** We have revised our web crawling directives to encourage automated crawlers to reduce the frequency of their requests. This change helps maintain system stability by minimizing the impact of high-volume automated traffic from compliant sources.
Resolved
This incident has been resolved.
Resolved
Between 1:35 PM PT and 3:25 PM PT users experienced partial disruption with Qwen 235b Instruct model. We have deployed a fix and the issue is now resolved.
Resolved
Elevated 500 errors on a back-end endpoint preventing some apps from loading
Resolved
We've confirmed that any backlog was processed and the system is stable.
Resolved
This incident has been resolved.
Resolved
The Paypal Payments Processing Issue was caused by something on Paypal’s end. For more information, when it is availble, please check here: [https://www.paypal-status.com/history/production](https://www.paypal-status.com/history/production)
Resolved
The issue has been fully resolved. All affected components are operating normally again.
Resolved
The incident has been resolved. If you continue to experience issues, please contact Support for assistance.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
From 2025/10/14 05:37 UTC to 2025/10/14 06:29 UTC, we experienced elevated levels of 206 Partial Content responses on the Search Box APIs due to a lack of POI results in the search.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Between 14:00 UTC to 16:00 UTC, we experienced automation rule functionality degradation for Confluence, Jira Work Management, Jira Service Management, and Jira. Impact Some customers using Jira, Confluence, and Jira Service Management were not able to create or update Automation rules, and some Automation rules which triggered webhooks might have been throttled and require re-running. Approximately 50 percent of Automation API calls were affected in the us-east-1 region with customers receiving 429 status codes. Current Status The incident has been mitigated by increasing the API Gateway rate limit service quota and disabling the internal service that was causing high traffic volume. Next Steps We are conducting a root cause analysis of the internal service. A post incident review will be conducted.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident is been resolved
Resolved
This incident has been resolved.
Resolved
This incident has been resolved. Dashboard Operations are back to normal.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Update: The issue is now resolved. The engineering team has implemented a fix and confirmed that normal provisioning operations have been restored.
Resolved
This incident has been resolved.
Resolved
This incident is now resolved. Thank you for your patience and understanding
Resolved
Delays have fully recovered to real-time. We have seen no indication of any data loss as part of this incident.
Resolved
This incident has been resolved.
Resolved
The issue has been fully resolved. All affected components are operating normally again.
Resolved
The performance issue with our API server has been resolved.
Resolved
Between approximately 13:42 and 14:18 UTC attempted webhook deliveries failed while our webhooks database was unavailable due to an unplanned outage. Webhooks messages during that period were lost and will not be delivered. We’ve prioritized work to make webhooks more resilient to this sort of failure. Email message sending was not impacted during this event.
Resolved
This incident has been resolved.
Resolved
From 15:44 - 16:33 UTC, Asana users may have seen stale data in the web app. In some cases, users may have experienced crashes and failed API requests.
Resolved
This incident is resolved now.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved