UpCloud[US-CHI1]: Managed Kubernetes Deployment
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The issue has been identified and a fix has been implemented. The issue is resolved.
Resolved
The immediate issue of email alerts having not been sent has now been worked through by manually republishing the travel advice for the affected countries. We have tested and identified a workaround for avoiding a repeat of the issue, which has been communicated with the publisher. We will continue to investigate a permanent fix over the next few working days, but will close the incident in the meantime.
Resolved
This incident has been resolved.
Resolved
The issues affecting the search functionality in Confluence have been resolved, and services now operate normally for all affected customers.
Resolved
The backlog of analytics data has been fully processed, and metrics are now up to date.
Resolved
Dear Valued Customer, We recently experienced intermittent outbound call failures affecting users in the India region during the time window specified below. The issue has since been resolved, and all services are currently operating normally. Incident Start Time: 5th June 2025 05:25 UTC Incident End Time: 5th June 2025 05:50 UTC If you have any questions or concerns regarding this incident, please contact Plivo Support. Support portal: https://support.plivo.com/hc/en-us
Resolved
This incident has been resolved.
Resolved
The issue that was preventing metrics pages from loading in the Cockroach Cloud console for Basic and Standard tier clusters has been fully resolved. Metrics are now loading as expected, and we’ve received confirmation from our service provider that the underlying issue has been addressed. We will continue to monitor for any signs of recurrence, but no further impact is expected at this time. Thank you for your patience.
Resolved
This incident has been resolved.
Resolved
A subset of customers experienced a delay in metrics. All metrics have caught up at this point.
Resolved
This incident has been resolved.
Resolved
The issue has been resolved.
Resolved
We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Service is restored in us-east-1 and all services are operating normally.
Resolved
On June 2, 2025, between approximately 19:05 and 22:30 UTC, an issue occurred preventing live speed and incident data from being served to Traffic Maps v1 and some customers using the Matrix API. This impacted services reliant on real-time traffic updates. Customers using Traffic Maps v1, experienced delayed live traffic updates. ETAs from the Matrix API provided during the incident period were based on traffic data from around 20:00 UTC, and did not reflect changes in traffic conditions until service restoration. This issue is now resolved and all services are restored.
Resolved
This incident has been resolved.
Resolved
On June 2nd, Trello's search bar functionality was not working correctly. The issue has now been resolved, and the service operates normally for all affected customers.
Resolved
Incident Start AWS experienced a brief network blip in the eu-west-1, eu-west-2, and eu-central-1 regions. Clusters in these regions were unavailable between June 1, 19:29–19:35 UTC. Incident End The network issue affecting clusters in the impacted AWS regions has been fully resolved *automatically* as of 19:35 UTC. All affected clusters are now operating normally. Please contact support if you have any concerns.
Resolved
This issue has been resolved
Resolved
All Services are back online & fully operational
Resolved
The issue has been fully resolved. All affected components are operating normally again.
Resolved
We are glad to inform you that the incident is resolved. Thanks for your understanding & sorry for any inconvenience this has caused. REF: ZD 868409 | INC-104
Resolved
On May 30th, there was an issue where Bitbucket pipeline steps were queued and delayed from starting. This problem has now been resolved, and the service operates normally for all customers.
Resolved
This incident has been resolved.
Resolved
There was an issue with one of our upstream providers. Service is remaining stable and we’re investigating the root cause with our provider.
Resolved
This issue has been resolved.
Resolved
The issue with our Federal web application has been resolved. Impacted services have returned to normal operation
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
### **SUMMARY** On May 15, 2025, between 13:55 and 14:18 UTC, Atlassian customers using the Trello product experienced errors or slow loading times when attempting to view their cards and boards. The event was triggered by a database plan cache expiring and high resource usage caused by subsequent database query planning operations. The particular database shard that was impacted held data that was required for every card load. The incident was detected within two minutes by the automated monitoring system and mitigated by increasing resources available to the affected database shard, which put Atlassian systems into a known good state. The total time to resolution was about 23 minutes. ### **IMPACT** The overall impact was between May 15, 2025, 13:55 and May 15, 2025, 14:18 UTC on the Trello product. The incident caused service disruption for all Trello customers. ### **ROOT CAUSE** The issue was caused by a query plan expiring from the database cache, which caused incoming queries to go through a replanning operation. These queries had multiple plans that could satisfy them, and depending on the size of the query, one plan might be significantly more efficient than another. This caused the query planner to perform a great many more replanning operations than usual, which consumed all of the CPU on the server for a brief moment. Once the CPU was consumed, the planning operations themselves began taking too long and therefore required constant replanning in an effort to find more efficient options. This negative feedback loop could not be broken without intervention. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. While we have a number of testing and preventative processes in place, this specific issue wasn’t identified because it would only occur under very distinct conditions, including the amount of load and the order of database queries. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Review our capacity planning thresholds and ensure that all shards have sufficient overhead to handle unexpected load. * Improve query planner performance by: * Implement hinting for known problematic query shapes to circumvent the query planner. * Investigate long-term generalized solutions to prevent query planner thrashing. Furthermore, we are prioritizing the following additional measures to reduce the impact of any future incidents: * Analyze and reduce single points of failure for loading Trello boards and cards. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support
Resolved
We have not seen a return of the traffic that caused this incident after removing the temporary block. We will open a new incident if we need to take action again.
Resolved
The issue with the Administration accessibility has now been resolved, and the service is operating normally for all affected customers.
Resolved
Monitoring has indicated that users are no longer receiving the error messages caused by this issue and it should now be fully resolved.
Resolved