BitbucketUsers receiving errors when attempting to load BitBucket
Our team has mitigated the issue that caused error messages when loading Bitbucket. Bitbucket should now function correctly for all previously impacted users.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
Our team has mitigated the issue that caused error messages when loading Bitbucket. Bitbucket should now function correctly for all previously impacted users.
Resolved
This incident has been resolved.
Resolved
Beginning at 20:07 to 20:49 May 25 UTC, some customers in the US region may have experienced delayed APM, Browser, Mobile data in queries, charts, and dashboards as well as delayed or missing alert notifications for related data. We suppressed alerts from 20:36 to 20:48 May 25 UTC. We have resolved this interruption and impacted services have returned to normal operations.
Resolved
Beginning at 01:00 and 02:58 May 25 UTC, some customers in the US region may have experienced delayed APM data in queries, charts, and dashboards as well as delayed or missing alert notifications for related data. We suppressed alerts from 01:32 to 02:58 May 25 UTC. We have resolved this interruption and impacted services have returned to normal operations.
Resolved
Earlier today, some of our H100 nodes ran out of disk space which caused them to go offline. If your workloads were scheduled on these nodes, some of your tasks may have timed out during this window. We have taken steps to prevent this from happening again, and we have added credits to the accounts that were affected.
Resolved
Between 04:35 May 24 UTC and 07:34 May 24 UTC some customers in the US region may have experienced delayed APM data in dashboards. We have resolved this interruption and impacted services have returned to normal operations.
Resolved
This incident has been resolved.
Resolved
This issue is now resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Due to the way the feature flag was set, only neon employees were affected. The issue has been fully resolved, and all affected components are operating normally again.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved. If you have any questions about this, we have our 24/7 support team available via live chat in the control panel or via email at support@upcloud.com.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident is resolved.
Resolved
We experienced a service degradation on the US-01 cluster between 11:46 ET and 11:58 ET. During this 12-minute period, SDK API and Dashboard services hosted on the US-01 cluster were impacted by increased errors. SDK API errors were successfully retried. Our engineering team responded immediately when the issue was detected and restored full service functionality at 11:58 ET. All systems are now operating normally.
Resolved
The upstream infrastructure provider issue has resolved and Accelerate is provisioning new query engines successfully.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Dear Valued Customer, We recently experienced intermittent SMS and MMS failures. Our engineering and operations teams promptly investigated and resolved the issue. The root cause was identified, and corrective actions have been implemented to prevent recurrence. Incident Start Time: 20th May 2025 07:30 UTC Incident End Time: 20th May 2025 07:50 UTC If you have any questions or concerns regarding this incident, please contact Plivo Support. Support portal: https://support.plivo.com/hc/en-us
Resolved
The issue has been resolved and Meta-Llama-3.1-8B-Instruct response should be back to normal.
Resolved
The server provisioning failure issue has been resolved.
Resolved
This incident has been resolved.
Resolved
We have eliminated the slightly elevated error rates.
Resolved
The issue has been fully resolved. All affected components are operating normally again.
Resolved
### Summary On May 8, 2025, at 3:26pm UTC, Bitbucket Cloud experienced website and API latency due to an overloaded primary database. The event was caused by a backfill job running from an internal Atlassian service, which triggered an excessive call volume of expensive queries and pressure on database resources. As a result, the primary database automatically failed over, and Bitbucket services recovered in 15 minutes. Our real-time monitoring detected the incident immediately, and the high-intensity backfill job was stopped. However, following the failover, a backlog of retries from downstream services continued to impact overall database performance. Customers may have seen intermittent errors or website latency during this time. During this period following the failover, the engineering team implemented several strategies to further shed database load, successfully alleviating pressure on resources and improving performance. On May 9th at 11:19 AM UTC, Bitbucket Cloud systems were fully operational. ### **Impact** The overall impact occurred between May 8th, 2025, at 3:26 PM and May 9th at 11:19 AM UTC on Bitbucket Cloud. The incident resulted in increased latency and intermittent failures across Bitbucket Cloud services, including the website, API, and Bitbucket Pipelines. ### **Root cause** The issue was caused by an internal high-scale backfill job that triggered excessive load on certain API endpoints, which eventually impacted the database through resource-intensive queries and operations. This led to additional load from retries by dependent services, increasing the total recovery time. ### **Remedial action plan and next steps** We know that outages impact your productivity. While we have several testing and preventative processes in place, this specific issue wasn’t identified during our testing, as it was related to a specific high-scale backfill job run by an internal component, which impacted highly resource-intensive database queries. To prevent this type of incident from recurring, we are prioritizing the following improvement actions: * Improve database request routing so that more reads go to read replicas instead of the write-primary database. * Adjust rate limits for internal API endpoints with resource-intensive database operations. * Optimize database queries so that they can run more efficiently. * Tune retry policies from downstream services. We apologize to customers whose services were interrupted by this incident, and we are taking immediate steps to improve the platform’s reliability. Thanks, Atlassian Customer Support
Resolved
This issue has now been resolved. A database setting was changed which conflicted with the way search works when searching all sections. This setting has now been reverted and search is functioning as before.
Resolved
Latencies have remained normal since the last update. We are confident the situation is fully resolved.
Resolved
The issue has been fully resolved. All affected components are operating normally again.
Resolved
We've rolled out the fix and verified that customers should no longer be seeing self-signed TLS certificate errors and that custom-domain provisioning capability has been restored.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We have resolved this incident, and Airtable is operating normally again. We deeply apologize for the inconvenience.
Resolved
This incident has been resolved.
Resolved
The issue has been fully resolved. All affected components are operating normally again.
Resolved
### **SUMMARY** On April 25, 2025, between 18:18 and 18:33 UTC, Atlassian customers using Trello may have experienced service interruptions. The event was triggered by temporarily reduced capacity following a rollback deployment, with insufficient nodes to handle the load. Automated monitoring systems detected the incident within one minute and mitigated it by scaling up the deployment, which put Atlassian systems into a known-good state. The total time to resolution was about 15 minutes. ### **IMPACT** The overall impact was on April 25, 2025, between 18:18 and 18:33 UTC, on Trello. The incident caused service disruption to Trello users, resulting in reduced functionality, slower response times, and errors when performing key actions such as loading boards and cards. ### **ROOT CAUSE** The root cause of the incident was a failure to scale our nodes to optimal capacity caused by a release rollback. If an issue is found during a deployment, we can roll back to a previous release. In this case, a rollback was executed to a previous release that had already undergone a scaling-down process. As that rollback happened, more compute nodes needed to be available to handle the high traffic. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity and strive to avoid incidents like these. We are prioritizing the following efforts as next steps: * Improvements to rollback tooling including UX upgrades and pre-scaling * Conduct an updated incident response training focused on rollback tooling and best practices We apologize to customers whose services were impacted during this incident; we are taking steps designed to improve the platform’s performance and availability. Thanks, Atlassian Customer Support
Resolved
This incident has been resolved.
Resolved