Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

May 26, 2025

May 25, 2025

  1. New RelicData and Alert Irregularities in the US Region

    Beginning at 20:07 to 20:49 May 25 UTC, some customers in the US region may have experienced delayed APM, Browser, Mobile data in queries, charts, and dashboards as well as delayed or missing alert notifications for related data. We suppressed alerts from 20:36 to 20:48 May 25 UTC. We have resolved this interruption and impacted services have returned to normal operations.

    Resolved

  2. New RelicAPM Data and Alert Irregularities in the US Region

    Beginning at 01:00 and 02:58 May 25 UTC, some customers in the US region may have experienced delayed APM data in queries, charts, and dashboards as well as delayed or missing alert notifications for related data. We suppressed alerts from 01:32 to 02:58 May 25 UTC. We have resolved this interruption and impacted services have returned to normal operations.

    Resolved

May 24, 2025

  1. BeamPending Tasks on H100

    Earlier today, some of our H100 nodes ran out of disk space which caused them to go offline. If your workloads were scheduled on these nodes, some of your tasks may have timed out during this window. We have taken steps to prevent this from happening again, and we have added credits to the accounts that were affected.

    Resolved

  2. New RelicAPM Dashboards lag in US region, no impact to alerts

    Between 04:35 May 24 UTC and 07:34 May 24 UTC some customers in the US region may have experienced delayed APM data in dashboards. We have resolved this interruption and impacted services have returned to normal operations.

    Resolved

May 23, 2025

May 22, 2025

  1. NeonVercel marketplace login doesn't work

    Due to the way the feature flag was set, only neon employees were affected. The issue has been fully resolved, and all affected components are operating normally again.

    Resolved

  2. UpCloudManaged Load Balancer

    This incident has been resolved. If you have any questions about this, we have our 24/7 support team available via live chat in the control panel or via email at support@upcloud.com.

    Resolved

May 21, 2025

  1. BrazeIncreased errors impacting US01

    We experienced a service degradation on the US-01 cluster between 11:46 ET and 11:58 ET. During this 12-minute period, SDK API and Dashboard services hosted on the US-01 cluster were impacted by increased errors. SDK API errors were successfully retried. Our engineering team responded immediately when the issue was detected and restored full service functionality at 11:58 ET. All systems are now operating normally.

    Resolved

May 20, 2025

  1. PlivoIntermittent SMS & MMS failure

    Dear Valued Customer, We recently experienced intermittent SMS and MMS failures. Our engineering and operations teams promptly investigated and resolved the issue. The root cause was identified, and corrective actions have been implemented to prevent recurrence. Incident Start Time: 20th May 2025 07:30 UTC Incident End Time: 20th May 2025 07:50 UTC If you have any questions or concerns regarding this incident, please contact Plivo Support. Support portal: https://support.plivo.com/hc/en-us

    Resolved

May 19, 2025

  1. BitbucketBitbucket has degraded performance

    ### Summary On May 8, 2025, at 3:26pm UTC, Bitbucket Cloud experienced website and API latency due to an overloaded primary database. The event was caused by a backfill job running from an internal Atlassian service, which triggered an excessive call volume of expensive queries and pressure on database resources. As a result, the primary database automatically failed over, and Bitbucket services recovered in 15 minutes. Our real-time monitoring detected the incident immediately, and the high-intensity backfill job was stopped. However, following the failover, a backlog of retries from downstream services continued to impact overall database performance. Customers may have seen intermittent errors or website latency during this time. During this period following the failover, the engineering team implemented several strategies to further shed database load, successfully alleviating pressure on resources and improving performance. On May 9th at 11:19 AM UTC, Bitbucket Cloud systems were fully operational. ### **Impact** The overall impact occurred between May 8th, 2025, at 3:26 PM and May 9th at 11:19 AM UTC on Bitbucket Cloud. The incident resulted in increased latency and intermittent failures across Bitbucket Cloud services, including the website, API, and Bitbucket Pipelines. ### **Root cause** The issue was caused by an internal high-scale backfill job that triggered excessive load on certain API endpoints, which eventually impacted the database through resource-intensive queries and operations. This led to additional load from retries by dependent services, increasing the total recovery time. ### **Remedial action plan and next steps** We know that outages impact your productivity. While we have several testing and preventative processes in place, this specific issue wasn’t identified during our testing, as it was related to a specific high-scale backfill job run by an internal component, which impacted highly resource-intensive database queries. To prevent this type of incident from recurring, we are prioritizing the following improvement actions: * Improve database request routing so that more reads go to read replicas instead of the write-primary database. * Adjust rate limits for internal API endpoints with resource-intensive database operations. * Optimize database queries so that they can run more efficiently. * Tune retry policies from downstream services. We apologize to customers whose services were interrupted by this incident, and we are taking immediate steps to improve the platform’s reliability. Thanks, Atlassian Customer Support

    Resolved

  2. ReadMeSearch results not showing up in "All" tab on hubs

    This issue has now been resolved. A database setting was changed which conflicted with the way search works when searching all sections. This setting has now been reverted and search is functioning as before.

    Resolved

May 18, 2025

May 16, 2025

  1. AirtableSome Bases Not Loading

    We have resolved this incident, and Airtable is operating normally again. We deeply apologize for the inconvenience.

    Resolved

  2. TrelloTrello is slow or unavailable

    ### **SUMMARY** On April 25, 2025, between 18:18 and 18:33 UTC, Atlassian customers using Trello may have experienced service interruptions. The event was triggered by temporarily reduced capacity following a rollback deployment, with insufficient nodes to handle the load. Automated monitoring systems detected the incident within one minute and mitigated it by scaling up the deployment, which put Atlassian systems into a known-good state. The total time to resolution was about 15 minutes. ### **IMPACT** The overall impact was on April 25, 2025, between 18:18 and 18:33 UTC, on Trello. The incident caused service disruption to Trello users, resulting in reduced functionality, slower response times, and errors when performing key actions such as loading boards and cards. ### **ROOT CAUSE** The root cause of the incident was a failure to scale our nodes to optimal capacity caused by a release rollback. If an issue is found during a deployment, we can roll back to a previous release. In this case, a rollback was executed to a previous release that had already undergone a scaling-down process. As that rollback happened, more compute nodes needed to be available to handle the high traffic. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity and strive to avoid incidents like these. We are prioritizing the following efforts as next steps: * Improvements to rollback tooling including UX upgrades and pre-scaling * Conduct an updated incident response training focused on rollback tooling and best practices We apologize to customers whose services were impacted during this incident; we are taking steps designed to improve the platform’s performance and availability. Thanks, Atlassian Customer Support

    Resolved

Older incidents