Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

May 16, 2025

  1. TrelloTrello is slow or unavailable for some users

    ### **SUMMARY** On May 5, 2025, between 2:08 p.m. and 4:29 p.m. UTC, some Atlassian customers using Trello were unable to view their boards or cards. The event was triggered by an unexpected error encountered by our infrastructure management tools, which resulted in an incorrect DNS configuration being deployed to a portion of our database. The incident was detected within four minutes by automated monitoring systems and mitigated by identifying the faulty portion of the database and performing a failover, which put Atlassian systems into a known good state. The total time to resolution was about two hours and 21 minutes. ### **IMPACT** The overall impact was on the Trello product on May 5, 2025, between 2:08 p.m. and 4:29 p.m. UTC. The incident caused service disruption to Trello customers whose accounts and boards contained or referenced data on the affected shard of our database. Additionally, some Trello customers would have experienced a service disruption due to our use of load-shedding tools during the incident to strategically block portions of our traffic to aid in recovery. ### **ROOT CAUSE** The day before the incident, on May 4, our infrastructure management tooling encountered an unexpected error when attempting to fetch the networking metadata on a particular host. This led to the host, which was a member of our database cluster, to incorrectly apply the default Operating System DNS configuration. This DNS configuration was not able to resolve internal domains, which led to a partial failure state of the node. The database continued to function normally and there was no immediate customer impact but in the background this incorrect DNS configuration led to the slow buildup of database sessions. These database sessions are usually short-lived and automatically expire when no longer needed, but the DNS misconfiguration prevented this automatic expiration. The database sessions eventually grew to the default maximum on this particular shard. At that point, the shard was unable to generate new sessions, which are required for all basic operations, and the Trello product began experiencing elevated error rates. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. While we have a number of testing and preventative processes in place, this specific issue wasn’t identified due to the isolated nature of the database session resource and monitoring gaps around this resource and around DNS resolution. We are prioritizing the following improvement actions designed to avoid repeating this type of incident: * Update our infrastructure management tool to use a safe fall-back DNS configuration in the case of unexpected errors. * Expand existing DNS monitoring to include the resolution of internal domains. * Expand existing database session count monitoring to include all database node types. Furthermore, we are prioritizing the following additional measures to reduce the duration of any future incidents: * Evaluate our incident response process to identify actions that can be streamlined for quicker resolution. We apologize to customers whose services were impacted during this incident; we are taking steps designed to improve the platform’s performance and availability. Thanks, Atlassian Customer Support

    Resolved

  2. StreamOutage in Us-East Edge

    There was a significant spike in requests that hit one of our Edge shards in the US-East region, leading to an exhaustion of resources. The incident lasted 4 minutes.

    Resolved

May 15, 2025

  1. CodecovExtended Queue Processing Times

    We are experiencing backups in our processing queues. This creates delays when processing coverage and test report upload files. This issue lasted from: 2025-05-14 8:00UTC to 2025-05-14 18:00UTC and has been fully resolved.

    Resolved

May 14, 2025

May 13, 2025

  1. NeonConsole unavailable

    The issue has been fully resolved. All affected components are operating normally again.

    Resolved

May 12, 2025

May 11, 2025

May 10, 2025

May 9, 2025

May 8, 2025

May 7, 2025

  1. AppsFlyer"Raw Data Export" page availability

    The issue with the "Raw Data Export" page availability is resolved. If you're still experiencing any related issues, please reach out to customer support.

    Resolved

May 6, 2025

May 5, 2025

  1. LeverCandidate count and search issue

    The issue where candidate searchability and pipeline count issues that were affecting customers with instances in our EU database has been resolved. The incident occurred from April 28, 2025 @3 PM EST to May 5, 2025 1:15 PM EST. There should be no further impact at this time, but please reach out to us at Support if any additional assistance is needed: https://help.lever.co/hc/en-us/requests/new

    Resolved

May 3, 2025

May 2, 2025

Older incidents