Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Dec 28, 2023

Dec 26, 2023

  1. BeamPending Tasks

    From approximately 9:30 - 12:15 UTC, a bottleneck in our queueing system resulted in an elevated number of pending tasks. We've identified the cause of the issue and it is now resolved.

    Resolved

Dec 23, 2023

Dec 21, 2023

  1. PandaDocCan't create, view and edit documents

    We're all set! PandaDoc is working properly and the incident has been fully resolved. If you continue to experience any issues with this, please reach out to us at support@pandadoc.com.

    Resolved

Dec 20, 2023

  1. BitlyIssues with the Bitly web application

    From `17:35Z` through `18:21Z` on `2023-12-20` Bitly experienced a major outage affecting Website, API and Link In Bio functionality. The root cause was a change to internal service load balancing health checks which prevented services from being updated during a deployment. Link redirects remained functional during the incident. Webhooks were delayed during the incident.

    Resolved

  2. PandaDocCan't create documents

    We're all set! PandaDoc is working properly and the incident has been fully resolved. If you continue to experience any issues with this, please reach out to us at support@pandadoc.com.

    Resolved

Dec 19, 2023

Dec 15, 2023

  1. KeeperDuo Push in US region

    The Keeper engineering team has been researching the cause for recent Duo Push issues. Our investigation has resulted in the identification of a couple issues. The description is below: 1. The Duo Push services also have a direct dependency on the Keeper Push services which utilize websocket communication from the Keeper backend to the client device. 2. We determined that scaling additional instances of our push services was necessary to meet the growing usage of Duo and general push features across the Keeper end-user client apps 3. Scaling the backend instances immediately resolved the latency and the intermittent issues encountered by users. 4. In addition, we met with the Duo team, revisited our Duo integration code and we decided to rewrite some portions of the integration to reduce the number of dependencies and asynchronous processes. This is planned for release by December 20, 2023.

    Resolved

Dec 13, 2023

Dec 11, 2023

Dec 10, 2023

Dec 9, 2023

  1. NeonStuck projects-Minor

    The incident has been successfully resolved, and normal operations have been restored to all affected systems and services.

    Resolved

Dec 8, 2023

  1. TrelloTrello is unavailable

    ### **SUMMARY** On Nov 30 2023, between 14:04 and 16:57 UTC, Atlassian customers using Trello experienced errors when accessing and interacting with the application. This incident impacted Trello users on the iOS and Android mobile apps as well as those using the Trello web app. The event was triggered by the release of a code change that eventually overloaded a critical part of the Trello database. The incident was detected immediately by our automated monitoring systems and was mitigated by disabling the relevant code change. The issue was extended by the failure of a secondary service whose recovery caused an increase in load on the same critical part of the Trello database, which created a negative feedback loop. This secondary service recovery involved reestablishing over a million connections, with each connection attempt adding load to the same part of the Trello database. We attempted to aid the service recovery by intentionally blocking some of the inbound Trello traffic to reduce load on the database and by increasing the capacity of the Trello database to better handle the high load. Over time the connections were all successfully reestablished, which returned Trello to a known good state. The total time to resolution was just under 3 hours. ### **IMPACT** The overall impact was between Nov 30 2023, 14:04 UTC and Nov 30 2023, 16:57 UTC on the Trello product. The incident caused service disruption to all Trello customers. Our metrics show there were elevated API response times and increased error rates through the entire incident period, which indicates that most users were unable to load Trello at all or easily interact with the application in any way. The particular database collection that was overloaded was one that is necessary for the Trello service to make authorization decisions, which meant that all requests were impacted. ### **ROOT CAUSE** The issue was caused by a series of changes intended to standardize Trello’s approach to authorizing requests, but had the unintended side effect of modifying a database query from a [targeted operation](https://www.mongodb.com/docs/manual/core/sharded-cluster-query-router/#std-label-sharding-mongos-targeted) to a [broadcast operation](https://www.mongodb.com/docs/manual/core/sharded-cluster-query-router/#std-label-sharding-mongos-broadcast). Broadcast operations are more resource-intensive as they must be sent to _all_ database servers to be satisfied. These broadcast operations eventually overloaded some of the Trello database servers as Trello approached its daily peak usage period on Nov 30 2023. 1. The first change of this type was deployed over a period of seven days at the end of August and changed the authorization type used by our websocket service. This meant that newly established websocket connections required this new broadcast query. At any given moment, we have a great deal of _established_ websocket connections, but the usual rate of _new_ websocket connections is relatively low. Therefore, our monitoring systems only detected a slight increase in resource usage and flagged this change as a low priority performance regression. We acknowledged the regression and created a task to identify and reduce the resource demands of these new queries. 2. The second change of this type was deployed over the course of a few days before being fully rolled out on Nov 29, 2023, the day before this incident. This change caused the Trello application server to use the new broadcast query while authorizing standard web browser traffic, which is the vast majority of our traffic. The change was fully deployed at 19:34 UTC on Nov 29, which was during a low traffic period. The next day, as the application approached its daily peak traffic period, our monitoring on the database servers indicated they were overloaded. When these database nodes were overloaded, users' HTTP requests received very slow responses or HTTP 504 errors. As we activated our load shedding strategies, some users received HTTP 429 errors. The incident’s length can be attributed to a secondary failure where our websocket servers experienced a rapid increase in memory leading to processes crashing with OutOfMemoryErrors. As new servers came online and the websockets attempted to reconnected, they once again generated the broadcast queries on the Trello database servers. These broadcast queries continued to put load on the database, which meant the Trello API continued to have high latency, thus perpetuating the negative feedback loop. We are working to determine the root cause of the OutOfMemoryErrors. We also determined after the incident that due to the Trello application server making the load shedding decision AFTER performing the authorization step, the overloaded database servers were still being queried before the request was rejected. We are working to improve our load shedding strategies post incident. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity and we are continually working to improve our testing and preventative processes to prevent similar outages in the future. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Increase the capacity of our database \(completed during the incident\). * This action is the most critical and is aimed at preventing a recurrence of this particular incident and gracefully recover if the websocket service were to fail again. * Refactor the new authorization approach to avoid [broadcast operations](https://www.mongodb.com/docs/manual/core/sharded-cluster-query-router/#std-label-sharding-mongos-broadcast). * Add pre-deployment tests to avoid releasing unnecessary broadcast operations. * Determine the root cause of the secondary failure of the websocket service. Furthermore, we deploy our changes only after thorough review and automated testing, and we deploy them progressively using feature flags to avoid broad impact. To minimize the impact of breaking changes to our environments, we will implement additional preventative measures: * Ensure that our load-shedding strategies fail fast. * Add monitoring to observe [broadcast operations](https://www.mongodb.com/docs/manual/core/sharded-cluster-query-router/#std-label-sharding-mongos-broadcast) in all our environments. We apologize to customers who were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support

    Resolved

  2. MezmoIntermittent user session timeouts, requiring periodic re-authentication

    **Dates:**  Start Time: Monday, December 4, 2023, at 10:29 UTC End Time: Monday, December 4, 2023, at 12:01 UTC Duration: 92 minutes ‌ **What happened:** Web UI users were logged out frequently – usually within 1-2 minutes of logging in. Users could successfully login again without any issues, but the session would expire shortly afterwards. ‌ **Why it happened:** It was identified that both Web UI pods and the Redis database pods, which are responsible for storing user sessions, experienced a critical memory shortage, leading to uncontrolled data purging. When this same issue happened in July 2023, our engineering team deployed a fix that enhanced how Redis stores the user session keys. This fix successfully prevented any recurrence of the problem until today. The team is still determining what made it exceed the memory limit this time. ‌ **How we fixed it:** Initially, the Web UI pods were restarted, but that did not resolve the problem permanently. The engineering team then restarted the Redis database pods and the session stopped expiring. ‌ **What we are doing to prevent it from happening again:** The team will revise the previous fix, including implementing a mechanism for the pod to automatically restart upon reaching its limit and setting up alerts to notify an engineer when it's approaching that threshold.

    Resolved

Dec 7, 2023

  1. ZillizPrivate Link Connection Issues

    The incident has been resolved. Due to an an incident, from 8:30 to 10:36 UTC some users were unable to connect to their clusters over private links. We estimate this affected around less than 5% of our users. Once we detected the problem, we were able to resolve it and our systems continued operating normally.

    Resolved

Dec 6, 2023

  1. TypeformTypeform Results - Users are unable to download results

    We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

Dec 5, 2023

  1. TypeformTypeform Checkout Page - Unavailable - Checkout page unavailable

    We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

Dec 4, 2023

  1. MezmoUser sessions are timing out and customers are required to login again

    **Dates:** Start Time: Monday, June 19, 2023, at 10:31 UTC End Time: Monday, June 19, 2023, at 12:35 UTC Duration: 124 minutes **What happened:** Users were being logged out of our WebUI frequently – within 1-2 minutes of logging in. Users could successfully login again, but the new session would also expire quickly. **Why it happened:** The cache of logged in users held in our Redis database was being cleared every 1-2 minutes. This caused all user sessions to expire and new logins to be required. We have yet to ascertain why the cache was being periodically cleared at frequent intervals. **How we fixed it:** We restarted the pods running the Redis database and the cache behavior returned to normal. **What we are doing to prevent it from happening again:** We will investigate further to learn why the Redis cache was being frequently cleared.

    Resolved

Dec 2, 2023

Dec 1, 2023

Nov 30, 2023

  1. NeonStuck projects in us-east-2

    The incident has been successfully resolved, and normal operations have been restored to all affected systems and services. Our team conducted a thorough investigation, implemented necessary actions, and analyzed the underlying cause to prevent similar incidents in the future.

    Resolved

Older incidents