Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Sep 29, 2023

Sep 28, 2023

Sep 27, 2023

  1. PandaDocDocument Creation, Sending, and Viewing

    We're all set! Users can now create, send, and view documents. If you continue to experience any issues with this, please reach out to us at support@pandadoc.com.

    Resolved

Sep 26, 2023

  1. MapboxDelayed live traffic updates on Navigation APIs

    Customers using the NavSDK, Directions, and Map Matching APIs in the AP-southeast-2 region may have received stale traffic information in ETA and Visual Map, including live incidents. This issue began around 14:45 UTC and ended around 18:30 UTC. The root cause is under investigation and we are monitoring.

    Resolved

  2. NeonConsole Performance Degradation

    The incident has been resolved. The primary cause of this issue was a sudden spike in user activity on a certain API endpoint, resulting in an higher load on our control plane. As a precautionary measure, we are deploying rate limiting to prevent similar occurrences in the future.

    Resolved

Sep 25, 2023

Sep 22, 2023

  1. OpsgenieAtlassian Account login issues

    ### **SUMMARY** On Sep 13, 2023, between 12:00 PM UTC and 03: 30 PM UTC, some Atlassian users were unable to sign in to their accounts and use multiple Atlassian cloud products. The event was triggered by a misconfiguration of rate limits in an internal service which caused a cascading failure in sign-in and signup-related APIs. The incident was quickly detected by multiple automated monitoring systems. The incident was mitigated on Sep 13, 2023, 03: 30 PM UTC by the rollback of a feature and additional scaling of services which put Atlassian systems into a known good state. The total time to resolution was about 3 hours & 30 minutes. ‌ ### **IMPACT** The overall impact was between Sep 13, 2023, 12:00 PM UTC and Sep 13, 2023, 03: 30 PM UTC on multiple products. The Incident caused intermittent service disruption across all regions. Some users were unable to sign in for sessions. Other scenarios that temporarily failed were new user signups, profile retrieval, and password reset. During the incident we had a peak of 90% requests failing across authentication, user profile retrieval, and password reset use cases. ‌ ### **ROOT CAUSE** The issue was caused due to a misconfiguration of a rate limit in an internal core service. As a result, some sign-in requests over the limit received HTTP 429 errors. However, retry behavior for requests caused a multiplication of load which led to higher service degradation. As many internal services depend on each other, the call graph complexity led to a longer time to detect the actual faulty service. ‌ ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We are continuously improving our system's resiliency. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Audit and improve service rate limits and client retry and backoff behavior. * Improve scale and load test automation for complex service interactions. * Audit cross-service dependencies and minimize them where possible related to sign-in flows. ‌ Due to the unavailability of sign-in, some customers were unable to create support tickets. We are making additional process improvements to: * Enable our unauthenticated support contact form and notify users that it should be used when standard channels are not available. * Create status page notifications more quickly and ensure that for severe incidents, notifications to all subscribers are enabled. ‌ We apologize to users who were impacted during this incident; we are taking immediate steps to improve the platform’s reliability and availability. Thanks, Atlassian Customer Support

    Resolved

Sep 21, 2023

Sep 20, 2023

Sep 19, 2023

  1. KeeperVault login issues in US

    Our team received notification of API errors at 10:20AM PST. Between 10:20AM and 10:50AM the errors were intermittent and causing vault login errors in the US data center. The DevOps team resolved the issue fully at 10:53AM PST. After resolution of the issue, the engineering team researched the cause and found that a bug in the desktop app and also a related bug down the chain of processes in the backend API caused some client devices to generate a high number of full-sync API calls. We have identified the root cause and a patch is being applied to both Desktop App and Backend API components over the next 24 hours.

    Resolved

Sep 18, 2023

  1. OpsgenieMultiple product logins

    ### **SUMMARY** On August 30, 2023, between 4:07 and 5:30 UTC, some customers were unable to login to Atlassian's Cloud products using [id.atlassian.com](http://id.atlassian.com).  Logged-in users were also unable to switch accounts, change passwords, or log out. Users with existing sessions were not impacted. Between 5:32 and 6:00 UTC, traffic was incrementally restored to a previous build, mitigating the impact for users. The total time to resolution was one hour and 53 minutes. ### **IMPACT** Users were not able to login using Atlassian's shared account management system \([id.atlassian.com](http://id.atlassian.com)\). This affected users who were trying to login to the following products: Jira, Confluence, Trello, Opsgenie, mobile apps and ecosystem apps. Aside from the inability to login, there was no impact on other Atlassian products or features. ### **ROOT CAUSE** Multiple Set-Cookie headers were unintentionally modified so that only the last Set-Cookie header remained in the response to user's browsers.  The issue was caused by a change to Network Extensions within the Edge Network. As a result, users that needed a new session could not login.  Upon login, the users were redirected to login again and no session was created for them. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. While we have a number of testing and preventative processes in place, this specific issue was not detected in Atlassian's staging environment.  End-to-end tests did not cover the use case of multiple Set-Cookie headers in the single response and therefore this bug went unnoticed. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Automated tests to be put in place to validate that cookies are not being removed from responses. * Configuration of networking extensions will be guaranteed to be identical in staging and production to ensure errors are picked up earlier. Furthermore, we typically deploy our changes progressively by cloud region to avoid broad impact, but in this case, the change was not deemed risky and was deployed to all regions. To minimize the impact of breaking changes to our environments, we will implement additional preventative measures: * Changes to network extensions in the future will use progressive rollouts. * With staging being properly utilized, errors similar to this one will not be deployed to any production environments. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support

    Resolved

Sep 15, 2023

Sep 14, 2023

Sep 13, 2023

Sep 12, 2023

Sep 11, 2023

Sep 10, 2023

Sep 8, 2023

Sep 6, 2023

  1. NeonConsole is down

    This incident has been resolved. We'll be sharing a postmortem in the coming days, once we've gathered all the details.

    Resolved

  2. MezmoWeb UI is unresponsive and ingestion of log lines halted

    **Dates:** Start Time: 8:32 pm UTC, Tuesday August 29th, 2023 End Time: 10:04 pm UTC, Tuesday August 29th, 2023 Duration: 92 minutes ‌ **What happened:** Our Kong Gateway service stopped functioning and all connection requests to our ingestion service and web service failed. The Web UI did not load and log lines could not be sent by either our agent or API. Log lines sent using syslog were unaffected. Kong was unavailable for two periods of time: one lasting 27 minutes \(8:32 pm UTC to 8:59 pm UTC\) and another lasting 9 minutes \(9:43 pm UTC to 9:52 pm UTC\). Once Kong became available, the Web UI was immediately accessible again. Agents resent locally cached log lines \(as did any APIs implemented with retry strategies\). Our service then processed the backlog of log lines, passing them to downstream services such as alerting, live tail, archiving, and indexing \(which makes lines visible in the Web UI for searching, graphing, and timelines\). The extra processing was completed ~20 minutes after Kong returned to normal usage the first time, and ~10 minutes after the second time. ‌ **Why it happened:** The pods running our Kong Gateway were overwhelmed with connection requests. CPU increased to a point that health checks started to fail and the pods were shut down. We’ve determined through research and experimentation that the cause was a sudden, brief increase in the volume of traffic directed to our service. Our service is designed to handle increases in traffic, but these were approximately 100 times above normal usage. The source\(s\) of the traffic are unknown. The increase came in two spikes, which correspond to the two periods when Kong became unavailable. ‌ **How we fixed it:** We manually scaled up the number of pods devoted to running our Kong Gateway. During the first spike of traffic, we doubled the number of pods; during the second, we quadrupled the number. This certainly helped speed up the processing of the backlog of log lines sent by agents once Kong was again available. It’s unclear whether the higher number of pods would have been able to process the spikes of traffic as they were happening. ‌ **What we are doing to prevent it from happening again:** We are running our Kong service with more pods so there are more resources to handle any similar spikes in traffic. We will add auto-scaling to the Kong service so more pods are made available automatically as needed. We’ll also add metrics to identify the origin of any similar spikes in traffic.

    Resolved

Sep 5, 2023

Sep 1, 2023

  1. WorkRampDegraded Performance for Some Features

    Due to a backup of some platform processes, some functions, such as Guide publishing or emails being sent out from the platform, are seeing delays from 30 minutes to 1 hour.

    Resolved

Older incidents