Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Nov 3, 2023

  1. MailChannelsOutbound Filtering Blocked

    From 13:20 PST to 14:00 PST A significant portion of the outbound email was temporarily blocked. The filter has been fixed and we are processing normally now.

    Resolved

Nov 2, 2023

  1. UserVoicePortal Outage

    A security hardening aspect of our cluster infrastructure inadvertently prevented a healthy restart of our load-balancing services. This caused a propogation failure that prevented traffic from being served. We have resolved the issue and are investigating additional testing resources to verify it can't reoccur.

    Resolved

Nov 1, 2023

Oct 31, 2023

Oct 30, 2023

Oct 28, 2023

Oct 27, 2023

  1. NeonConsole and public API timeouts -Critical

    The incident has been successfully resolved, and normal operations have been restored to all affected systems and services. We found the root cause in doing too many processing operations for some of the API endpoints we currently use to provision new database connections. We find that this amount of processing operations can and should be reduced and this will stop these kind of errors from appearing in the future, and we will work on that.

    Resolved

Oct 26, 2023

  1. NeonIssues with connections to pooled endpoints

    The incident has been successfully resolved, and normal operations have been restored to all affected systems and services. Our team conducted a thorough investigation, implemented necessary actions, and analyzed the underlying cause to prevent similar incidents in the future.

    Resolved

Oct 24, 2023

  1. KeeperGovCloud Vault Login

    GovCloud Vault Login issues have been resolved. We will update this status event with additional details shortly.

    Resolved

Oct 23, 2023

  1. GOV.UKGOV.UK Search and some pages displaying error messages

    A deployment of Content Store caused a slow down and error rates to rise resulting in several user-facing issues on GOV.UK, including but not limited to, GOV.UK Search displaying error messages and some GOV.UK pages displaying errors to users.

    Resolved

Oct 19, 2023

Oct 18, 2023

Oct 17, 2023

Oct 16, 2023

Oct 14, 2023

  1. NeonIncreased latencies in Console API

    The incident has been successfully resolved, and normal operations have been restored to all affected systems and services. Our team conducted a thorough investigation, implemented necessary actions, and analyzed the underlying cause to prevent similar incidents in the future.

    Resolved

Oct 13, 2023

  1. PandaDocAPI Outage

    It has come to our attention that some PandaDoc API users experienced temporary deactivation of their API keys recently, causing a need to generate new ones. We are pleased to inform you that our investigation into the matter is complete and we have addressed the issue.  ‌ To be clear, PandaDoc was not part of a data breach and was not hacked. PandaDoc’s Sandbox API was deactivated for trialists to improve our security measures, and unfortunately this change temporarily affected paying customers.  ‌ If you have been using our API and were affected by this, we extend our sincere apologies for any disruption it caused or impact on your experience using PandaDoc. ‌ Thank you again for using PandaDoc, and please don’t hesitate to reply to this email if you have any questions.

    Resolved

Oct 9, 2023

Oct 7, 2023

Oct 6, 2023

  1. Atlassian StatuspageAtlassian Account login issues

    ### **SUMMARY** On Sep 13, 2023, between 12:00 PM UTC and 03: 30 PM UTC, some Atlassian users were unable to sign in to their accounts and use multiple Atlassian cloud products. The event was triggered by a misconfiguration of rate limits in an internal service which caused a cascading failure in sign-in and signup-related APIs. The incident was quickly detected by multiple automated monitoring systems. The incident was mitigated on Sep 13, 2023, 03: 30 PM UTC by the rollback of a feature and additional scaling of services which put Atlassian systems into a known good state. The total time to resolution was about 3 hours & 30 minutes. ‌ ### **IMPACT** The overall impact was between Sep 13, 2023, 12:00 PM UTC and Sep 13, 2023, 03: 30 PM UTC on multiple products. The Incident caused intermittent service disruption across all regions. Some users were unable to sign in for sessions. Other scenarios that temporarily failed were new user signups, profile retrieval, and password reset. During the incident we had a peak of 90% requests failing across authentication, user profile retrieval, and password reset use cases. ‌ ### **ROOT CAUSE** The issue was caused due to a misconfiguration of a rate limit in an internal core service. As a result, some sign-in requests over the limit received HTTP 429 errors. However, retry behavior for requests caused a multiplication of load which led to higher service degradation. As many internal services depend on each other, the call graph complexity led to a longer time to detect the actual faulty service. ‌ ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We are continuously improving our system's resiliency. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Audit and improve service rate limits and client retry and backoff behavior. * Improve scale and load test automation for complex service interactions. * Audit cross-service dependencies and minimize them where possible related to sign-in flows. ‌ Due to the unavailability of sign-in, some customers were unable to create support tickets. We are making additional process improvements to: * Enable our unauthenticated support contact form and notify users that it should be used when standard channels are not available.  * Create status page notifications more quickly and ensure that for severe incidents, notifications to all subscribers are enabled. ‌ We apologize to users who were impacted during this incident; we are taking immediate steps to improve the platform’s reliability and availability. Thanks, Atlassian Customer Support

    Resolved

Older incidents