MailChannelsOutbound Filtering Blocked
From 13:20 PST to 14:00 PST A significant portion of the outbound email was temporarily blocked. The filter has been fixed and we are processing normally now.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
From 13:20 PST to 14:00 PST A significant portion of the outbound email was temporarily blocked. The filter has been fixed and we are processing normally now.
Resolved
The issues at Cloudflare seem to be resolving
Resolved
This incident has been resolved.
Resolved
Data synchronization issues with GitHub have been fully resolved and all systems are operational
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This issue has been resolved. Data export has returned to normal operation
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
A security hardening aspect of our cluster infrastructure inadvertently prevented a healthy restart of our load-balancing services. This caused a propogation failure that prevented traffic from being served. We have resolved the issue and are investigating additional testing resources to verify it can't reoccur.
Resolved
We fixed the configuration error preventing users from connecting to inactive endpoints. Please accept our apologies for the inconvenience.
Resolved
The error is now resolved and all systems are functional.
Resolved
This incident has been resolved.
Resolved
Subprocessor is now stable
Resolved
This incident has been resolved.
Resolved
The upstream provider incident has been resolved, and SSO authentication is working again.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Operation back to normal.
Resolved
Vendor is available Billing will processed
Resolved
The incident has been successfully resolved, and normal operations have been restored to all affected systems and services. We found the root cause in doing too many processing operations for some of the API endpoints we currently use to provision new database connections. We find that this amount of processing operations can and should be reduced and this will stop these kind of errors from appearing in the future, and we will work on that.
Resolved
The incident has been successfully resolved, and normal operations have been restored to all affected systems and services. Our team conducted a thorough investigation, implemented necessary actions, and analyzed the underlying cause to prevent similar incidents in the future.
Resolved
This incident has been resolved.
Resolved
The incident has been resolved, console is back online.
Resolved
GovCloud Vault Login issues have been resolved. We will update this status event with additional details shortly.
Resolved
This incident has been resolved.
Resolved
A deployment of Content Store caused a slow down and error rates to rise resulting in several user-facing issues on GOV.UK, including but not limited to, GOV.UK Search displaying error messages and some GOV.UK pages displaying errors to users.
Resolved
We're all set! Users can now use the application. If you continue to experience any issues with this, please reach out to us at support@pandadoc.com.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The incident has been successfully resolved, and normal latencies are restored in the affected region. Our team conducted a thorough investigation, implemented necessary actions, and analyzed the underlying cause to prevent similar incidents in the future.
Resolved
This incident has been resolved.
Resolved
The incident has been resolved
Resolved
The incident has been successfully resolved, and normal operations have been restored to all affected systems and services. Our team conducted a thorough investigation, implemented necessary actions, and analyzed the underlying cause to prevent similar incidents in the future.
Resolved
Quay.io has indicated that the incident is resolved on their end.
Resolved
It has come to our attention that some PandaDoc API users experienced temporary deactivation of their API keys recently, causing a need to generate new ones. We are pleased to inform you that our investigation into the matter is complete and we have addressed the issue. To be clear, PandaDoc was not part of a data breach and was not hacked. PandaDoc’s Sandbox API was deactivated for trialists to improve our security measures, and unfortunately this change temporarily affected paying customers. If you have been using our API and were affected by this, we extend our sincere apologies for any disruption it caused or impact on your experience using PandaDoc. Thank you again for using PandaDoc, and please don’t hesitate to reply to this email if you have any questions.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
### **SUMMARY** On Sep 13, 2023, between 12:00 PM UTC and 03: 30 PM UTC, some Atlassian users were unable to sign in to their accounts and use multiple Atlassian cloud products. The event was triggered by a misconfiguration of rate limits in an internal service which caused a cascading failure in sign-in and signup-related APIs. The incident was quickly detected by multiple automated monitoring systems. The incident was mitigated on Sep 13, 2023, 03: 30 PM UTC by the rollback of a feature and additional scaling of services which put Atlassian systems into a known good state. The total time to resolution was about 3 hours & 30 minutes. ### **IMPACT** The overall impact was between Sep 13, 2023, 12:00 PM UTC and Sep 13, 2023, 03: 30 PM UTC on multiple products. The Incident caused intermittent service disruption across all regions. Some users were unable to sign in for sessions. Other scenarios that temporarily failed were new user signups, profile retrieval, and password reset. During the incident we had a peak of 90% requests failing across authentication, user profile retrieval, and password reset use cases. ### **ROOT CAUSE** The issue was caused due to a misconfiguration of a rate limit in an internal core service. As a result, some sign-in requests over the limit received HTTP 429 errors. However, retry behavior for requests caused a multiplication of load which led to higher service degradation. As many internal services depend on each other, the call graph complexity led to a longer time to detect the actual faulty service. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We are continuously improving our system's resiliency. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Audit and improve service rate limits and client retry and backoff behavior. * Improve scale and load test automation for complex service interactions. * Audit cross-service dependencies and minimize them where possible related to sign-in flows. Due to the unavailability of sign-in, some customers were unable to create support tickets. We are making additional process improvements to: * Enable our unauthenticated support contact form and notify users that it should be used when standard channels are not available. * Create status page notifications more quickly and ensure that for severe incidents, notifications to all subscribers are enabled. We apologize to users who were impacted during this incident; we are taking immediate steps to improve the platform’s reliability and availability. Thanks, Atlassian Customer Support
Resolved