BeamIncreased cold stay delays
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Amazon AWS has resolved their SMS delivery issue to Verizon customers.
Resolved
This incident has been resolved.
Resolved
We're all set! Users can now create, send, and view documents. If you continue to experience any issues with this, please reach out to us at support@pandadoc.com.
Resolved
Customers using the NavSDK, Directions, and Map Matching APIs in the AP-southeast-2 region may have received stale traffic information in ETA and Visual Map, including live incidents. This issue began around 14:45 UTC and ended around 18:30 UTC. The root cause is under investigation and we are monitoring.
Resolved
The incident has been resolved. The primary cause of this issue was a sudden spike in user activity on a certain API endpoint, resulting in an higher load on our control plane. As a precautionary measure, we are deploying rate limiting to prevent similar occurrences in the future.
Resolved
The issue causing the delay in message delivery has been resolved.
Resolved
### **SUMMARY** On Sep 13, 2023, between 12:00 PM UTC and 03: 30 PM UTC, some Atlassian users were unable to sign in to their accounts and use multiple Atlassian cloud products. The event was triggered by a misconfiguration of rate limits in an internal service which caused a cascading failure in sign-in and signup-related APIs. The incident was quickly detected by multiple automated monitoring systems. The incident was mitigated on Sep 13, 2023, 03: 30 PM UTC by the rollback of a feature and additional scaling of services which put Atlassian systems into a known good state. The total time to resolution was about 3 hours & 30 minutes. ### **IMPACT** The overall impact was between Sep 13, 2023, 12:00 PM UTC and Sep 13, 2023, 03: 30 PM UTC on multiple products. The Incident caused intermittent service disruption across all regions. Some users were unable to sign in for sessions. Other scenarios that temporarily failed were new user signups, profile retrieval, and password reset. During the incident we had a peak of 90% requests failing across authentication, user profile retrieval, and password reset use cases. ### **ROOT CAUSE** The issue was caused due to a misconfiguration of a rate limit in an internal core service. As a result, some sign-in requests over the limit received HTTP 429 errors. However, retry behavior for requests caused a multiplication of load which led to higher service degradation. As many internal services depend on each other, the call graph complexity led to a longer time to detect the actual faulty service. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We are continuously improving our system's resiliency. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Audit and improve service rate limits and client retry and backoff behavior. * Improve scale and load test automation for complex service interactions. * Audit cross-service dependencies and minimize them where possible related to sign-in flows. Due to the unavailability of sign-in, some customers were unable to create support tickets. We are making additional process improvements to: * Enable our unauthenticated support contact form and notify users that it should be used when standard channels are not available. * Create status page notifications more quickly and ensure that for severe incidents, notifications to all subscribers are enabled. We apologize to users who were impacted during this incident; we are taking immediate steps to improve the platform’s reliability and availability. Thanks, Atlassian Customer Support
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Our team received notification of API errors at 10:20AM PST. Between 10:20AM and 10:50AM the errors were intermittent and causing vault login errors in the US data center. The DevOps team resolved the issue fully at 10:53AM PST. After resolution of the issue, the engineering team researched the cause and found that a bug in the desktop app and also a related bug down the chain of processes in the backend API caused some client devices to generate a high number of full-sync API calls. We have identified the root cause and a patch is being applied to both Desktop App and Backend API components over the next 24 hours.
Resolved
This incident has been resolved. All systems operational.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
### **SUMMARY** On August 30, 2023, between 4:07 and 5:30 UTC, some customers were unable to login to Atlassian's Cloud products using [id.atlassian.com](http://id.atlassian.com). Logged-in users were also unable to switch accounts, change passwords, or log out. Users with existing sessions were not impacted. Between 5:32 and 6:00 UTC, traffic was incrementally restored to a previous build, mitigating the impact for users. The total time to resolution was one hour and 53 minutes. ### **IMPACT** Users were not able to login using Atlassian's shared account management system \([id.atlassian.com](http://id.atlassian.com)\). This affected users who were trying to login to the following products: Jira, Confluence, Trello, Opsgenie, mobile apps and ecosystem apps. Aside from the inability to login, there was no impact on other Atlassian products or features. ### **ROOT CAUSE** Multiple Set-Cookie headers were unintentionally modified so that only the last Set-Cookie header remained in the response to user's browsers. The issue was caused by a change to Network Extensions within the Edge Network. As a result, users that needed a new session could not login. Upon login, the users were redirected to login again and no session was created for them. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. While we have a number of testing and preventative processes in place, this specific issue was not detected in Atlassian's staging environment. End-to-end tests did not cover the use case of multiple Set-Cookie headers in the single response and therefore this bug went unnoticed. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Automated tests to be put in place to validate that cookies are not being removed from responses. * Configuration of networking extensions will be guaranteed to be identical in staging and production to ensure errors are picked up earlier. Furthermore, we typically deploy our changes progressively by cloud region to avoid broad impact, but in this case, the change was not deemed risky and was deployed to all regions. To minimize the impact of breaking changes to our environments, we will implement additional preventative measures: * Changes to network extensions in the future will use progressive rollouts. * With staging being properly utilized, errors similar to this one will not be deployed to any production environments. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We experienced elevated error rates on client and server side SDK initialization endpoints between midnight - 12:25am Sep 13 UTC.
Resolved
This incident has been resolved.
Resolved
This issue has been fully mitigated and continues to remain stable. We will continue to monitor this issue but do not expect any further service impact. Thanks you for your patience and partnership through this.
Resolved
This incident has been resolved.
Resolved
We experienced elevated error rates on client and server side SDK initialization endpoints between midnight - 2:10am Sep 12 UTC.
Resolved
The issue with Apigee Integrated Developer Portal has been resolved for all affected users as of Monday, 2023-09-11 12:25 US/Pacific. We thank you for your patience while we worked on resolving the issue.
Resolved
The signing authority certificate in the chain for smtp expired on September the 11th, the replacement has already been deployed. Mailchannels.net certificate did not expire until the 22nd of September
Resolved
This incident has been resolved.
Resolved
The incident has been successfully resolved, and normal operations have been restored to all affected systems and services.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
SDKs making polling requests experience increased error rates between 9:40PM - 11:15PM PT (09/08/2023 4:40 - 6:15 UTC). We've identified and fixed the issue and this has now been resolved.
Resolved
This incident has been resolved. We'll be sharing a postmortem in the coming days, once we've gathered all the details.
Resolved
**Dates:** Start Time: 8:32 pm UTC, Tuesday August 29th, 2023 End Time: 10:04 pm UTC, Tuesday August 29th, 2023 Duration: 92 minutes **What happened:** Our Kong Gateway service stopped functioning and all connection requests to our ingestion service and web service failed. The Web UI did not load and log lines could not be sent by either our agent or API. Log lines sent using syslog were unaffected. Kong was unavailable for two periods of time: one lasting 27 minutes \(8:32 pm UTC to 8:59 pm UTC\) and another lasting 9 minutes \(9:43 pm UTC to 9:52 pm UTC\). Once Kong became available, the Web UI was immediately accessible again. Agents resent locally cached log lines \(as did any APIs implemented with retry strategies\). Our service then processed the backlog of log lines, passing them to downstream services such as alerting, live tail, archiving, and indexing \(which makes lines visible in the Web UI for searching, graphing, and timelines\). The extra processing was completed ~20 minutes after Kong returned to normal usage the first time, and ~10 minutes after the second time. **Why it happened:** The pods running our Kong Gateway were overwhelmed with connection requests. CPU increased to a point that health checks started to fail and the pods were shut down. We’ve determined through research and experimentation that the cause was a sudden, brief increase in the volume of traffic directed to our service. Our service is designed to handle increases in traffic, but these were approximately 100 times above normal usage. The source\(s\) of the traffic are unknown. The increase came in two spikes, which correspond to the two periods when Kong became unavailable. **How we fixed it:** We manually scaled up the number of pods devoted to running our Kong Gateway. During the first spike of traffic, we doubled the number of pods; during the second, we quadrupled the number. This certainly helped speed up the processing of the backlog of log lines sent by agents once Kong was again available. It’s unclear whether the higher number of pods would have been able to process the spikes of traffic as they were happening. **What we are doing to prevent it from happening again:** We are running our Kong service with more pods so there are more resources to handle any similar spikes in traffic. We will add auto-scaling to the Kong service so more pods are made available automatically as needed. We’ll also add metrics to identify the origin of any similar spikes in traffic.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Due to a backup of some platform processes, some functions, such as Guide publishing or emails being sent out from the platform, are seeing delays from 30 minutes to 1 hour.
Resolved