Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Oct 1, 2025

  1. ConfluenceDegraded performance to multiple Atlassian experiences

    ### **SUMMARY** On September 22, 2025, between 04:38 and 04:48 UTC, Atlassian customers experienced connection errors preventing access to their [atlassian.net](http://atlassian.net/) sites. Some customers observed intermittent errors as services gradually recovered until 06:17 UTC. The event was triggered by a faulty configuration change to our Content Delivery Network \(CDN\). The change included an invalid hostname which prevented customers from successfully connecting to [atlassian.net](http://atlassian.net/) domains. The incident was detected within 1 minute by our monitoring systems and mitigated by rolling back the change, which put Atlassian systems into a known good state. The acute impact was resolved in 10 minutes and all lingering errors resolved in 1 hour 38 minutes. ### **IMPACT** The acute impact occurred on September 22, 2025, between 04:38 UTC and 04:48 UTC to Confluence, Compass and Jira, including Jira Service Management. The incident caused service disruption to customers when they attempted to load those products in their browser or interact with APIs. Between 04:48 UTC and 06:17 UTC some customers continued to observe intermittent errors as services gradually recovered. ### **ROOT CAUSE** The issue was caused by a change to the hostname configuration of our Content Delivery Network \(CDN\). As a result, the products mentioned could not receive connections, and the users received TLS handshake errors, followed by HTTP 503 and 403 errors. More specifically a new CDN configuration contained a resource name which conflicted with the existing customer-serving CDN resource, and was able to be deployed, overwriting it. The root cause of the incident was the failure in the detection of the bug by our pre-deployment validations and tests. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that you rely on our products for your daily operations and productivity, and we sincerely apologize for any impact this disruption has had on you, your team or your organisation. We are prioritizing the following improvement actions to help avoid repeating this type of incident: * Improved pre-deployment change controls and testing. * Improved validation of target configurations before deployment to ensure customer-serving CDN resources cannot their hostnames changed. * Sharding of our CDN configuration to enable progressive changes. We sincerely appreciate your understanding and patience as we improve our processes to provide a better customer experience. Thank you, Atlassian Customer Support

    Resolved

Sep 30, 2025

Sep 29, 2025

  1. JamfJamf School Outage - eu-central-1

    **Incident Date:** September 24, 2025 **Duration:** 15 minutes **Impact:** Temporary unavailability of Jamf School services in eu-central-1 **Summary and Timeline** On September 17th, Jamf School experienced a performance degradation. To remediate and prevent recurrence, we scheduled a schema change to add a supporting index to high-concurrency table. When executed, the index creation acquired an unexpected lock that blocked concurrent application queries, resulting in a full Jamf School service interruption for eu-central-1 customers. Automated and manual monitoring triggered rapid investigation; engineers terminated the locking DDL session and service fully recovered within 15 minutes. **Root Cause & Timeline** **Root Cause:** An online index creation on a high-concurrency table resulted in a blocking table-level lock due to contention and execution plan timing. This prevented critical application queries from proceeding, causing service unavailability. **Key Events:** * 08:30 UTC – Planned schema/index change initiated * 08:31 UTC – Monitoring alert: elevated errors / unresponsive application * 08:34 UTC – Incident response bridge \(war room\) activated * 08:38 UTC – Root cause isolated: blocking lock on core table from index creation * 08:44 UTC – Lock cleared / blocking process stopped * 08:46 UTC – Services fully recovered; continued elevated monitoring initiated. **Business & User Impact** * Jamf School web console: Unavailable for 15 minutes * Device management \(MDM communication\): Temporarily degraded \(devices unable to check in\) * Scope: eu-central-1 environment only * Data integrity: No data loss or corruption observed **Gaps Identified** * The eu-central-1 environment exhibited higher peak concurrent activity than other regions where the same index change had already been applied successfully. * The change was executed during a period of high traffic in eu-central-1 region * Staging and prior rollout experience did not fully reproduce the eu-central-1 peak concurrency, reducing the predictive value of the pre-change assessment. **Remediation Items & Next Steps** * The status page was updated to reflect the outage and ensure customer awareness – Complete * Implement detailed change risk assessment for all database changes \(even those identified as low risk\). * Add pre-change load/lock simulation harness \(prod-like concurrency\) * Enforce execution windows for high-risk database changes \(off-peak / maintenance window\)

    Resolved

Sep 28, 2025

Sep 26, 2025

  1. New RelicDelays in data and alerts in EU/US

    Between 21:48 UTC and 22:36 UTC on September 26, Friday, some customers in the US & EU region may have experienced delayed data in queries, charts, and dashboards as well as delayed or missing alert notifications for related data. We have investigated and resolved this service interruption and impacted services have returned to normal operations. To prevent our customers from receiving a high volume of false positive alert notifications, we temporarily suppressed notifications for affected data between 22:11 UTC and 22:30 UTC.

    Resolved

  2. BrexIncorrect data returned by ListExpenses Partner API

    Transactions retrieved by calling the list expenses API endpoint (https://developer.brex.com/openapi/expenses_api/#operation/listExpenses) between 1:30pm PT on September 18 and 5:30pm PT on September 22 may have contained incorrect expense data. This issue caused certain transactions to be assigned to the wrong users or to have incorrect merchant information.

    Resolved

  3. FrontApp not loading

    A cache node was lost and automatically recovered in 6 minutes. All systems should now be operational.

    Resolved

  4. 1PasswordProvisioning invite links are not working

    # **Provisioning Invites could not be accepted** **Date of Incident:** 2025-09-23 **Time of Incident \(UTC\):** 17:18 - 00:46 **Service\(s\) Affected:** Sign Up **Impact Duration:** 7h 28m ## **Summary** For 7 hours and 28 minutes, 1Password Provisioning invites could not be accepted, presenting to the user as an invite expiry. Invites could not be accepted due to a web browser routing defect that was not caught during development, review, or release. First identified by customer reports approximately two and a half hours after release, the issue was escalated to development teams and an incident was immediately called. The root cause was identified as a defect introduced by a web client modification, and a fix was created, tested, and released. By 00:46 UTC, the fix was deployed to all environments and service was fully restored. ## **Impact on Customers** * **Sign-up:** Provisioning invites could not be accepted. * **Number of Affected Customers \(approximate\):** 100% of provisioning invites could not be accepted * **Customer-facing impact**: Users clicking their invite links encountered a misleading ‘Invite Expired’ message. * **Geographic Regions Affected:** 1Password USA/Canada/EU/Enterprise ## **What Happened?** A change to the web client contained a router defect that incorrectly rendered provisioning invites as expired. Users were presented with an error message that erroneously stated the invite was expired. The change responsible for introducing the defect was able to be released because it was not captured under automatic change notification rules, was lacking automated test coverage, and was not included in the set of manual tests. * **Timeline of Events \(UTC\):** * 17:18: 1Password Release containing defect * 19:53 \(2 hours, 35 minutes later\) First customer report * 20:49 \(56 minutes later\) Escalation to developer teams * 21:01: \(12 minutes later\) Incident called * 22:02: \(1 hour, 1 minute later\) Root cause identified * 22:29: \(27 minutes later\) Fix created and testing initiated * 23:49: \(1 hour, 30 minutes later\) Fix merged * 00:46: \(57 minutes later\) Fix released and service fully restored * **Root Cause Analysis:** A change modified the order in which key provisioning web routes were rendered. As a result, the route handling provisioning invitations failed to use the correct query parameters and the invite rendered as expired. * **Contributing Factors:** Automated tests on this endpoint do not exist. Manual testing missed testing the Provisioning routes. The modified code was not covered by automatic change notification rules to notify the Provisioning team. An existing bug that can fail the resending of invites was an initial red herring during the investigation. ## **How Was It Resolved?** * **Resolution Steps:** The defect in the 1Password web client was corrected so provisioning invites would render correctly. * **Verification of Resolution:** 1Password engineering tested the changes and validated that the functionality was restored, as well as verifying that requests for the affected endpoints were successful after the fix was deployed. ## **What We Are Doing to Prevent Future Incidents** * **Improve automated tests:** We are enhancing our automated tests for the Provisioning Invite routes. * **Expand automatic change notifications:** Expanding coverage of automatic change notification rules for areas of code owned by the Provisioning team. ## **Next Steps and Communication** * No action is required from our customers at this time. Existing invites do not need to be resent and may be accepted. * If you are still experiencing issues, please contact our support team at [support@1password.com](mailto:support@1password.com). We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding. Sincerely, The 1Password Team

    Resolved

  5. KeeperEU Region login API errors

    On **September 10 at 8:42 AM PDT**, our monitoring detected elevated errors with the **Auth API in the EU region**. The issue was identified promptly and fully resolved by **8:49 AM PDT**. Total impact duration was approximately **7 minutes**. ### Impact * **Region affected:** EU * **Service affected:** Authentication API * **Customer impact:** Customers in the EU experienced failed authentication attempts during the incident window. ### Timeline \(PDT\) * **08:42** – Incident detected. Elevated Auth API errors in the EU region reported. * **08:43** – Engineering and DevOps teams engaged. * **08:46** – Root cause isolated to a Terraform-related configuration change involving a VPC endpoint. * **08:49** – Configuration change rolled back; Auth API errors resolved. ### Root Cause The outage was caused by a configuration change in our production tenant during an ongoing Terraform migration. As part of the migration, a new VPC endpoint was being added to production environments. During Terraform apply, a subnet association was also being created. A network disruption occurred between the creation of the VPC endpoint and the subnet association \(behavior not seen in prior testing\) leading to temporary Auth API connectivity failures in the EU region. A new Terraform migration plan has been created that eliminates the possibility of disruption.

    Resolved

  6. SpeechmaticsPortal and temporary auth key API outage

    We have continued to monitor this incident and can confirm it has now been resolved. We apologise for any inconvenience caused. If you are still facing issues with the service, please reach our to our Support team https://support.speechmatics.com.

    Resolved

Sep 25, 2025

  1. New RelicData Irregularities in the EU region

    Between 15:11 UTC and 15:38 UTC on September 25th, some customers in the EU region may have experienced delayed data in queries, charts, and dashboards as well as delayed or missing alert notifications for related data. There was no impact to data ingest. We have investigated and resolved this service interruption and impacted services have returned to normal operations.

    Resolved

Sep 24, 2025

Older incidents