monday.comInvestigating reports of issues with our automations
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
The incident has been resolved. Any missing statistics during the incident haven't been lost and will show up in the statistics page later today.
Resolved
### **SUMMARY** On September 22, 2025, between 04:38 and 04:48 UTC, Atlassian customers experienced connection errors preventing access to their [atlassian.net](http://atlassian.net/) sites. Some customers observed intermittent errors as services gradually recovered until 06:17 UTC. The event was triggered by a faulty configuration change to our Content Delivery Network \(CDN\). The change included an invalid hostname which prevented customers from successfully connecting to [atlassian.net](http://atlassian.net/) domains. The incident was detected within 1 minute by our monitoring systems and mitigated by rolling back the change, which put Atlassian systems into a known good state. The acute impact was resolved in 10 minutes and all lingering errors resolved in 1 hour 38 minutes. ### **IMPACT** The acute impact occurred on September 22, 2025, between 04:38 UTC and 04:48 UTC to Confluence, Compass and Jira, including Jira Service Management. The incident caused service disruption to customers when they attempted to load those products in their browser or interact with APIs. Between 04:48 UTC and 06:17 UTC some customers continued to observe intermittent errors as services gradually recovered. ### **ROOT CAUSE** The issue was caused by a change to the hostname configuration of our Content Delivery Network \(CDN\). As a result, the products mentioned could not receive connections, and the users received TLS handshake errors, followed by HTTP 503 and 403 errors. More specifically a new CDN configuration contained a resource name which conflicted with the existing customer-serving CDN resource, and was able to be deployed, overwriting it. The root cause of the incident was the failure in the detection of the bug by our pre-deployment validations and tests. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that you rely on our products for your daily operations and productivity, and we sincerely apologize for any impact this disruption has had on you, your team or your organisation. We are prioritizing the following improvement actions to help avoid repeating this type of incident: * Improved pre-deployment change controls and testing. * Improved validation of target configurations before deployment to ensure customer-serving CDN resources cannot their hostnames changed. * Sharding of our CDN configuration to enable progressive changes. We sincerely appreciate your understanding and patience as we improve our processes to provide a better customer experience. Thank you, Atlassian Customer Support
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved. Please contact support@crusoe.ai if you are still experiencing any issues.
Resolved
This incident has been resolved.
Resolved
We have continued to see good performance for pre-recorded audio so we are now marking this issue as resolved.
Resolved
This incident has been resolved following a fix from our upstream provider. Form notification emails are now being delivered as expected. We appreciate your patience while we worked to restore normal service.
Resolved
This incident is resolved.
Resolved
Between 13:10 UTC to 14:00 UTC, we experienced notification delay for Jira Service Management, Opsgenie, and Compass in only EU-region. The issue has been resolved and the service is operating normally.
Resolved
This incident has been resolved.
Resolved
A fix has been implemented. This incident is now resolved
Resolved
We have continued to see good performance since rolling back the update that was identified to be the cause of this issue, so we are marking this issue as resolved.
Resolved
The incident has been resolved by reverting to a previous release.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The issue has been successfully resolved. Please refresh your browser to access the platform. Thank you for your patience
Resolved
We have resolved the issue.
Resolved
The Update Feed issue has been successfully resolved. Please refresh your browser to access the platform. Thank you so much for your patience!
Resolved
**Incident Date:** September 24, 2025 **Duration:** 15 minutes **Impact:** Temporary unavailability of Jamf School services in eu-central-1 **Summary and Timeline** On September 17th, Jamf School experienced a performance degradation. To remediate and prevent recurrence, we scheduled a schema change to add a supporting index to high-concurrency table. When executed, the index creation acquired an unexpected lock that blocked concurrent application queries, resulting in a full Jamf School service interruption for eu-central-1 customers. Automated and manual monitoring triggered rapid investigation; engineers terminated the locking DDL session and service fully recovered within 15 minutes. **Root Cause & Timeline** **Root Cause:** An online index creation on a high-concurrency table resulted in a blocking table-level lock due to contention and execution plan timing. This prevented critical application queries from proceeding, causing service unavailability. **Key Events:** * 08:30 UTC – Planned schema/index change initiated * 08:31 UTC – Monitoring alert: elevated errors / unresponsive application * 08:34 UTC – Incident response bridge \(war room\) activated * 08:38 UTC – Root cause isolated: blocking lock on core table from index creation * 08:44 UTC – Lock cleared / blocking process stopped * 08:46 UTC – Services fully recovered; continued elevated monitoring initiated. **Business & User Impact** * Jamf School web console: Unavailable for 15 minutes * Device management \(MDM communication\): Temporarily degraded \(devices unable to check in\) * Scope: eu-central-1 environment only * Data integrity: No data loss or corruption observed **Gaps Identified** * The eu-central-1 environment exhibited higher peak concurrent activity than other regions where the same index change had already been applied successfully. * The change was executed during a period of high traffic in eu-central-1 region * Staging and prior rollout experience did not fully reproduce the eu-central-1 peak concurrency, reducing the predictive value of the pre-change assessment. **Remediation Items & Next Steps** * The status page was updated to reflect the outage and ensure customer awareness – Complete * Implement detailed change risk assessment for all database changes \(even those identified as low risk\). * Add pre-change load/lock simulation harness \(prod-like concurrency\) * Enforce execution windows for high-risk database changes \(off-peak / maintenance window\)
Resolved
All data for journey analytics has been successfully restored and systems are operating normally.
Resolved
Between 21:48 UTC and 22:36 UTC on September 26, Friday, some customers in the US & EU region may have experienced delayed data in queries, charts, and dashboards as well as delayed or missing alert notifications for related data. We have investigated and resolved this service interruption and impacted services have returned to normal operations. To prevent our customers from receiving a high volume of false positive alert notifications, we temporarily suppressed notifications for affected data between 22:11 UTC and 22:30 UTC.
Resolved
This incident has been resolved. Our team recommends reconnecting the Confluence integration in Vanta to resume policy syncing and documents importing.
Resolved
Transactions retrieved by calling the list expenses API endpoint (https://developer.brex.com/openapi/expenses_api/#operation/listExpenses) between 1:30pm PT on September 18 and 5:30pm PT on September 22 may have contained incorrect expense data. This issue caused certain transactions to be assigned to the wrong users or to have incorrect merchant information.
Resolved
A cache node was lost and automatically recovered in 6 minutes. All systems should now be operational.
Resolved
# **Provisioning Invites could not be accepted** **Date of Incident:** 2025-09-23 **Time of Incident \(UTC\):** 17:18 - 00:46 **Service\(s\) Affected:** Sign Up **Impact Duration:** 7h 28m ## **Summary** For 7 hours and 28 minutes, 1Password Provisioning invites could not be accepted, presenting to the user as an invite expiry. Invites could not be accepted due to a web browser routing defect that was not caught during development, review, or release. First identified by customer reports approximately two and a half hours after release, the issue was escalated to development teams and an incident was immediately called. The root cause was identified as a defect introduced by a web client modification, and a fix was created, tested, and released. By 00:46 UTC, the fix was deployed to all environments and service was fully restored. ## **Impact on Customers** * **Sign-up:** Provisioning invites could not be accepted. * **Number of Affected Customers \(approximate\):** 100% of provisioning invites could not be accepted * **Customer-facing impact**: Users clicking their invite links encountered a misleading ‘Invite Expired’ message. * **Geographic Regions Affected:** 1Password USA/Canada/EU/Enterprise ## **What Happened?** A change to the web client contained a router defect that incorrectly rendered provisioning invites as expired. Users were presented with an error message that erroneously stated the invite was expired. The change responsible for introducing the defect was able to be released because it was not captured under automatic change notification rules, was lacking automated test coverage, and was not included in the set of manual tests. * **Timeline of Events \(UTC\):** * 17:18: 1Password Release containing defect * 19:53 \(2 hours, 35 minutes later\) First customer report * 20:49 \(56 minutes later\) Escalation to developer teams * 21:01: \(12 minutes later\) Incident called * 22:02: \(1 hour, 1 minute later\) Root cause identified * 22:29: \(27 minutes later\) Fix created and testing initiated * 23:49: \(1 hour, 30 minutes later\) Fix merged * 00:46: \(57 minutes later\) Fix released and service fully restored * **Root Cause Analysis:** A change modified the order in which key provisioning web routes were rendered. As a result, the route handling provisioning invitations failed to use the correct query parameters and the invite rendered as expired. * **Contributing Factors:** Automated tests on this endpoint do not exist. Manual testing missed testing the Provisioning routes. The modified code was not covered by automatic change notification rules to notify the Provisioning team. An existing bug that can fail the resending of invites was an initial red herring during the investigation. ## **How Was It Resolved?** * **Resolution Steps:** The defect in the 1Password web client was corrected so provisioning invites would render correctly. * **Verification of Resolution:** 1Password engineering tested the changes and validated that the functionality was restored, as well as verifying that requests for the affected endpoints were successful after the fix was deployed. ## **What We Are Doing to Prevent Future Incidents** * **Improve automated tests:** We are enhancing our automated tests for the Provisioning Invite routes. * **Expand automatic change notifications:** Expanding coverage of automatic change notification rules for areas of code owned by the Provisioning team. ## **Next Steps and Communication** * No action is required from our customers at this time. Existing invites do not need to be resent and may be accepted. * If you are still experiencing issues, please contact our support team at [support@1password.com](mailto:support@1password.com). We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding. Sincerely, The 1Password Team
Resolved
This issue is now resolved and SCORM module statuses are syncing as expected.
Resolved
On **September 10 at 8:42 AM PDT**, our monitoring detected elevated errors with the **Auth API in the EU region**. The issue was identified promptly and fully resolved by **8:49 AM PDT**. Total impact duration was approximately **7 minutes**. ### Impact * **Region affected:** EU * **Service affected:** Authentication API * **Customer impact:** Customers in the EU experienced failed authentication attempts during the incident window. ### Timeline \(PDT\) * **08:42** – Incident detected. Elevated Auth API errors in the EU region reported. * **08:43** – Engineering and DevOps teams engaged. * **08:46** – Root cause isolated to a Terraform-related configuration change involving a VPC endpoint. * **08:49** – Configuration change rolled back; Auth API errors resolved. ### Root Cause The outage was caused by a configuration change in our production tenant during an ongoing Terraform migration. As part of the migration, a new VPC endpoint was being added to production environments. During Terraform apply, a subnet association was also being created. A network disruption occurred between the creation of the VPC endpoint and the subnet association \(behavior not seen in prior testing\) leading to temporary Auth API connectivity failures in the EU region. A new Terraform migration plan has been created that eliminates the possibility of disruption.
Resolved
This incident has been resolved. We are currently investigating the root cause of the issue.
Resolved
We have continued to monitor this incident and can confirm it has now been resolved. We apologise for any inconvenience caused. If you are still facing issues with the service, please reach our to our Support team https://support.speechmatics.com.
Resolved
We have resolved the issue. We will continue monitoring for stability.
Resolved
All the impacted customer bases should be back and running again.
Resolved
This incident has been resolved.
Resolved
Between 15:11 UTC and 15:38 UTC on September 25th, some customers in the EU region may have experienced delayed data in queries, charts, and dashboards as well as delayed or missing alert notifications for related data. There was no impact to data ingest. We have investigated and resolved this service interruption and impacted services have returned to normal operations.
Resolved
Service Maps functionality has been restored.
Resolved
This incident has been resolved, and prior uploads are now visible when creating and editing videos. Our sincere apologies for the delay.
Resolved
This incident has been resolved.
Resolved
Google has reported the issue has been resolved and we've verified cluster operations are functional.
Resolved