HeapHeap Connect Syncs: Possibility Of Stale Data
This incident has been resolved
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Description: We identified an upstream infrastructure provider issue affecting the eu-central-1 region. Accelerate experienced issues provisioning resources for new query engine instances. Affected users: Users provisioning new Accelerate query engines in the eu-central-1 region. Root cause: Upstream infrastructure provider Workarounds: None identified
Resolved
This incident has been resolved.
Resolved
Front has now completed the fix and cleanup associated with this incident. If you continue to see errors loading conversations, please make sure you've done a browser or app refresh, and if the issue persists please contact support. Summary: Conversations for which Front attempted to provide a Suggested Reply between 21:30 – 21:50 UTC may have been unreadable until the fix was completed at 22:49 UTC. During that initial time, suggested replies were generated with an additional field that caused an unrecoverable error in the frontend. No data or messages were lost or corrupted during the incident. Suggested replies for conversations that arrived during that time may need to be regenerated.
Resolved
We have resolved the issue.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We experienced an issue with our Streaming API between 09:00 GMT and 10:04 GMT. This resulted in reduced quality of service for streaming sessions and some sessions were closing unexpectedly. A fix was implemented and our Streaming API is now functioning normally.
Resolved
Our engineers have identified and resolved the issue, so load times should now be back to normal once again. Thank you for your patience while we worked on this one!
Resolved
Between 03:30 UTC to 03:50 UTC, we experienced delyas in running pipelines for Atlassian Bitbucket. The issue has been resolved and the service is operating normally.
Resolved
The issues with Search in Live Tail have been resolved, and Search performance is back to normal.
Resolved
This issue has been resolved. Conversations and folders should now be up to date and working as expected. Apologies for the trouble!
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This issue has been resolved. A fix was deployed to ensure interview scheduling now correctly uses the timezone set in the project’s settings. We apologize for the inconvenience.
Resolved
The issue has been identified & resolved, and everything should be back to normal once again. Thank you for your patience with this!
Resolved
An update has been released by Microsoft to resolve this issue for users of the Edge browser. To verify that you are running the required version of Edge, you can navigate to edge://components/ in the address bar, click the 'Check for update' button for the 'Trust Protections List', ensuring that you are on version 1.0.0.31 or higher to ensure you are running the version that contains the fixed release. All Edge browsers should otherwise receive this update within 10 hours without user intervention.
Resolved
Our upstream provider has resolved the issue on their end, and SSL provisioning is operating normally.
Resolved
All channels should now be recovered
Resolved
The issue has been resolved and the service is operating normally.
Resolved
This incident has been resolved.
Resolved
Due to a race condition in the throttling code, some message may have been throttled for a much longer time than they were supposed to. The issue has been fixed and everything should be back in order. No action needed.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
All systems are operating normally at this time. Our engineers continue to monitor our systems for any recurrence or new issues.
Resolved
The incident has been resolved.
Resolved
The 3rd party outage causing issues with transactional email sends has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The outage of our 3rd party provider that was blocking new user sign ups has been resolved.
Resolved
This incident has been resolved.
Resolved
On Thursday, September 11, 2025 1:00PM UTC, Trello users may have experienced performance degradation. The issues has now been resolved, and the service is operating normally for all affected customers.
Resolved
# **What happened** On September 4, between approximately 10:00 AM and 11:00 AM EDT, some Vimeo users experienced partially rendered pages across the site. Affected pages displayed the top menu bar but were missing portions of the expected page content. # **What caused it** During a site update, some required files weren’t uploaded correctly. # **How we responded** We received internal notification of the issue almost as it occurred and began investigating immediately. We quickly confirmed the outage and reverted to a prior stable version of Vimeo. This restored all affected pages within one hour. We then investigated the root cause and implemented a permanent fix by roughly 1:30 PM. # **What we’re doing to prevent a recurrence** We changed the order in which Vimeo source code is uploaded and checked so that lagging updates won’t cause similar outages in the future.
Resolved
# Performance Degradation **Date of Incident:** 2025-09-03 **Time of Incident \(UTC\):** 11:06 - 12:07 **Service\(s\) Affected:** All APIs **Impact Duration:** 61 minutes ## Summary For 61 minutes on the morning of September 3rd, 2025, all 1Password APIs in the US/Global environment had degraded performance or returned an error for approximately 20% of requests. 92% of the impact was mitigated within 13 minutes at 11:19 by automation scaling up infrastructure. By 12:06 a manual restart of the remaining infrastructure completed mitigation. A permanent fix was implemented and deployed to prevent the issue from reoccurring. ## Impact on Customers * **APIs:** High latency, or a 500 Internal Server Error. * **Number of Affected Customers:** 20% of all requests returned errors for 13 minutes, 1% thereafter. * **Geographic Regions Affected \(if applicable\):** 1Password USA/Global ## What Happened? * **Timeline of Events \(UTC\):** * 11:05: A customer started a stream of an unusually high volume of requests to an API with sub-optimal performance. * 11:06: Some servers started consuming abnormally high memory, causing slow response times and high error rates. * 11:19: Automation scaled up infrastructure to service additional load * 11:30: Increased errors trigger escalation, on-call engineer begins investigation * 11:51: Engineers declare an incident and alert response teams * 12:02: Response team begins restarting affected servers. * 12:07: All servers completed restarts, and error rates returned to normal levels * **Root Cause Analysis:** A poorly performing cache operation was triggered repeatedly in a short period of time across multiple servers, leading directly to greatly delayed responses. ## How Was It Resolved? * **Mitigation Steps:** Automatic instance scaling restored over 98% of operational capacity after 13 minutes. Full capacity was restored through manual intervention * **Resolution Steps:** We refactored the poorly performing query. * **Verification of Resolution:** We tested the affected API to confirm refactoring of query produced the desired performance improvement. We deployed the fix and monitored it for 24 hours to assert the issue was resolved. ## What We Are Doing to Prevent Future Incidents * We are auditing services for sub-optimal query performance. ## Next Steps and Communication * No action is required from our customers at this time. We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding. Sincerely, The 1Password Team
Resolved
The incident with model gpt-oss-120b has been resolved.
Resolved
This incident has been resolved.
Resolved