HotjarEngage is unavailable
This incident is now resolved. Thank you for bearing with us!
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident is now resolved. Thank you for bearing with us!
Resolved
The fix has been confirmed and the issue is now resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Incident: A gradual rollout changing Asana’s network routing resulted in certain internal requests for our non-North American clusters being misrouted. This caused sporadic errors loading information for users whose Asana domains are hosted in non-US clusters. Impact: Users with SAML SSO configured saw intermittent errors when attempting to login to Asana between November 18, 11:02 UTC and November 19, 01:27 UTC. Moving forward: We are improving our internal monitoring to promptly identify issues with future networking changes and working to make our login flow more resilient to data loading errors.Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Resolved
Between 13:19 and 13:36 UTC, we experienced performance degradation in crowdin.com and Crowdin Enterprise. The issue has been resolved, and the services are now operating normally.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We have fixed the problem and have returned to normal operation.
Resolved
This incident has been resolved.
Resolved
The Engineering team has isolated and resolved the elevated error rate for all affected clusters.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident is now resolved and audit logs are being successfully uploaded. Thank you for your patience.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident is now resolved. Thanks for bearing with us!
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
**Product:** QStash **Impact:** Degraded performance, delayed processing of events, and duplicate event deliveries for some customers ## Incident Summary QStash experienced an incident marked by a sudden and extreme load on our servers. This caused a degradation in performance, with extremely high latency for event processing for all users. We also noticed some of the events being delivered multiple times to some of the users. To mitigate the high load, we have increased the capacity as our initial response while investigation proceeds. Eventually, fixes for the issues are confirmed with an issue reproducer and deployed to production. ## Root Cause Analysis In a certain type of usage, failure handling of [failureFunction](https://upstash.com/docs/workflow/basics/serve#failurefunction) can cause recursive calls which causes a leak in the queue of the tasks, causing a severe load on the QStash servers. This also triggered an edge case which caused some of the events to be delivered multiple times. ## Resolution Two hotfixes to the QStash processes are deployed 1. Prevent recursive calls within the failure function. 2. Eliminate duplicate deliveries while keeping "at least once delivery" guarantee. These are verified to successfully resolve the root cause, normalizing server load and restoring standard event processing operations. ## Impact on Customers High latency of event processing is observed for all users. Some users received duplicate event deliveries. No events were lost, and all were delivered as part of our "at least once delivery" guarantee. Customers do not need to take any corrective action, as workflows have returned to normal and preventive fixes are deployed.
Resolved
On Tuesday Nov 12, at approximately 1am UTC, we received alerts of elevated errors due to an underlying host hardware failure. The workloads on this host were migrated to a healthy host and the errors resolved once the migration was complete.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Retroactive update - Dropbox.com was down for some users. We quickly identified and fixed the problem, so we're back up and running!
Resolved
This incident has been resolved.
Resolved
Between 13:56 UTC and 16:16 UTC on November 8th, some customers in the US region may have experienced a persistent error after visiting the ‘Destinations’ tab within the Alerts UI. Alert notifications were not impacted This issue has been resolved and impacted services have returned to normal operations.
Resolved
This incident has been resolved.
Resolved
We have resolved this incident. We apologize for the interruption to our customers’ operations.
Resolved
This incident has been resolved.
Resolved
We are glad to inform you that the fix has been implemented and the incident is resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We have fixed the support@imgix.com email address and are processing our backlog of support requests. If you sent an email and have not received a reply until today, please reach back out to us through our contact form: https://dashboard.imgix.com/contact
Resolved
This incident has been resolved.
Resolved
Our engineers have been closely monitoring the platform and are declaring this incident resolved. Thank you for your patience.
Resolved