Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Nov 25, 2024

Nov 23, 2024

Nov 22, 2024

  1. AsanaIntermittent issues with SAML login

    Incident: A gradual rollout changing Asana’s network routing resulted in certain internal requests for our non-North American clusters being misrouted. This caused sporadic errors loading information for users whose Asana domains are hosted in non-US clusters.  ‌ Impact: Users with SAML SSO configured saw intermittent errors when attempting to login to Asana between November 18, 11:02 UTC and November 19, 01:27 UTC. ‌ Moving forward: We are improving our internal monitoring to promptly identify issues with future networking changes and working to make our login flow more resilient to data loading errors.Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

    Resolved

Nov 21, 2024

Nov 19, 2024

Nov 18, 2024

Nov 16, 2024

Nov 15, 2024

Nov 13, 2024

  1. UpstashPerformance degradation on QStash

    **Product:** QStash **Impact:** Degraded performance, delayed processing of events, and duplicate event deliveries for some customers ## Incident Summary QStash experienced an incident marked by a sudden and extreme load on our servers. This caused a degradation in performance, with extremely high latency for event processing for all users. We also noticed some of the events being delivered multiple times to some of the users. To mitigate the high load, we have increased the capacity as our initial response while investigation proceeds. Eventually, fixes for the issues are confirmed with an issue reproducer and deployed to production. ## Root Cause Analysis In a certain type of usage, failure handling of [failureFunction](https://upstash.com/docs/workflow/basics/serve#failurefunction) can cause recursive calls which causes a leak in the queue of the tasks, causing a severe load on the QStash servers. This also triggered an edge case which caused some of the events to be delivered multiple times. ## Resolution Two hotfixes to the QStash processes are deployed 1. Prevent recursive calls within the failure function. 2. Eliminate duplicate deliveries while keeping "at least once delivery" guarantee. These are verified to successfully resolve the root cause, normalizing server load and restoring standard event processing operations. ## Impact on Customers High latency of event processing is observed for all users. Some users received duplicate event deliveries. No events were lost, and all were delivered as part of our "at least once delivery" guarantee. Customers do not need to take any corrective action, as workflows have returned to normal and preventive fixes are deployed.

    Resolved

  2. Cockroach LabsElevated errors on serverless platform

    On Tuesday Nov 12, at approximately 1am UTC, we received alerts of elevated errors due to an underlying host hardware failure. The workloads on this host were migrated to a healthy host and the errors resolved once the migration was complete.

    Resolved

Nov 12, 2024

  1. DropboxDropbox.com was down

    Retroactive update - Dropbox.com was down for some users. We quickly identified and fixed the problem, so we're back up and running!

    Resolved

  2. New RelicAlert Destinations UI Irregularities - US Region

    Between 13:56 UTC and 16:16 UTC on November 8th, some customers in the US region may have experienced a persistent error after visiting the ‘Destinations’ tab within the Alerts UI. Alert notifications were not impacted This issue has been resolved and impacted services have returned to normal operations.

    Resolved

Nov 11, 2024

Nov 9, 2024

Nov 8, 2024

  1. TypeformTypeform Renderer - Forms intermittently unavailable

    We are glad to inform you that the fix has been implemented and the incident is resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

Nov 6, 2024

  1. imgixIssue with receiving support requests

    We have fixed the support@imgix.com email address and are processing our backlog of support requests. If you sent an email and have not received a reply until today, please reach back out to us through our contact form: https://dashboard.imgix.com/contact

    Resolved

Older incidents