Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Feb 12, 2025

  1. MailChannelsOutbound Filtering Blocked Emails

    A significant portion of the outbound email was temporarily blocked between 14:10 UTC and 14:25 UTC. The filter has been fixed, and we are processing normally now. This was caused by the introduction of a bad rule, which was quickly detected and reverted.

    Resolved

  2. HoneybadgerDelay in stats and check-in processing

    We had an interruption in processing of check-in alerts and in-app stats from about 12:30 to about 13:30 UTC. This may have resulted in a failure or a delay in reporting check-ins that went missing during that period.

    Resolved

Feb 11, 2025

  1. CerebrasPartial Service Disruption

    Between 7:30 PM UTC and 8:00 PM UTC users experienced service disruption with Llama 3.3 70b and Llama 3.1 8b. We have deployed a fix and the issue is now resolved.

    Resolved

Feb 10, 2025

  1. KeeperLogin errors in US region

    At 9:21AM PST we received notifications of API errors coming from our backend systems. The issue resolved itself without any required action at 9:31AM PST. During that 10 minute period, there was a database lock in the primary writer instance causing new login sessions to fail. Recently, Keeper upgraded all production RDS Aurora databases to MySQL 8.x. This updated database engine is sensitive to one particular SQL query that was causing large table scans to occur. The code changes and QA process was already under way when this issue occurred. The backend code change which addresses this issue passed QA at 12:48PM PST and we released the fix to prevent this issue from occurring again \(Release Jira ticket REL-4633\). We’re keeping a close eye and will issue additional updates as necessary.

    Resolved

Feb 8, 2025

Feb 7, 2025

  1. AsanaAsana down

    Incident: At 2025-02-06 15:37 UTC, a configuration change caused some of our servers to crash, leading to complete downtime for 20 minutes. We rolled back the configuration, but this event triggered a cascade failure in network routing components; our systems recovered over the next 30 minutes, with full recovery by 2025-02-06 16:30. Impact: Users were unable to use Asana during this event in all regions, with downtime including App, API, and Mobile. No customer data was lost. Moving forward: We have added validation to prevent a repeat of this issue, and are updating our networking components to avoid this form of cascade failure. We will be moving to staged configuration rollouts to identify this class of issue without causing downtime for customer traffic. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

    Resolved

  2. AsanaAsana down in US

    Incident: Starting at 2025-02-05 21:05 UTC, a configuration change caused a large increase in server logs, overloading logging infrastructure and causing server restarts. This caused cascade failure as other servers were overloaded. The configuration change was reverted, and our application recovered after the servers restarted, with recovery by around 2025-02-05 21:30. Impact: Users saw errors loading our application and loading data within the application. No customer data was lost. Moving forward: We have removed the source of logs which resulted in overload, and are improving monitoring and resilience of the system which triggered failure. We will also be moving to staged configuration rollouts to help reduce or avoid customer impact for similar issues. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

    Resolved

Feb 6, 2025

  1. UpstashQStash Workflow Run Failure

    Latest QStash release caused some Workflow runs to fail due to a bug in Workflow URL detection mechanism. Affected workflows did not start at all and moved directly to DLQ. These workflow runs, which show "detected non-workflow destination" message in the response body, can be retried from DLQ. This was a partial failure, lasted from 10:10 to 11:00 UTC, not all users' workflows were affected.

    Resolved

  2. AblyPush notification delivery issues

    8th May 2024 08:30:00 AM We're experiencing issues delivering push notifications. The root cause has been identified and we are fixing this now. We are rolling out a fix which we expect to complete before 1355UTC today. 8th May 01:45 PM All push notifications that were attempted earlier today are now being delivered. 8th May 01:50 PM We've completed the deployment.

    Resolved

Feb 5, 2025

Feb 4, 2025

Feb 3, 2025

Feb 1, 2025

Jan 31, 2025

Jan 30, 2025

  1. CodecovError processing uploads using the global upload token

    On 28 January from 12:00 to 30 January 13:00 EST There was a disruption to global upload token validation that resulted in the Codecov system accepting uploads but internally denying the upload. This resulted in failed upload processing for repositories configured to use the global upload token. We now have increased monitoring for this scenario and have put measures in place to prevent it altogether. The underlying issue has been resolved and global upload token uploads should again be working as expected.

    Resolved

  2. GOV.UKPublishing Incident

    On the 1/30/2025, from 11:20am to about 12:30pm, there was degradation in performance on Whitehall, causing knock on issues for Publishers.

    Resolved

Jan 29, 2025

  1. UpstashDisk failure in some Regional Databases

    We observed a disk failure on some instances on 16:52 UTC and restarted affected components to bring them back online. Issue is resolved on 17:13 UTC. During the time of the issue databases reachability are affected, no data is lost.

    Resolved

Older incidents