Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Apr 8, 2025

  1. HotjarSurvey changes are not saving correctly

    We’ve validated the deployed fix has worked, and we’re closing this incident. Thanks so much for bearing with us while we investigated and resolved this! We’re sorry for the trouble as well. If you don’t think the incident is resolved, or you’re still seeing problems on your end please raise a ticket through our Help Center: https://help.hotjar.com/hc/en-us/requests/new Thanks!

    Resolved

  2. CerebrasPartial Service Disruption

    Between 00:28 UTC and 02:58 UTC users experienced partial degradation with Llama3.1-8B. We have deployed a fix and the issue is now resolved.

    Resolved

Apr 7, 2025

Apr 5, 2025

Apr 4, 2025

Apr 3, 2025

  1. BrazeIssue impacting US01 Services

    We apologize for mischaracterizing the impact of the outage in the original updates. We confirm that the incident is now resolved and that the impact was minor. During the incident window (12:45 to 14:00 ET) customers may have experienced the following: SDK Data Collection Errors - this will automatically retry and no action is needed by Braze customers Outbound Messaging latency

    Resolved

  2. UpstashQStash: Degraded performance in request processing and event logs

    ### Product: QStash ### Incident Summary Due to high load, the volume of QStash event logs reached to a point which caused latency in the underlying data store operations. Event log creation was slowed down and lead to performance degradation in QStash request processing. In order to resolve the performance degradation in QStash requests, event logging module was turned off temporarily. After deploying a hot fix and configuration changes, we eventually turned on event logging and system went back to stable state again. ### Root Cause At 07:15 UTC we received alerts on the performance degradation and started the investigation. We discovered long running queries for synching event logs from main QStash servers to QStash event server. In order to resolve performance degradation, we turned off event logging functionality as an immediate action. This action turned the performance back to normal levels for QStash requests but left event log processing disabled. We deployed a hotfix during the day to remove some redundant calls and alleviate the impact. Around 16:20 UTC, we observed another performance degradation on QStash requests due to a load increase, and disabled event log processing again. In the following hours, we deployed a configuration change to relax the job interval durations for event log tasks and turned on event logging again. This configuration change helped to resolve the performance issues without any further issues. ### Impact During the problematic timeframes, when the slow event log processing was observed, QStash requests experienced high latency and caused timeouts for customers. No events were lost. Duplicate event deliveries were observed due to a number of restarts during the incident. ### Resolution Improvements are applied to the event logging module to prevent the same issue from happening again. Also, we have planned to upgrade underlying disks to stronger models.

    Resolved

Apr 2, 2025

Apr 1, 2025

Mar 31, 2025

  1. imgixMissing Images in Asset Manager

    # What happened Between **March 17th, 5:48 AM UTC** and **March 18th, 6:56 PM UTC**, the Asset Manager experienced an indexing outage. During this period, newly uploaded assets and a very small percentage of pre-existing assets were not appearing in the Asset Manager UI. These assets were still successfully uploaded to the Origin and were accessible via the Rendering API. # How it happened An unannounced maintenance from our upstream provider caused a node failure in our Asset Manager infrastructure, resulting in the temporary loss of indexed asset data. # What went wrong Several things went wrong during this incident: * Our service provider failed to notify us of the maintenance window * Our Asset Manager infrastructure was not adequately provisioned to tolerate failures during the maintenance * A large amount of data was required to be restored, resulting in a severely prolonged restoration process # What we are doing to address this incident * We have migrated and optimized infrastructure configurations to better tolerate node failures * We created additional backups and introduced additional indexing layers to improve redundancy and resilience * We are evaluating alternative upstream providers to reduce dependency risks

    Resolved

Mar 28, 2025

  1. AirtableAutomation Run History temporarily unavailable

    The issue with our Automation Run History feature has been resolved. We apologize for any inconvenience this may have caused. If you encounter any further issues, please reach out to our support team. Thank you for your patience and understanding.

    Resolved

  2. LeverLatency in Hire.lever.co

    The issue with prolonged delays on Lever activity has been resolved. The incident occurred from 3/12/2025 until 3/18/2025. There should be no further impact at this time, but please reach out to us at Support if any additional assistance is needed: https://help.lever.co/hc/en-us/requests/new

    Resolved

Mar 27, 2025

  1. AsanaApp and API downtime

    **Incident**: A configuration change triggered a bug affecting how internal systems process configuration data, leading to application downtime. The issue impacted internal server uptime, resulting in approximately 28 minutes of degraded service on March 25, 2025, from 22:32 UTC to 23:00 UTC \(3:32 PM to 4:00 PM PDT\). ‌ **Impact**: During the incident, customers experienced difficulties accessing Asana on web and mobile apps, including login failures and errors while using the application. Internal services gradually recovered after the problematic change was reverted. ‌ **Moving forward**: We are enhancing our configuration validation processes to prevent similar issues. Additional monitoring and safeguards are being introduced to ensure rapid identification and recovery from similar incidents. ‌ Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

    Resolved

Older incidents