Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Apr 23, 2025

  1. AirtableIssue with Box and Hootsuite syncs

    We're pleased to report that the issue affecting the Box and Hootsuite syncs to Airtable has been resolved. Our recent hotfix has been successfully deployed overnight, and both syncs should now be functioning as expected. Thank you for your patience and understanding as we addressed this matter.

    Resolved

Apr 22, 2025

Apr 21, 2025

  1. LiveblocksSlower or unavailable rooms

    Some rooms were slow or unable to connect to. We have found the root cause for this incident to be caused by a downstream provider issues and are working with them to get the issues resolved.

    Resolved

  2. ZuoraPerformance Degradation in OneID for subset of users

    We observed a performance degradation in OneID that impacted a subset of users attempting to log in to the tenant. These users were redirected back to the OneID page instead of being taken to the Zuora page. The subset of users are the users who had performed the update user profile operation in one-id The impact window was: April 16, 2025, 12:00 AM to April 21, 2025, 11:30 PM Pacific. Our team has addressed the issue for the impacted users, and we are actively monitoring the results.

    Resolved

Apr 19, 2025

Apr 18, 2025

  1. Voyage AIDashboard is down

    A missing DNS config change caused the dashboard frontend couldn't connect with backend. Unable to view the web page and operate on it.

    Resolved

  2. JamfJamf Pro Outage

    **Root Cause Analysis of Jamf Pro Outage** During scheduled maintenance on a low-level component of the Jamf Pro cloud infrastructure, an update failed, resulting in an outage that affected Jamf Pro customers. The component in question is a critical part of our auto-scaling infrastructure, which enables Jamf to support millions of devices. The failure of this upgrade led to a widespread outage. * The update was not backwards-compatible, making it impossible to roll back to the previous version. * We were able to quickly restore services for some configurations, including Premium Cloud customers. * Larger clusters could not be recovered through the standard workflow and required manual intervention. We had to downsize these clusters to enable recovery. * For customers with custom domains, an additional step was needed before service could be fully restored.

    Resolved

  3. UpstashQStash: Degraded performance

    An internal cleanup task for QStash events, coinciding with disc layer compaction task has caused performance degradation to some users. ‌ Mitigation: Team has mitigated the event by pausing some of these tasks and monitored the status for a while. ‌ Fixes: Improvements are being applied to these tasks to use resources more gracefully. Disk resources are increased to be able to handle a bigger burst of load.

    Resolved

Apr 17, 2025

Apr 16, 2025

  1. AsanaSAML Login Failures

    **Incident:** From April 4 to April 8, 2025, a subset of users were blocked from logging in with SAML, despite no changes to their identity provider setup. We believe the root cause to be a combination of recent code changes upgrading our infrastructure. These changes inadvertently affected how SAML responses were processed. **Impact:** Approximately 0.5% of users who authenticate through SAML were unable to log in to their Asana accounts during this period. No customer data was lost or compromised, and users who do not use SAML were unaffected. Our Customer Support team worked with affected domains to restore access while we resolved the issue. **Moving forward:** Our existing monitors did not trigger for this incident, which led to an extended period of downtime. We are improving existing monitoring and increasing monitor coverage to allow us to detect and respond to authentication issues more quickly in the future. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

    Resolved

  2. New RelicUI Irregularities and Alert Degradation in US Region

    Between 12:52 UTC and 15:28 UTC on April 16th, 2025, some customers in the US may have experienced delayed data in queries, charts, and dashboards as well as delayed, missing, or false positive alert notifications for related data. We have resolved this interruption and impacted services have returned to normal operations.

    Resolved

  3. BrazeIssue impacting Canvases with Exit Criteria

    Our engineering team has completed all necessary restorations for the Canvases affected by the Exit Criteria issue. As of 7:30pm ET yesterday, we had successfully restored more than 99% of impacted Canvases to their correct state. We are now working directly with the remaining affected customers to address their specific situations and will complete all customer communications by the end of the day. This incident is now resolved. We appreciate your patience throughout this process and apologize for any disruption this may have caused.

    Resolved

Apr 15, 2025

  1. MixpanelDegraded Ingestion API Performance (/import)

    On April 15th, from 12:38 AM UTC to 12:45 PM UTC, our US /import API sent incorrect API responses for up to 10% of traffic. During this period, some requests to the US /import API \(and only this endpoint\) may have received a 200 OK response, even though the events were not successfully ingested. This disruption only affected projects hosted in our US data center. ### Potential Action Required Data sent to the US /import API during this period of time will require resubmission to ensure complete data. * Please ensure those events maintain the same **$insert\_id, time, distinct\_id and event name** so that Mixpanel properly deduplicates any existing data. If you exclusively use Warehouse Connectors and/or the /track API to send data to Mixpanel, no action is required. We are committed to delivering accurate, timely analysis to support your critical decision-making, and we prioritize fast, high-quality customer support. If you have any questions or concerns, please feel free to reach out.

    Resolved

Apr 14, 2025

Apr 12, 2025

Apr 11, 2025

Apr 10, 2025

Apr 8, 2025

  1. AirtableRun script automation actions not executing

    The issue affecting the "Run Script" action has been resolved. Our system is processing the backlog, and all pending automations will be completed. If you experienced any script failures during the outage, you can manually retrigger those automations. Thank you for your patience and understanding as we work to ensure everything returns to normal. We appreciate your continued support.

    Resolved

  2. BitlyIssues with uploaded Images

    From `16:41Z` - `23:56Z` operations on and access to user uploaded images failed. This affected user ability to add new images to QR codes and Pages, and affected display of user uploaded images on Page buttons and backgrounds. The root cause was human error by our service provider during the application of a policy change. A restriction intended for one object incorrectly applied at the bucket level. We understand the significant impact this has had and we are taking steps with our service provider to prevent this from happening again.

    Resolved

Older incidents