Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Jan 17, 2025

Jan 16, 2025

Jan 15, 2025

Jan 14, 2025

Jan 13, 2025

Jan 12, 2025

  1. CerebrasDegraded Service

    Between 6:15 AM PST and 6:35 AM PST users experienced service disruption with Llama 3.3 70b. We have deployed a fix and the issue is now resolved.

    Resolved

Jan 11, 2025

Jan 10, 2025

  1. TypeformTypeform Builder - Degraded builder performance

    We are glad to inform you that the fix has been implemented and the incident resolved. If for any reason you are still impacted by this incident please contact support at https://www.typeform.com/help/contact/ We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

  2. UdemySite is degraded

    We have resolved the stability issue on our site, and the performance is back to normal. We appreciate your patience.

    Resolved

Jan 9, 2025

  1. AppsFlyerDelays with Postbacks

    The in-apps and sessions postbacks are back to normal and the issue is now fully resolved. Thank you again for your patience.

    Resolved

  2. PulumiStack details not accessible

    A deployment went out this morning at 09:49 PST that prevented a subset of users from accessible stack pages. This change has been rolled back and impacted users should be able to access stack pages again.

    Resolved

Jan 8, 2025

  1. AirtableIncident - Jan 8, 2025

    We have resolved this incident, and Airtable’s services are functioning normally. We’re sorry for any inconvenience this has caused.

    Resolved

Jan 7, 2025

  1. AppsFlyerDashboard access

    Access to our dashboards is fully available, and we are actively monitoring the related services. If you experience any issues, please try logging in again. Thank you for your continued patience.

    Resolved

Jan 6, 2025

Jan 3, 2025

  1. WarpmacOS auto-update failures

    We've made changes to ensure Warp will not install a corrupted update. If affected when updating from an older Warp version, please reinstall.

    Resolved

Jan 2, 2025

Jan 1, 2025

Dec 31, 2024

  1. SvixSome rate-limited messages were significantly delayed

    We have experienced an issue that caused some messages on the platform to be significantly delayed before delivery. Only a small fraction of our customers and an even smaller fraction of messages are affected. There is NO ACTION REQUIRED on your end; the issue is already resolved. Before I go into further details I want to apologize to those that are affected by this. We know you rely on us for reliable webhook delivery, and we don't take this responsibility lightly. We take extensive measures in order to prevent issues from ever happening, though unfortunately this was caused by a rare race condition, which is why both our manual reviews and testing as well as automated testing didn't catch it. Additionally, since rate-limited (throttled) messages are delayed (by definition) they are excluded from our usual delivery latency metrics and alerting (as they should be delayed). So we weren't alerted either. What is rate-limiting (throttling)? While Svix can handle however many messages you send us, your customers' endpoints may not be able to. In order to avoid overloading your customer endpoints, they can define a maximum delivery rate. If they are sent more messages a second than they can handle, Svix will buffer the requests and send it to them at the rate they set. You can read more about it in the rate-limiting (throttling) docs. We recently made a few improvements to the rate-limiting mechanism, and the issue was related to that. Because the issue was related to rate-limiting it only affected endpoints that had rate-limiting enabled (either by you or your customers). In addition to requiring rate-limiting to be set and enabled, it was only triggered when rate limiting was actually hit (e.g. if you had a rate limit of 100 and 50 messages were sent, there was no rate-limiting so the bug couldn't trigger), and when there was backpressure on the rate-limit (so going above the limit was sustained). In addition to that, it also required AWS SQS to suffer from increased latency (happens on occasion). The race condition that was triggered there, made it so that some messages were not immediately scheduled for an additional trigger (after the throttling ends) which was what was causing the delayed invocation. The issue was first introduced when the new rate-limiting mechanism was rolled to production. Following a testing period in staging, we decided it was ready for production use. We then enabled it for 0.1% of the traffic on the 20th of November, ramping up all the way to 10% of the traffic on the 16th of December. We deployed a fix for the core issue on the 16th of December. The fix was for a benign symptom that was caused by the same core issue. At that stage, however, it was not clear messages rate-limiting could cause delays to such an extent so we didn't proactively search for affected messages. This is why alerting affected customers was delayed. We are very proud of our reliability track record, and we know we fell short of it in this scenario. As mentioned above, we have many measures in place to help us prevent issues like these from ever happening, though in this case they were insufficient due to the rare scenario required to trigger this race condition. All of this doesn't change the fact that your service was adversely affected by this, and I'm extremely sorry about that. Please let me know if you have any questions or if there's anything else I can help you with. – Tom

    Resolved

Dec 30, 2024

  1. TypeformTypeform Results - Delays in new responses

    We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

Older incidents