Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

May 2, 2025

May 1, 2025

Apr 30, 2025

  1. Front2025-04-29 Application unavailable for some customers

    On Tuesday, April 29, Front experienced an application outage lasting roughly 4 hours for a subset of US-based customers—roughly 3% of users. This was the result of a human error during a planned database maintenance operation that resulted in one database shard becoming unrecoverable. Front was able to restore this database shard from an up-to-date backup and then catch up any missed messages during that period, which resulted in no data lost as a result of this issue. Front customers are divided into database shards so that an issue with one shard does not affect all others. Earlier this week Front performed an upgrade of all database shards as a routine procedure, and while the upgrade was successful with no customer impact, there was an error in the cleanup step that caused one shard to be irrevocably invalidated. This occurred at 4:16 pm PDT \(23:16 UTC\). Front maintains up-to-date database replicas for this purpose, from which we were able to initiate a restoration. This involves copying data, and therefore took nearly four hours to complete. Once the database was restored at 8:07 pm PDT \(03:07 UTC Wednesday\), the application became available again, and Front was able to initiate the next task of applying all changes since the incident began, like syncing in emails from Gmail and O365. Front apologizes to the affected customers for the inconvenience caused by this extended outage. We strive to be always available to customers and have made great investments over the last several years to minimize the possibility and the impact of this kind of error. We can always do more, and are taking the following steps based on lessons from this incident. We have modified our database upgrade procedures and the permissions granted to the operators to significantly reduce the likelihood of this same error. At the same time, we are accelerating our upgrade frequency, which means we’ll have more practice and more automation with this procedure, further reducing risk. More upgrades also means faster adoption of performance and security improvements. Front is also investigating opportunities to improve the recovery time from the backup. Most of the time during this outage was waiting for data to be replicated from one storage region to another, and there may be ways to cut that down. Once again, Front would like to apologize for any disruption caused by this incident and emphasize our commitment to high availability and transparency.

    Resolved

  2. AirtableInterface Button Issue

    We have resolved the issue causing interface-only users to see “Not connected to an automation” when clicking Interface buttons that trigger automations. Interface-only users should be able to trigger automations from interfaces as expected. We apologize for the impact and thank you for your patience.

    Resolved

Apr 29, 2025

Apr 28, 2025

Apr 27, 2025

Apr 25, 2025

  1. AsanaTransient application errors

    We've paused upgrades for the remaining customer databases, and will complete the remaining upgrades after looking into why we saw more errors than expected.

    Resolved

Apr 24, 2025

  1. PyPIPyPI Elevated Error Rates

    This incident is resolved. We're going to do some further investigation to understand what led to the lock contention.

    Resolved

Apr 23, 2025

  1. BrazeEmail latency due to Sendgrid issue

    Twilio SendGrid has confirmed that it understands the root cause. They have mitigated that and are confident that this issue will not recur. Their engineers are working toward a long-term fix. Please refer to SendGrid's Statuspage for additional details: https://status.sendgrid.com/incidents/6fj45kvpw8n3

    Resolved

  2. PandaDocDocument search results are not loaded

    We're all set! PandaDoc is working properly and the incident has been fully resolved. If you continue to experience issues with this, please contact our Support team (https://support.pandadoc.com/en/articles/9715061-how-to-contact-pandadoc-support).

    Resolved

Older incidents