CodecovTest Analytics Outage
This incident is resolved and Test Analytics is fully operational
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident is resolved and Test Analytics is fully operational
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Between 04/30 23:20 UTC and 05/01 01:35 UTC, users experienced partial degradation with Llama-3.3-70B. We have deployed a fix, and the issue is now resolved.
Resolved
Issues affecting the Aerial Omniverse Digital Twin service have been resolved and the service is operating normally.
Resolved
This incident has been resolved.
Resolved
On Tuesday, April 29, Front experienced an application outage lasting roughly 4 hours for a subset of US-based customers—roughly 3% of users. This was the result of a human error during a planned database maintenance operation that resulted in one database shard becoming unrecoverable. Front was able to restore this database shard from an up-to-date backup and then catch up any missed messages during that period, which resulted in no data lost as a result of this issue. Front customers are divided into database shards so that an issue with one shard does not affect all others. Earlier this week Front performed an upgrade of all database shards as a routine procedure, and while the upgrade was successful with no customer impact, there was an error in the cleanup step that caused one shard to be irrevocably invalidated. This occurred at 4:16 pm PDT \(23:16 UTC\). Front maintains up-to-date database replicas for this purpose, from which we were able to initiate a restoration. This involves copying data, and therefore took nearly four hours to complete. Once the database was restored at 8:07 pm PDT \(03:07 UTC Wednesday\), the application became available again, and Front was able to initiate the next task of applying all changes since the incident began, like syncing in emails from Gmail and O365. Front apologizes to the affected customers for the inconvenience caused by this extended outage. We strive to be always available to customers and have made great investments over the last several years to minimize the possibility and the impact of this kind of error. We can always do more, and are taking the following steps based on lessons from this incident. We have modified our database upgrade procedures and the permissions granted to the operators to significantly reduce the likelihood of this same error. At the same time, we are accelerating our upgrade frequency, which means we’ll have more practice and more automation with this procedure, further reducing risk. More upgrades also means faster adoption of performance and security improvements. Front is also investigating opportunities to improve the recovery time from the backup. Most of the time during this outage was waiting for data to be replicated from one storage region to another, and there may be ways to cut that down. Once again, Front would like to apologize for any disruption caused by this incident and emphasize our commitment to high availability and transparency.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We have resolved the issue causing interface-only users to see “Not connected to an automation” when clicking Interface buttons that trigger automations. Interface-only users should be able to trigger automations from interfaces as expected. We apologize for the impact and thank you for your patience.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We've balanced the load for our Community plan users and service has returned to normal.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Increased webhook response time from April 27th 2025, 15:35 EDT to April 28th 2025, 16:00 EDT
Resolved
There was an issue with our upstream provider which is now resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We've paused upgrades for the remaining customer databases, and will complete the remaining upgrades after looking into why we saw more errors than expected.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Issue is not resolved, opening a partner incident instead: https://front-partners.statuspage.io/incidents/sdln9wvbqdb7 Please contact Microsoft for assistance with an affected mailbox.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Issues affecting the Base Command service have been resolved and the service is operating normally.
Resolved
Issues affecting the Private Registry service have been resolved and the service is operating normally.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident is resolved. We're going to do some further investigation to understand what led to the lock contention.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Twilio SendGrid has confirmed that it understands the root cause. They have mitigated that and are confident that this issue will not recur. Their engineers are working toward a long-term fix. Please refer to SendGrid's Statuspage for additional details: https://status.sendgrid.com/incidents/6fj45kvpw8n3
Resolved
We're all set! PandaDoc is working properly and the incident has been fully resolved. If you continue to experience issues with this, please contact our Support team (https://support.pandadoc.com/en/articles/9715061-how-to-contact-pandadoc-support).
Resolved