Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Feb 26, 2025

  1. UpstashRegional AWS eu-west-1 Cluster Performance Degradation Issue

    ### Incident Summary During a maintenance update to the regional Upstash Redis databases in AWS eu-west-1, several databases hosted in that region has unnecessarily triggered a full synchronisation between their primary and backup replicas. ### Root Cause A full synchronisation is the invalidation of the whole data in the target replica and starting a fresh re-population from the source replica. Under normal circumstances, full synchronisation is required only in cases where the data integrity is lost in one of the replicas, which was not the case here. ### Impact This incident impacted the performance of regional databases on AWS eu-west-1 only. Full synchronisation has caused a very high CPU load and caused a performance degradation on some of the databases that has a replica in this region. Moreover, our system throttles the databases that are going through this operation to allocate more CPU to the synchronisation to finish it sooner. No data or consistency has been lost. ### Resolution As a quick remediation, we have unthrottled affected databases on 15:06UTC and enabled more throughput, however high latency has still been observed until the full synchronisation is completed on 21:23UTC. A fix has been prepared to avoid this unnecessary full synchronisation on regional databases, and will be deployed shortly. This issue is not present on Upstash Global databases, which is our new generation infrastructure and is now our default offering. We will reach out to our regional users on how to migrate to Upstash Global going forward.

    Resolved

Feb 25, 2025

Feb 24, 2025

  1. KeeperKeeper vault login errors - resolved

    We identified a backend API and resulting database query which triggered a high database load and affected production services. We are publishing a permanent fix on Monday, Feb 24.

    Resolved

Feb 22, 2025

Feb 21, 2025

Feb 20, 2025

  1. UpstashWe experienced a very short period of API downtime for the incoming requests to QStash due to urgent maintenance to ensure the stability and performance of our services. Our team acted quickly to address the issue, and everything is now fully operational.

    We experienced a very short period of API downtime for the incoming requests to QStash due to urgent maintenance to ensure the stability and performance of our services. Our team acted quickly to address the issue, and everything is now fully operational.

    Resolved

  2. UpCloud[UK-LON1]: Storage backend issue

    Our team has resolved the issue and all affected services are now working as expected. If you continue to experience any further issues, please contact our Support team via Live Chat or email support@upcloud.com Pleae accept our sincere apologise again for any inconvenience caused.

    Resolved

Feb 19, 2025

  1. HotjarSurveys not recording impressions metric

    We’ve deployed the fix we mentioned earlier, and we’re closing this incident. Thanks so much for bearing with us while we investigated and resolved this! We’re sorry for the trouble as well. If you don’t think the incident is resolved, or you’re still seeing problems on your end please raise a ticket through our Help Center (https://help.hotjar.com/hc/en-us/requests/new). Thanks!

    Resolved

Feb 18, 2025

Feb 17, 2025

Feb 16, 2025

Feb 15, 2025

Feb 14, 2025

  1. TrelloThe Trello Desktop app is re-directing users to the home page

    Following a period of monitoring, we have confirmed that the issue affecting the MacOS and Windows Desktop Apps is resolved. Please update to the latest version for the latest fix. For any further issues please contact our Support team. Thank you for your patience.

    Resolved

Feb 13, 2025

  1. NeonNew branch and project creation failing in us-east-2

    The issue has been fully resolved. All affected components are operating normally again. We have identified several improvements on our end that will make sure that similar issue will not reoccur and will implement them in short order.

    Resolved

Feb 12, 2025

Older incidents