SparkPostOutbound message delivery is delayed/slow for some SparkPost customers (EU hosted only)
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
### Incident Summary During a maintenance update to the regional Upstash Redis databases in AWS eu-west-1, several databases hosted in that region has unnecessarily triggered a full synchronisation between their primary and backup replicas. ### Root Cause A full synchronisation is the invalidation of the whole data in the target replica and starting a fresh re-population from the source replica. Under normal circumstances, full synchronisation is required only in cases where the data integrity is lost in one of the replicas, which was not the case here. ### Impact This incident impacted the performance of regional databases on AWS eu-west-1 only. Full synchronisation has caused a very high CPU load and caused a performance degradation on some of the databases that has a replica in this region. Moreover, our system throttles the databases that are going through this operation to allocate more CPU to the synchronisation to finish it sooner. No data or consistency has been lost. ### Resolution As a quick remediation, we have unthrottled affected databases on 15:06UTC and enabled more throughput, however high latency has still been observed until the full synchronisation is completed on 21:23UTC. A fix has been prepared to avoid this unnecessary full synchronisation on regional databases, and will be deployed shortly. This issue is not present on Upstash Global databases, which is our new generation infrastructure and is now our default offering. We will reach out to our regional users on how to migrate to Upstash Global going forward.
Resolved
This incident has been resolved.
Resolved
The issue has been fully resolved. All affected components are operating normally again.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The incident has been resolved.
Resolved
The dashboard has remained up ever since. We're continuing to investigate to find a root cause.
Resolved
We identified a backend API and resulting database query which triggered a high database load and affected production services. We are publishing a permanent fix on Monday, Feb 24.
Resolved
After monitoring, performance has improved significantly
Resolved
The issue has been resolved.
Resolved
The issue is resolved
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
After monitoring, this incident is now resolved. If you continue to see issues, please ensure you are not using a stale / old tab and are loading a fresh version of Bitbucket in your browser.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We experienced a very short period of API downtime for the incoming requests to QStash due to urgent maintenance to ensure the stability and performance of our services. Our team acted quickly to address the issue, and everything is now fully operational.
Resolved
Our team has resolved the issue and all affected services are now working as expected. If you continue to experience any further issues, please contact our Support team via Live Chat or email support@upcloud.com Pleae accept our sincere apologise again for any inconvenience caused.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We’ve deployed the fix we mentioned earlier, and we’re closing this incident. Thanks so much for bearing with us while we investigated and resolved this! We’re sorry for the trouble as well. If you don’t think the incident is resolved, or you’re still seeing problems on your end please raise a ticket through our Help Center (https://help.hotjar.com/hc/en-us/requests/new). Thanks!
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident is now resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Error rates have been back to normal since 0255 (UTC).
Resolved
This incident has been resolved.
Resolved
The issue has been fully resolved. Metrics are available in our Console again.
Resolved
The issue has been fully resolved. All affected components are operating normally again.
Resolved
The issue has been fully resolved. All affected components are operating normally again.
Resolved
Following a period of monitoring, we have confirmed that the issue affecting the MacOS and Windows Desktop Apps is resolved. Please update to the latest version for the latest fix. For any further issues please contact our Support team. Thank you for your patience.
Resolved
The issue has been fully resolved. All affected components are operating normally again. We have identified several improvements on our end that will make sure that similar issue will not reoccur and will implement them in short order.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved