Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Apr 9, 2024

Apr 8, 2024

Apr 5, 2024

Apr 4, 2024

  1. PusherService Disruption Notice: Pusher US2 Cluster

    On Saturday March 30th from 04:15 UTC until 11:30 UTC Pusher channels experienced a partial outage and higher than usual latencies on our US2 cluster. The incident was triggered by a single, high volume customer sending requests that was causing an authentication error. These errors are collated and stored in our error logging system, which can be viewed on the Pusher Dashboard. Due to the exceptionally high number of errors, the logging system was overwhelmed, and caused all writing to the log system to backlog on our socket processes. This in turn resulted in the socket processes taking longer to process messages, and to start failing health checks. The failing health checks caused pods to restart, which resulted in connections being dropped. We identified this issue and took steps to mitigate, attempting to switch off non-critical components of the system to alleviate stress on the system. Eventually it was clear that these steps were having little impact and we deployed a hotfix to disable the error logging system. During the deployment of this hotfix the system hit a limit on the amount of registered targets on our cloud provider’s load balancer. This caused the rolling deployment to take much longer than expected, as it waited for old socket processes to be fully drained and terminated. After coordination with our cloud provider we were able to increase this limit, allowing the deployment to complete and the incident was resolved. There were 2 main windows of impact From 4:15 till 5:40 UTC and From 7:15 till 9:15 UTC From 9:15 till 11:30 service was operating normally and we saw a ramp up in connections.

    Resolved

Apr 3, 2024

  1. TrelloAdmin Portal Feature Access Issue

    Between 6:30 AM UTC to 9:50 AM UTC, we experienced failures in accessing some features from the Admin Portal. The issue has been resolved and the service is operating normally.

    Resolved

  2. OpsgenieAdmin Portal Feature Access Issue

    Between 6:30 AM UTC to 9:50 AM UTC, we experienced failures in accessing some features from the Admin Portal. The issue has been resolved and the service is operating normally.

    Resolved

Apr 2, 2024

  1. imgixIncrease in Origin fetch requests

    We have identified an issue that was preventing some newly fetched Origin images from being inserted into our Origin image cache, causing the same images to be re-fetched multiple times for new render requests. The impact time (UTC) was 3/21 20:30 - 3/29 0:50. The issue is resolved.

    Resolved

Apr 1, 2024

Mar 30, 2024

Mar 28, 2024

Mar 27, 2024

Mar 26, 2024

  1. TrelloSome paginated queries in Forge hosted storage kept repeating the last page

    We have identified and resolved a problem with Forge hosted storage, where some paginated queries kept repeating the last page. The incident was detected by our internal monitoring and was resolved quickly after detection by reverting the deployment. Activating changes recently made to the query cursors for paginated queries introduced a bug that impacted some apps. A small number of requests were impacted over a 16-minute window, while the incident lasted. Timeline: - 25/Mar/24 10:08 p.m. UTC - Impact started, when the changes were deployed to production - 25/Mar/24 10:09 p.m. UTC - Incident was detected - 25/Mar/24 10:24 p.m. UTC - Incident was resolved and impact ended The impact of this incident has been completely mitigated and our monitoring tools confirm that query operations are back to the pre-incident behaviour. We have also resolved the underlying bug and deployed the fix to production, completely eliminating the cause of this incident. We apologise for any inconvenience this may have caused to our customers and our developer community.

    Resolved

Mar 24, 2024

  1. TypeformTypeform Platform unavailable

    We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused. REF: INFRA-6404

    Resolved

Mar 22, 2024

  1. TypeformTypeform Results View Delay

    We are glad to inform you that the fix has been implemented and the incident with viewing form results has been resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

Mar 20, 2024

Mar 19, 2024

Mar 18, 2024

  1. TypeformTypeform Checkout & Create Page - Unavailable

    We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

Mar 17, 2024

Mar 13, 2024

Older incidents