Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Jun 24, 2026

  1. AssemblyAIPre-recorded Universal-3-pro and Universal-3-5-pro degradation in the EU

    The issue affecting EU pre-recorded transcription for Universal-3-Pro and Universal-3-5-Pro has been fully resolved, the fix was implemented at 10:05 UTC. All systems are operating normally. We apologize for the disruption to your workflows. If you have any questions, please reach out to support@assemblyai.com.

    Resolved

  2. HarnessIDP Workflows and Backend failures

    # **Customer RCA: Executive Summary** On June 17, 2026, between 9:30 PM and 10:15 PM IST, customers in certain production environments experienced failures when accessing IDP \(Internal Developer Portal\) workflows. The issue was caused by a configuration mismatch in service authentication settings. ## **Impact** * **Affected Service:** IDP workflows * **Duration:** Approximately 45 minutes \(9:30 PM - 10:15 PM IST\) * **Customer Impact:** Users were unable to load IDP workflows during this period. * **Data Loss:** None * **Security Impact:** None ## **Root Cause** During a planned service deployment, a configuration mismatch occurred in the authentication settings used for internal service-to-service communication. Specifically, the authentication credentials configured in the newly deployed service did not match the credentials expected by existing services in prod environments. This mismatch caused authentication failures when services attempted to communicate with each other, resulting in workflow loading failures for customers. ## **Mitigation** The issue was resolved by reverting the configuration change to restore the previous working authentication settings. The engineering team was proactively monitoring logs immediately after deployment and identified error patterns several minutes before the first customer report. ## **Action Items/Next Steps** 1. **Enhanced Pre-Deployment Validation:** Automated validation checks will be introduced to verify that all required configurations, secrets, and service dependencies are correctly provisioned and consistent across services before any production deployment. 2. **Improved Deployment Procedures:** Updated deployment checklists with explicit verification of service authentication settings across services before rollout.

    Resolved

Jun 23, 2026

  1. HootsuiteConnecting/Reconnecting Facebook accounts in Hootsuite

    We are aware of a service disruption that recently impacted Meta. Between approximately 1:53 PM PST and 3:00 PM PST on June 23, some users may have experienced issues connecting or reconnecting Facebook accounts in Hootsuite. This was caused by service disruption on Meta's end. The issue has since been resolved and all functionality has been restored. No action is required from users at this time. If you still encounter an issue connecting/reconnecting Facebook accounts in Hootsuite, please contact our customer support team.

    Resolved

  2. HubSpotSome HubSpot tools may be unavailable for some users

    Between 03:48 PM (UTC -04:00) and 04:46 PM (UTC -04:00), many customers in all regions experienced issues with the Files tool. This was caused by a configuration issue. As of 04:46 PM (UTC -04:00), our Files tool is working properly and the incident has been fully resolved. HubSpot conducts a thorough review after each incident to understand the cause and prevent it from happening again. Learn more about HubSpot's commitment to reliability at www.HubSpot.com/reliability. The information on this page reflects our understanding of the incident and impact at the time of the update.

    Resolved

  3. HootsuiteHootsuite Analytics loading difficulties

    We have resolved the loading issues impacting Hootsuite Analytics and the Planner Calendar. If you continue to have issues successfully loading Hootsuite Analytics or the Planner Calendar please reach out to our customer support team.

    Resolved

  4. PostHogPostHog AI: chat panel failed to load correctly

    For some users, opening PostHog AI resulted in an error, and the chat panel failed to load correctly. A frontend change introduced a bug in how PostHog AI chat sessions initialize. Our team identified the cause and shipped a fix. A hard refresh should fix the error if you're still encountering it.

    Resolved

  5. ConfluenceDegraded performance with Confluence Cloud

    ### Summary On Jun 12, 2026, between 07:04 and 12:10 UTC and on Jun 16, 2026, between 12:56 and 14:35 UTC, a subset of Atlassian customers using Confluence Cloud experienced errors and degraded performance when attempting to view or edit pages. The event was triggered when a database cluster within our infrastructure stopped responding to connection close requests. This caused idle connections to accumulate rather than be released — resulting in increased memory pressure and cascading errors for users. The incident was detected by our automated monitoring within 18 minutes and mitigated by scaling database resources and redeploying, which recycled the idle connections. A permanent fix has since been developed and deployed. ### IMPACT The overall impact was between Jun 12, 2026 07:04 and 12:10 UTC and Jun 16, 2026 12:56 and 14:35 UTC on Confluence Cloud. The June 12 incident caused service disruption to a subset of customers in the EU region while the June 16 incident caused service disruption to a subset of customers in the US region. Impacted users experienced errors or degraded performance when attempting to view, edit, or access Confluence content and spaces. ‌ ### ROOT CAUSE ‌ 1. A database cluster within our infrastructure stopped responding to connection close requests. 2. Idle connections accumulated across clusters rather than being closed, creating a connection leak and increasing memory pressure on the database layer. 3. Hitting max connections on the cluster degraded query performance. 4. Users attempting to view or edit Confluence content experienced errors and slow load times as a result. ‌ ### REMEDIAL ACTIONS PLAN & NEXT STEPS ‌ We know that outages impact your productivity. While we have a number of testing and preventative processes in place, this specific design limitation in our connection cleanup process was not identified until it was triggered by an adverse infrastructure condition in production. ‌ We have already deployed the following fix: * Redesigned the database connection cleanup process to prevent unresponsive connections from impacting the closure of others. To minimize the impact of similar incidents in the future, we will implement additional preventative measures such as: * Monitoring and alerting for connection cleanup latency and failure rates, so that slowdowns in connection closure are detected early. * Enhanced safeguards on database connection creation to limit the blast radius of any future connection accumulation events. We apologize to customers whose services were impacted during this incident. We are taking immediate steps to improve the resilience of our platform_._ ‌ Thanks, Atlassian Customer Support

    Resolved

  6. ConfluenceConfluence Cloud experiencing degraded viewing experience

    ### Summary ### On Jun 12, 2026, between 07:04 and 12:10 UTC and on Jun 16, 2026, between 12:56 and 14:35 UTC, a subset of Atlassian customers using Confluence Cloud experienced errors and degraded performance when attempting to view or edit pages. The event was triggered when a database cluster within our infrastructure stopped responding to connection close requests. This caused idle connections to accumulate rather than be released — resulting in increased memory pressure and cascading errors for users. The incident was detected by our automated monitoring within 18 minutes and mitigated by scaling database resources and redeploying, which recycled the idle connections. A permanent fix has since been developed and deployed. ### IMPACT ### The overall impact was between Jun 12, 2026 07:04 and 12:10 UTC and Jun 16, 2026 12:56 and 14:35 UTC on Confluence Cloud. The June 12 incident caused service disruption to a subset of customers in the EU region while the June 16 incident caused service disruption to a subset of customers in the US region. Impacted users experienced errors or degraded performance when attempting to view, edit, or access Confluence content and spaces. ### ROOT CAUSE ### 1. A database cluster within our infrastructure stopped responding to connection close requests. 2. Idle connections accumulated across clusters rather than being closed, creating a connection leak and increasing memory pressure on the database layer. 3. Hitting max connections on the cluster degraded query performance. 4. Users attempting to view or edit Confluence content experienced errors and slow load times as a result. ### REMEDIAL ACTIONS PLAN & NEXT STEPS ### We know that outages impact your productivity. While we have a number of testing and preventative processes in place, this specific design limitation in our connection cleanup process was not identified until it was triggered by an adverse infrastructure condition in production. We have already deployed the following fix: - Redesigned the database connection cleanup process to prevent unresponsive connections from impacting the closure of others. To minimize the impact of similar incidents in the future, we will implement additional preventative measures such as: - Monitoring and alerting for connection cleanup latency and failure rates, so that slowdowns in connection closure are detected early. - Enhanced safeguards on database connection creation to limit the blast radius of any future connection accumulation events. We apologize to customers whose services were impacted during this incident. We are taking immediate steps to improve the resilience of our platform_._ Thanks, Atlassian Customer Support

    Resolved

  7. ConvexAvailability issues across all Convex deployments

    We had approximately 10 minutes of availability impact caused by a cascading failure after performing a configuration change to migrate a database cluster. We apologize for the impact and will be refining these processes to avoid this happening again.

    Resolved

  8. LiveKitElevated rate of egress ending early in India region

    **Summary** Between June 5 and June 16, 2026, an intermittent loss of connectivity between internal systems caused room state to be altered incorrectly in a single region in India. As a result, a small number of active sessions in that region ended earlier than expected, and affected egress recordings could stop mid-session. While the intermittent issue was undetected for some time, the impacted number of egresses was low and limited to the one affected region in India - other regions were unaffected. **Timeline \(UTC\)** * 2026-06-05 05:00 - Intermittent loss of connectivity between internal systems begins causing room state to be incorrectly altered in one India region, leading to some egresses being terminated prematurely. * 2026-06-16 06:45 - Mitigation was deployed and monitoring began. Eligible connections were moved to a different region and added redundancy was deployed to the affected internal tooling. * 2026-06-16 20:42 - No further instances were observed since the mitigation was applied. Incident was marked resolved. **Resolution** We mitigated the outage by moving eligible connections to a different region and by adding redundancy to the affected internal tooling so it could better tolerate intermittent connectivity loss. As a longer-term fix, we hardened the communication channels between the involved systems. We have not observed any further occurrences since the mitigation was applied, and we are continuing to evaluate additional improvements to our tolerance for this class of failure.

    Resolved

  9. LiveKitConnection failures and SIP transfer errors in US Central

    ## Summary LiveKit Cloud is a globally distributed system. Our data centers continuously probe one another, and our overlay network uses those probes to choose healthy paths between regions. When a brief network disruption occurs in or between data centers, the overlay is designed to mark affected paths as unhealthy, route traffic around them, and return them to "healthy" once the underlying network has cleared. On June 18, 2026 \(US East 1\) and again on June 22, 2026 \(US Central\), a short packet-loss event in the affected data center triggered a compound failure in this mechanism. Two separate defects, one in the overlay network and one in our link-monitoring service, interacted in a way that left the affected region effectively isolated from other regions long after the underlying network had cleared. For most customers and most products, the impact was bounded. Our global routing automatically moved traffic to other US regions within approximately one minute of onset. SIP transfers anchored in the affected region took longer to recover but cleared within approximately thirty minutes. A small number of LiveKit Agents running self-hosted workers experienced an extended outage in agent dispatch, up to about two hours in some cases. The majority of agent workloads continued to receive jobs normally after the routing flip. The workaround at the time was to restart the affected agent worker process so that it re-registered against a controller in a healthy region. This is a separate defect in our agent dispatch routing, and we are fixing it. This post-mortem describes what happened, why the second incident occurred so soon after the first, and the steps we are taking to prevent recurrence. ## Impact * **Realtime connections:** minimal impact. Clients automatically reconnected to alternative data centers. * **API:** approximately 1 minute of degraded availability while traffic rerouted to a nearby region. * **SIP transfers:** approximately 30 minutes for in-progress transfers anchored in the affected region. * **Agent dispatch:** approximately 1 minute for the majority of agents. A small number of agent workers saw extended dispatch outages of up to approximately 2 hours until the worker process was restarted. * **Egress**: A few number of in-progress egresses running in the affected regions were ended prematurely during the approximately 1-minute outage window, between 16:47 and 16:48 UTC on June 18, and between 12:24 and 12:25 UTC on June 22. These egresses were incorrectly marked as successful and no error was surfaced to indicate the recording had failed. ## Root Cause The trigger in both incidents was the same: a short packet-loss event in the affected data center. The expected behavior was that affected paths would be marked unhealthy briefly while the overlay rerouted traffic around them, and would return to "healthy" once the underlying network cleared. Two separate defects interacted in a way that kept routing state stuck long after the network had recovered: 1. **Inbound and outbound freshness collapsed onto a single indicator.** In the overlay network, the two directions of a connection \(inbound and outbound\) shared a single "freshness" indicator. When fresh data arrived in one direction, our code incorrectly assumed data in the other direction was also fresh. As a result, the affected region continued to treat its outbound connections to other regions as unreachable based on stale inbound data, even after outbound traffic was not impacted. 2. **Link-monitor key exchange could de-sync during a disruption.** Our link-monitoring service relies on a key exchange between regions to validate probe traffic. The packet-loss event caused this key exchange to fall temporarily out of sync between the affected region and its neighbors. With probes failing validation, the link monitors in neighboring regions marked the affected region as "to avoid," and held that state past the actual network recovery. Either defect on its own would not have produced an outage. Together, they produced a stuck state in both directions: neighboring regions believed they could not reach the affected region \(because probe validation kept failing\), and the affected region believed it could not reach the others \(because the freshness collapse kept its outbound state pinned to "unreachable"\). Traffic stayed routed away from the affected region until the state was manually cleared. Following Incident 1 on June 18, we developed and merged a fix for the overlay freshness defect on the same day. The fix shipped as a new version of the overlay software and was being validated in staging at the time of Incident 2. On June 22, the same compound failure occurred on US Central, which was still running the unpatched version of the overlay. At the time of writing, the fix is now in active rollout to production. ## Corrective Actions & Prevention **Immediate \(in active rollout\)** * **Roll the overlay freshness fix to production region-by-region.** As of this writing, the rollout is underway and we expect global production coverage in the next two days. We are giving this rollout priority given the severity. **Underway** * **Improve resilience of the link-monitor probing process to key-exchange de-sync.** This addresses the second of the two defects described above. Eliminating it is necessary to prevent the compound failure even after the overlay freshness fix is in place. _ETA: Thursday, June 25._ * **Migrate existing agent worker connections on health changes.** We correctly detected the data center as unhealthy and rerouted new traffic away quickly, but did not migrate existing agent worker connections. The change is to automatically migrate agent worker connections to healthy data centers as soon as the health indicator starts failing. _ETA: June 30._ * **Make egress resilient to region disruptions.** In-progress Egress should use the same mechanism available in our realtime SDK to reconnect to alternative regions. _ETA: June 30._ ## For customers running self-hosted agent workers Until our agent dispatch routing fix is deployed, the most reliable mitigation if you observe a dispatch outage is to restart your agent worker processes. This causes them to re-register against controllers in healthy regions. To catch the condition early and automate the recovery, consider adding a health check that: * Tracks the time since the worker last received a dispatch \(or last completed a session\). * Triggers a restart of the worker process if this exceeds an expected idle window for your workload. We recognize that this workaround places a burden on customers and is not a substitute for the underlying server-side fix. Eliminating it is a priority. We sincerely apologize for the disruption to customers whose traffic and agent workloads were affected. Thank you for your patience, and we welcome any additional feedback from customers who were impacted.

    Resolved

  10. LovablePublishing errors

    Affected projects should now be publishable. If not, please reach out to support for further assistance. Thank you for your patience.

    Resolved

  11. HootsuiteTalkwalker service disruption

    We can confirm that the issues related to Talkwalker services have been resolved. If you continue to experience loading issues or overall slowness with the platform, please reach out to our support team. Thanks for your patience while our team sorted this out.

    Resolved

Older incidents