Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Sep 10, 2026

  1. PostHogSome warehouse sources are failing to sync

    This incident has been resolved. Any warehouse sources that failed to sync will catch up on their next scheduled run. A manual sync can be triggered to refresh data for affected sources sooner.

    Resolved

  2. HarnessPipelines are stuck in Prod1

    # Summary On September 3, 2026, between approximately 2:54 AM and 3:44 AM PDT, customers running pipelines on Prod1 experienced delays in pipeline execution graph rendering and execution status updates. Pipeline execution itself was not affected — pipelines continued to run and complete — but the visual graph and status information in the Harness UI lagged behind actual execution progress. Harness engineering identified the cause, added processing capacity, and restored normal graph and status updates. Extended monitoring confirmed full recovery the following day. # Root Cause A planned database failover activity in the Prod1 region temporarily increased network latency between the pipeline execution service and its database while traffic was briefly served cross-region. This reduced the rate at which the service could process pipeline orchestration events, and a processing backlog began to build. Pipeline execution graphs and status indicators in the UI depend on these orchestration events being processed in near real time. As the backlog grew, graph rendering and status updates fell increasingly behind actual execution progress, and some graphs had to be rebuilt rather than served from cache. Pipeline execution itself does not depend on this same processing path, so pipelines continued to run and complete throughout the incident. # Impact * Affected users: Customers with pipeline executions running on Prod1 during the incident window. * Symptom: Delayed pipeline execution graph rendering and delayed execution status updates in the Harness UI. * Not reported as impacted: Pipeline execution itself — pipelines continued to run and complete. * Data: No data loss, corruption, or exposure was identified. * Suggested workaround: None required from customers. The issue was resolved entirely by Harness engineering. * Duration of impact: Approximately 45–50 minutes of degraded graph and status visibility, from 2:54 AM to 3:44 AM PDT on September 3, 2026. Harness continued monitoring after the backlog cleared and confirmed full recovery. # Remediation ## Immediate Harness engineering added processing capacity to the affected service and restarted it, which cleared the event-processing backlog and restored normal pipeline execution graph and status updates. # Action Items | **Action Item** | **Objective** | | --- | --- | | Queue based alarms | We are raising priority for queue based alarms so we can take faster remediation actions \(increase capacity\) |

    Resolved

  3. NotionUnexpected workspace verification banner displayed to some users

    This issue has been resolved. The unexpected workspace verification banner is no longer being displayed, and this issue did not indicate that any workspaces were compromised. If the banner is still visible, please reload the page or restart the Notion app. We apologize for the confusion and concern this caused.

    Resolved

Sep 9, 2026

  1. SnowflakeINC20000211

    Current status: We've monitored the environment and confirmed that service has been restored. If you experience additional issues or have questions, please open a support case via Snowflake Community or the Support page in Snowsight. Customer experience: Customers hosted in the specified regions may have been unable to access or use multiple core Snowflake services and features. Affected users may have been unable to sign in, execute queries, or manage data. Workaround: There were no available workarounds for this issue. Incident start time: 21:26 UTC September 09, 2026 Incident end time: 21:55 UTC September 09, 2026 Preliminary root cause: A subset of our metadata database infrastructure became unavailable, resulting in failures across multiple Snowflake services. A root cause analysis (RCA) will be conducted and a summary will be made available within 5 business days.

    Resolved

  2. HarnessSlowness in Prod1 and Prod2 environment

    ### Summary On September 8, 2026, customers in Prod 1 and Prod 2 experienced elevated platform latency and pipeline failures. The issue was caused by a regression in a newly released capability that triggered cascading failures under high load. Because the capability was behind a feature flag, it was quickly disabled, and service was restored after a brief monitoring period.  ### Customer Impact * Customers encountered slowness and failures during pipeline execution and UI operations. Some API calls returned errors or timed out. * No data loss or corruption occurred. ### Root Cause The new capability introduced a regression that created contention on a shared backend resource used by multiple Harness components. This saturated the shared platform infrastructure and caused the cascading failures. ### Mitigation * Disabled the capability across all environments * Temporarily increased platform capacity to restore stability ### Next Steps To prevent recurrence, Harness will: 1. **Permanently fix the capability** by profiling and eliminating the sub-optimal code path and query 2. **Improve detection** by enhancing alerting for resource-intensive queries on high-frequency platform paths

    Resolved

  3. SumUpTouchscreen not responding on POS devices

    The issue has been fixed and we’re confident it’s resolved SumUp services have returned to normal. You can stay up to date on the status of our services anytime here on our status page: https://status.sumup.com.

    Resolved

  4. Liquid WebStyleSmuggler Vulnerability – Magento & Adobe Commerce

    Our engineering teams continue to make progress on containment, investigation, and remediation across affected environments. Adobe has released a patch addressing the underlying vulnerability. We have contacted customers running affected versions of Magento and recommend applying the appropriate security patch as soon as possible. Customers should also review their environments and ensure their sites are clean and have not been impacted by the vulnerability. If you have any questions or concerns. please contact Liquid Web Support so we can guide you further.

    Resolved

  5. Box[Medium] Issue with Microsoft Office Integration and Box Edit

    We recently addressed issues affecting Box Edit, Microsoft Office Online, iWork, and other integrations in Box. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future. ‌ Between 04:17 PM PDT and 6:30 PM PDT on August 19, 2026 some users may have experienced difficulties while working in Box. During this time, impacted users were unable to open files in Box Edit, Microsoft Office Online, iWork, and other integrations. Additionally, some users saw errors, timeouts, or slow responses while working in other areas of Box. ‌ The issue occurred after a deployment of a backend service that provides integration and platform functionality. The deployment introduced a regression that caused the service’s performance to degrade under peak load, which led to client timeouts and errors. We restored functionality by reverting the deployment to a previous version. We are taking steps to reduce recurrence by improving our pre-release validation and deployment safeguards, including validating the performance of the service under load, improving observability, and making defensive changes to the affected service. ‌ # Analysis ‌ This incident highlighted opportunities for improving our pre-release validation to test how a change behaves under peak traffic. In this situation, a change in a backend service which handles integration and platform functionality modified its scalability characteristics under load. The service operated normally during non-peak traffic but was not able to keep up with peak traffic demand. Under peak load, the service’s thread pool became saturated, causing requests to queue while waiting for an available worker and increasing latency. # Corrective Actions Box has initiated the following corrective actions: * Strengthening pre-release validation to include tests that better simulate peak traffic and scaling behavior for the affected service. * Improving observability so that scaling issues are more readily detected. * Implementing defensive changes in the affected service to reduce impact if a similar performance degradation were to occur. ‌ We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. ‌ Sincerely, ‌ The Box Team

    Resolved

  6. ZapierZap History not displaying Zap runs

    We have implemented a fix regarding Zap history runs not showing, so they should be available in your Zap history page now. If you have questions or need help, please contact our Support team at: zapier.com/get-help "zapier.com/get-help": http://zapier.com/get-help.

    Resolved

  7. GitHubDelays in commit processing

    On September 1, 2026, between approximately 14:01 and 16:01 UTC, updates in response to pushes were delayed, temporarily showing stale diffs. The median time to refresh a diff after a push rose from the normal level of about 3 seconds to over 2 minutes at the peak, and more than 140,000 customer accounts had at least one delayed refresh during the most affected 75 minutes. Pushing commits and opening pull requests continued to work normally. The incident was caused by a sharp, concentrated surge in push volume that saturated worker pools and job queueing infrastructure. Autoscaling did not increase capacity as intended, so the backlog did not clear on its own. <br /><br />The incident was mitigated by manually scaling the affected worker pools and increasing push-processing capacity. This allowed the system to process the backlog, after which refresh times returned to normal. To reduce the likelihood and impact of similar incidents, we are adding quotas and throttling earlier in the push path so a single concentrated source of load cannot saturate shared capacity, improving worker-pool autoscaling so capacity is added automatically, and improving monitors for background job processing so on-call is paged before customers experience delayed pull request updates.

    Resolved

  8. HarnessProd2 was intermittently unavailable

    ## **Summary** Between 12:34am PST and 12:38am PST on 5th September, the Delegate service manager experienced some elevated exceptions when attempting to write to the database. Consequently, delegate connections were dropped, causing them to disconnect. Delegate automatically re-attempts registration back to the `delegate service manager`  and majority of the delegates got connected back after the incident. For Docker and ECS delegates the automatic restart is not enabled unless these delegates have health monitoring enabled. For these delegates a manual restart is needed and was recommended.  Post restart the delegate would re-connect and the issue was resolved. ## **Root cause** On Prod2 cluster we identified a performance bottleneck in the delegate service that, under certain conditions, can increase database write latency and delay heartbeat processing which leads to delegates being disconnected. ## **Impact** All K8s delegates and \`Docker/ECS\` delegates got connected back immediately within 4 mins and started to function normally. The impact can be scoped to those specific types of delegates that didn’t have health monitoring enabled.  ## **Remediation** * Immediate: We have added additional monitoring and increased resources for handling the influx of traffic.  * Permanent: We have identified a hotspot in the code that can cause high latency when writing to a database which we are actively working on resolving.  ## **Action Items** To prevent such issues from happening again, Harness will work on the following: 1. Increased targeted monitoring and alerting to initiate timely mitigation and prevent this from happening again. 2. Fix the identified delegate service managers database client reconnect failures 3. Fix the hotpots that can cause query latency.

    Resolved

Older incidents