Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Oct 2, 2026

  1. Box[Medium] Issue with File Requests

    From approximately 2:50 AM to 3:30 AM PDT on Oct 01, 2026, we observed an issue impacting File Requests. There is no current impact and no further updates will be provided here. If you continue to experience any issues, please contact Box Support at https://support.box.com.

    Resolved

  2. SteelSession creations are failing

    We have successfully rolled out the fix out across the affected fleet. Session connectivity and error rates have returned to normal.

    Resolved

  3. HarnessMonitoring - Pipelines Stuck - Prod2

    ## **Summary** On August 6, 2026 \(morning PDT\), some customers running pipelines in the Prod2 production environment observed pipeline executions that stopped making progress — stages that did not advance and produced no further output or status updates. The issue was reported by affected customers. Harness engineers identified the cause, mitigated the impact, and pipeline executions returned to normal operation. The issue was caused by a self-referential pipeline expression. A Git webhook triggered a pipeline that referenced the contents of the webhook payload, and the payload itself contained further copies of that same expression. Each round of expression resolution therefore produced more expressions to resolve, doubling the amount of work each time. This exhausted the resources of the service instance processing that execution, and other executions assigned to the same instance were unable to progress while it was in that state. ## **Impact** During the incident window \(approximately 6:11 AM to 11:23 AM PDT on August 6, 2026\): * Some customers' pipeline executions on Prod2 stalled mid-execution and made no further progress. * Affected executions produced no new step output or status updates, and had to be aborted and re-run after mitigation. * Behavior was limited to executions being processed by the affected service instance — pipelines handled by other instances continued to execute normally. There was **no data loss**. Pipeline definitions, execution history, and stored state were unaffected. The majority of pipelines on Prod2 continued to execute successfully throughout the incident; the primary impact was that some in-flight executions could not complete and needed to be re-run once the issue was mitigated. ## **Root Cause** Harness pipelines support expressions that are resolved at runtime — for example, an expression that inserts the contents of the Git webhook payload that triggered the pipeline. In this case, a Git commit message contained the literal text of the payload expression itself, twice, and the pipeline referenced that same payload expression. Because the commit message is part of the webhook payload, resolving the expression inserted the entire payload — including the two literal copies of the expression carried in the commit message. Those newly inserted copies were then treated as expressions to be resolved, and each pass inserted two more full copies of the payload. The size of the value being processed, and the work required to process it, therefore doubled on every pass and grew exponentially rather than converging. Harness has a safeguard intended to stop exactly this: expression resolution is bounded by a maximum nesting depth, beyond which resolution halts and the pipeline fails with an explicit error. A defect in that safeguard meant the limit was not applied in this specific self-referential case, so resolution continued unchecked. Expression resolution runs inline on the threads that start pipeline steps. As each pass consumed progressively more memory and CPU without ever completing, the service instance performing that work stopped making progress, and every execution assigned to that instance stalled — which is what customers reported. ## **Mitigation** Harness completed the following immediate mitigation steps: * Identified the pipeline and the expression pattern responsible for the runaway resolution. * Stopped the affected service instance so that it would take on no further work. The remaining healthy instances picked up and processed queued executions normally. * Confirmed that pipeline executions returned to normal and closed the incident. These actions restored normal pipeline execution behavior and resolved the customer-facing impact. ## **Action Items** To reduce the risk of recurrence and improve detection, the following actions are in various stages of being implemented: * Fix the defect in the expression depth and loop-detection safeguard so that self-referential expressions are caught and fail fast with a clear error instead of consuming resources without bound. * Prevent payload expressions from being resolved out of trigger payload content, removing the self-referential path entirely. * Tighten the maximum expression nesting depth and evaluate explicit loop detection in addition to the existing depth limit. * Enhance automated tests in pre-production environments that reproduce self-referential expression patterns and verify that the safeguard detects and stops them. * Add monitoring for this pattern in pipeline executions so that it is detected proactively.

    Resolved

Oct 1, 2026

  1. TemporalUI Disruptions in GCP us-central1

    Between 20:40 and 21:12 UTC, some users in GCP us-central1 experienced disruptions to the UI. We continued monitoring through the affected period and have seen no further reports since. This incident is now resolved.

    Resolved

  2. Etsy2.7.1 hotfix created

    This incident has been resolved. If you are still experiencing an issue, please let us know in our Etsy Forums (https://community.etsy.com/t5/Technical-Issues/bd-p/bugs).

    Resolved

  3. SlackFree Plan Customers Are Experiencing Message Failures

    A messaging integration briefly caused some free-plan workspaces to exceed their daily messaging limit, blocking them from sending messages. We took action to lift the sending block for all affected free-plan workspaces, and have confirmed messaging is now restored. We apologize for how this incident affected you and your business. We will undertake a full investigation of the incident, establishing the technical trigger, the underlying cause, and preventive action to avoid a repeat in the future.

    Resolved

  4. Liquid WebSecurity Advisory: Critical WordPress Vulnerability - "Click2Shell"

    This incident is now resolved. All customers are strongly encouraged to confirm the Wordpress version for all of the websites, including those in staging or development, have been updated to the latest minor version of Wordpress released on September 22nd in order to protect the website from this exploit. For additional information, please refer to the official WordPress security announcements: https://wordpress.org/news/2026/09/wordpress-7-1-2-release/ https://wordpress.org/documentation/wordpress-version/version-7-1-2/

    Resolved

  5. LinearEU sync failure

    We've published a post mortem with details on this incident including a timeline of events and prevention steps for the future: https://linear.app/docs/eu-sync-outage

    Resolved

  6. SparkPostMail delivery delays

    This issue has been resolved. Since our changes went in, we've seen no further delivery delays, including through peak sending periods. Email is being delivered normally. We apologize for the disruption and thank you for your patience.

    Resolved

  7. HarnessFile Store Expression Degraded : Returning Null

    ### Summary On September 28, 2026, some customers on Prod-2 experienced failures in CD pipeline steps that read File Store content using pipeline expressions \(e.g., `<+fileStore.getAsString(...)>`\). Affected expressions returned null instead of the expected file content, causing dependent steps to fail or behave incorrectly. The issue was traced to a platform update that tightened authorization checks on File Store APIs. The update was rolled back, and File Store expressions have been working normally since. ### Customer Impact * Only pipeline steps on Prod-2 that read File Store content via expressions were affected. Pipelines that do not use File Store expressions were not impacted. * Affected expressions resolved to null, causing steps that depend on configuration or settings files to fail or produce incorrect output. * No File Store content was modified. No files, secrets, credentials, or artifacts were exposed, and no data was lost. ### Root Cause As part of an ongoing security initiative, Harness deployed stricter authorization checks across several platform APIs, including the API that serves File Store content. When a pipeline evaluates a File Store expression, one internal Harness service calls another to retrieve the file. The new authorization checks failed, resulting in the requests being rejected and the expressions returning null. ### Mitigation Once the platform update was identified as the cause, it was rolled back across the production environment. File Store expressions have resolved correctly since the rollback. No customer action is required. ### Preventive Measures & Next Steps * **Audit affected APIs:** Review all APIs where the stricter authorization checks were applied to ensure internal service-to-service calls are correctly accounted for. * **Improve test coverage:** Enhance automated test suites to include File Store expression evaluation across pipeline execution paths

    Resolved

Older incidents