SnykSnyk Partial Test Degradation
The incident has been resolved after we have monitored the overall performance.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
The incident has been resolved after we have monitored the overall performance.
Resolved
This incident has been resolved.
Resolved
The issue has now been resolved. We apologize for any inconvenience caused.
Resolved
This incident has been resolved.
Resolved
This issue has been resolved.
Resolved
## Summary On July 2, 2026, from 15:12 UTC to 16:44 UTC, some CircleCI customers experienced failures starting pipelines and difficulty accessing the CircleCI UI, including an inability to log in. Customers triggering pipelines via the API may also have had pipelines fail to run. The incident originated from a combination of internal maintenance and customer-initiated project deletions running concurrently, which caused elevated load on a data service. The connection layer between our API and that data service was not configured to time out slow responses under the elevated load conditions seen during this incident. This eventually exhausted available threads and prevented the request routing layer from handling any other requests, resulting in customer-facing errors. The affected services were scaled up and the internal maintenance job was canceled to reduce pressure, allowing all affected functionality to return to normal. Customers whose jobs failed during this window may rerun affected jobs. A small number of customers may still see workflows that appear to be running; this is actively being corrected and does not impact billing. We thank you for your patience while our team worked on implementing a fix. ## What Happened \(all times UTC\) At 13:30 on July 2, 2026, as part of normal platform maintenance, our engineering team began a data deletion job, divided into several hundred individual maintenance tasks. At 14:40, unrelated to our platform maintenance, a significant number of customer project deletion requests were received and began executing. At 15:12, the combined load from the maintenance tasks and the customer-driven projection deletions caused an internal data service to slow down, and this began cascading into our API layer. Because connections between the API layer and the workflows data service were not configured to time out slow responses under elevated load conditions, the API-handling components became overwhelmed, went unresponsive, and began failing their health checks. This caused our request routing layer to lose available upstream requests, resulting in customers seeing errors when attempting to access the UI, to log in via GitHub or Bitbucket OAuth, or to start pipelines. At 15:16, our monitoring systems alerted and engineers were paged. The team investigated and confirmed the source of the elevated load. At 15:30 Engineers began to scale up the affected services and cancelled a long-running maintenance task to drain the load. The team noted signs of that the system was recovering around 16:12 and adjusted the status to monitoring at 16:24. At 16:44, the team confirmed that all customer-facing services had recovered to normal operating levels. Background load from the deletion job drained by approximately 17:00. Following the incident, we discovered a small number of workflows continued to inaccurately report that they were in a running state. This should be corrected within the next few days. In all cases, customers will only be billed for actual compute time used. ## Future Prevention and Process Improvement Our team is implementing the following improvements as a result of this incident: * After the incident was resolved we added timeout protections on connections between our API layer and internal data services, so that slow responses from a downstream service cannot exhaust API worker capacity. * We are actively investigating ways to enforce resource limits on background data tasks to prevent them from impacting customer-facing services. * We are improving observability and alerting on the affected data service so that resource pressure is detected earlier, before it reaches a level that impacts customers.
Resolved
This issue is now resolved.
Resolved
All service availability has been fully restored. Customers using server-side SDKs might need to restart their applications to re-enable the SDKs' connection to LaunchDarkly. Customers using the Relay Proxy may also need to restart their Relay Proxy; in this case, the Relay Proxy should be restarted before restarting their applications that connect to it. To determine if an application needs to be restarted, customers can look at their application logs. Affected SDKs will emit (depending on language) logs containing one of "giving up permanently", "Invalid SDK key", "unauthorized", or "not authorized", or "401". Services which emitted logs such as these before 9:13AM PT may be unable to receive flag updates and will be unable to send events to LaunchDarkly until they are restarted.
Resolved
This incident has been resolved and any delays have been fully mitigated.
Resolved
LaunchDarkly UI has recovered. We will continue to monitor Flag Delivery and event ingest here: https://status.launchdarkly.com/incidents/chywwz01ptb0.
Resolved
The issue affecting Build Logs has been resolved. Between Jul 8 19:05 UTC and Jul 9 13:16 UTC, some Pro customers using standard machine types may have experienced missing Build Logs. New builds after the impact period are unaffected.
Resolved
This incident has now been resolved.
Resolved
This incident is only affecting a handful of customers and is not widespread, we are following up with affected customers directly.
Resolved
This incident has been resolved.
Resolved
This incident has now been resolved and the service is now fully operational. We're replaying any jobs that may have failed during the window. All our processing queues are fully caught up. We will share a full post mortem as soon as possible.
Resolved
Let's Encrypt service has returned to normal.
Resolved
We are concluding this incident based on a confirmed fix with our infrastructure provider and successful tests. For any customers seeing failed runs, please run again for a successful connection.
Resolved
This incident has been resolved.
Resolved
On July 9, 2026, at 15:43 UTC, customers were unable to login to [cloud.linode.com](http://cloud.linode.com) using username and password. Customers were getting "wrong password" error message. Investigation revealed that the issue was due to an internal certificate issue. To mitigate the impact, we fixed the certificate issue on the impacted servers at 17:24 UTC on July 9, 2026. After monitoring our systems for some time, we confirmed the issue was fully resolved. Akamai will deploy a permanent fix to prevent a recurrence of the issue.
Resolved
This issue was resolved.
Resolved
This incident has been resolved.
Resolved
This issue should now be resolved and all services are restored to normal functionality. Please don’t hesitate to reach out to team@intercom.io: mailto:team@intercom.io if you have any further questions or if you’re still seeing any unexpected behaviour
Resolved
The issue has been resolved.
Resolved
This issue has been resolved. Log ingestion and log visibility have returned to normal operation. If you experience missing or delayed logs from the affected time period, please contact Support and we'll be happy to assist.
Resolved
This incident has been resolved.
Resolved
The issue has been resolved, and we can confirm that services are operating normally again.
Resolved
This incident has been resolved.
Resolved
The issue has been resolved.
Resolved
This incident has been resolved.
Resolved
The fix has been released. Outbound WhatsApp messaging is operating normally.
Resolved
We experienced a degraded performance on our Payout products. Clients may have seen settlement delays. The service has recovered. Impact start time: 09/07/2026, 22:00 UTC Impact end time: 09/07/2026, 22:28 UTC Client actions: None
Resolved
This incident has been resolved.
Resolved
**Incident Summary** Between July 2, 2026, and July 8, 2026, Some customers experienced intermittent failures affecting email and calendar services. The primary impacts included errors when sending or syncing emails and occasional "500 errors" during login. These issues were caused by a technical regression in the underlying runtime environment \(Node.js\) that handles outbound connections to third-party services. For a subset of customers using specific email integration settings, the connection instability caused the system to incorrectly perceive a successful email send as a failure. This triggered an automated retry mechanism, which in some cases resulted in a "retry loop" where the same communication was re-sent to recipients at regular intervals until the connection was stabilized. **Detection** The issue was identified following reports of intermittent service disruptions. Our support and engineering teams conducted a thorough investigation into these reports, identifying a specific pattern of connection failures within our system logs. Further analysis confirmed that the impact began on approximately July 2, 2026, following a routine update to our service infrastructure. **Root Cause** The root cause was a defect in a specific version of Node.js \(version 24.17.0\), the software environment used to run our services. This version contained a flaw in how it managed "keep-alive" connections—reusable connections used to communicate with external services. Under certain conditions, the software would prematurely close these connections before a task was completed, resulting in failed email sends and data sync errors. **Resolution** To resolve the issue, our engineering team performed the following steps: * **Identification:** Correlated the "Premature close" errors to a known regression in the Node.js runtime. * **Upgrade:** Updated our base system images to Node.js version 24.18.0, which contains the official fix for this connection issue. * **Deployment:** Rebuilt and redeployed critical services \(including email and calendar components\) on the corrected version. * **Verification:** Confirmed that connection errors ceased and service stability returned to normal levels on July 8, 2026. **Preventative Measures** To prevent similar issues in the future, we are implementing the following improvements: * **Version Pinning:** We have updated our build process to ensure we deploy only versions that have been fully verified. * **Enhanced Monitoring:** We are adding specific alerts for outbound connection errors to detect and respond to similar communication failures more rapidly. * **Automated Testing:** We are implementing new synthetic checks for email and calendar synchronization to proactively identify issues before they impact customers. * **Improved Visibility:** We are updating our internal dashboards to provide better visibility into the exact software versions running in our production environment to speed up future diagnoses.
Resolved
The underlying issue has been identified and the Account Portal is now fully functioning again.
Resolved
All impacted services have now fully recovered.
Resolved
**Incident Summary** Harness FME \(Feature Management & Experimentation\) faced substantial delays in scheduled impressions data exports on Jul 2, 2026. This issue stemmed from performance degradation within Tinybird's infrastructure, affecting query execution crucial to our export pipeline. **Root Cause** The core issue was a temporary **performance degradation on the provider's infrastructure**. This directly impacted the metadata retrieval endpoint, which failed to return job IDs intermittently. Consequently, completed export jobs underwent repeated retries, leading to a backlog in the export queue. **Impact** * The export backlog accumulated, peaking at more than 6 hours behind the planned schedule. * No data loss was detected during the incident. **Mitigation** 1. **Adjusted Temporal workflows:** Extended timeouts from ~30 to ~50 minutes and increased polling attempts from ~180 to ~300. 2. **Expanded resource capacity.** 3. **Traffic redirection:** Systematically shifted export workloads over to an optimized in-house cluster. 4. **Manual intervention:** Terminated stalling workflows stuck in retry loops from failed metadata lookups. **Next Steps** Complete the migration of all remaining export jobs to the internal analytics cluster \(In progress\).
Resolved
This incident has been resolved.
Resolved
We recently addressed issues affecting Box web access. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future. Between 10:50 AM PST and 11:25 AM PST on February 20, 2026, some users may have experienced difficulties accessing Box via web browser, including failed page loads and intermittent connectivity. During this time, requests to our site that relied on a third-party network service did not route reliably for affected networks, causing DNS resolution and connectivity failures for that hostname. We were able to restore service after the third-party re-advertised the affected network prefixes, at which point we safely re-enabled the dependent routing configuration, restoring normal service behavior. In addition, we are working with the third-party provider and updating our procedures and monitoring to reduce the risk of recurrence and improve detection and recovery. # Analysis This incident highlighted a dependency risk on third-party network prefix advertisement and revealed opportunities to improve early detection and customer-facing communication. It also reinforced the importance of clear playbooks for safely changing routing configurations during third-party outages and ensuring monitoring captures customer-impacting DNS and connectivity failures quickly. # Corrective Actions Box has initiated the following corrective actions: * Work with the third-party provider to complete follow-up and incorporate their post-incident findings * Update our routing and operational runbooks to clarify safe, pre-approved steps for toggling routing configurations during third-party network incidents * Enhance external-facing monitoring and alert thresholds for DNS and site availability to detect and communicate customer impact more quickly We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. Sincerely, The Box Team
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved