Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Sep 17, 2026

  1. GrafanaElevated Latency Managing Cloud Provider Integrations in GCP US Central

    Between 03:30 UTC on September 14, 2026 and 10:50 UTC on September 17, 2026 (approximately 3 days), customers managing cloud provider integrations in GCP US Central may have experienced elevated latency when creating, viewing, or removing integrations. This has been fully resolved and all services are operating normally.

    Resolved

  2. monday.comPartial UI functionality degradation

    The issue affecting UI platform functionalities, including the Search functionality, has been successfully resolved. Please refresh your browser to resume regular service usage. Thank you for your patience

    Resolved

  3. monday.comIssues with automations and integrations

    The issue with our automations, workflows, and integrations has been successfully resolved. Please refresh your browser to resume regular service usage. Thank you for your patience

    Resolved

  4. ElasticElastic Agent enrollment/check-in failures on 9.5.3 (and 9.4.6) with Fleet remote Elasticsearch output

    We have confirmed that Elastic Agents are no longer experiencing enrollment failures related to this issue. Corrected Fleet Server releases (9.5.4 / 9.4.6) have been available on Elastic Cloud Hosted and Elastic Cloud Enterprise stack packs since September 10, 2026, and all known affected deployments have confirmed recovery. If you upgraded to an affected release before that time and have not yet applied the mitigation, guidance is available here: https://support.elastic.co/knowledge/bee1c75c.

    Resolved

  5. VercelFailures delivering Logs to Drains

    Between September 16, 2026 11:06 UTC and September 17, 2026 07:11 UTC, some customers experienced failed drain deliveries. An update caused larger payloads to exceed a 5 MB delivery-size limit at some Drain destinations. We have implemented a fix that splits larger payloads into smaller payloads before delivery. Affected Drains data could not be redelivered.

    Resolved

  6. ChargebeeAPI Service Degradation

    Between 01:59 UTC and 02:09 UTC, we experienced a brief service interruption in the EU region due to an AWS platform issue, which led to failed API requests for some users. All services have fully recovered. We apologize for any disruption this may have caused

    Resolved

  7. Liquid WebSupport Services Impact

    The issue affecting the third-party platform used by our Support teams has been resolved. The provider has confirmed that service stability and performance have returned to expected levels, and our teams are no longer seeing ongoing impact. Support case access and case management functionality have returned to normal. We apologize for the disruption and any delays this may have caused.

    Resolved

  8. StripeElevated declines on Visa

    Between 20:15 September 16th and 00:08 September 17th UTC, we saw elevated card authorization errors related to an issuing bank in United States which impacted some Visa transactions. This is now resolved.

    Resolved

  9. ClickHouse CloudClickPipes failing on Kinesis in AWS us-east-1

    This incident has been resolved. AWS has confirmed that the Enhanced Fanhout (EFO) Kinesis pipes in us-east-1 region are fully operational. Please contact ClickHouse Support if you experience any further issues: https://console.clickhouse.cloud/support

    Resolved

Sep 16, 2026

  1. Box[Medium] Issues affecting multiple Box services

    We recently addressed issues affecting some customers at Box. We would like to take this opportunity to explain what happened and the steps we have taken to prevent them from occurring again. ‌ On August 6, 2026, between 6:42 PM PDT and 6:48 PM PDT, and again between 7:27 PM PDT and 7:33 PM PDT, some customers may have experienced intermittent disruptions affecting login, uploads/downloads, Notes, Box AI, and API access in Box. The disruptions were a symptom of a temporary capacity issue in the infrastructure that managed incoming traffic to Box. We restored service by increasing capacity and adjusting our systems to better handle traffic under load. # Analysis We recently introduced automatic capacity scaling to help Box respond more efficiently to changes in customer traffic. However, a configuration error caused our systems to release capacity too quickly, leading to repeated cycles of scaling up and scaling down. These rapid fluctuations disrupted active customer sessions and contributed to the issues described above. # Corrective Actions Box has initiated the following corrective actions: ‌ * Restoring and increasing front-end capacity, and updating our scaling configuration to prevent capacity from being reduced during periods of high traffic. * Updating our internal procedures to roll out capacity changes gradually, so that any impact is limited to a small portion of our infrastructure and broader customer impact is avoided. * Adding alerts for connection failures and capacity thresholds to our critical escalation processes, enabling our teams to respond to similar issues more quickly in the future. ‌ We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope this provides clarity and we would be happy to answer any remaining questions you may have regarding this matter. ‌ Sincerely, ‌ The Box Team

    Resolved

  2. Box[Critical] Issue with Downloads

    We recently addressed issues affecting Downloads \(including ZIP downloads\) in Box. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future. Between August 24, 2026 at 11:25 PM PDT and August 25, 2026 at 12:05 AM PDT, some Box Zones Japan users may have had a degraded download experience. The disruption was caused by an unusual traffic pattern that placed unexpected strain on capacity in the region. Once the impact was identified we restored service by directing download traffic to another region and adding capacity. Downloads returned to normal by 12:05 AM PDT. Once confirming that the issue had been addressed, we returned traffic to the original region. Of note, even during the period of temporary traffic redirection, all Box Zones customer content was stored in the correct Box Zones region. # Analysis This incident highlighted opportunities to make our systems more resilient to unusual traffic patterns, to strengthen regional capacity safeguards, and to further improve monitoring so that conditions like these are detected and addressed before affecting customers. # Corrective Actions Box has initiated the following corrective actions: * Improving safeguards so that unusual activity in one part of the system cannot affect unrelated services. * Strengthening regional capacity protections and automated recovery. * Enhancing monitoring and alerting so we detect and resolve similar conditions faster. We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. Sincerely, The Box Team

    Resolved

  3. WorkOSForm POST failures for some Authkit customers

    We're sorry for the disruption this incident caused for you and your users. Authentication is a critical path, and during this window we did not meet the reliability standard you expect from WorkOS. We take responsibility for the impact. Service was restored the same day, and we have already shipped several lasting fixes while continuing broader cleanup and detection work. # Postmortem: AuthKit custom domain sign-in failures **Date:** September 9, 2026 **Duration:** 21:09 – 22:30 UTC \(~80 minutes of customer impact\) **Status:** Resolved ## Summary On September 9, 2026, from approximately **21:09 to 22:30 UTC**, a small subset of customers using **AuthKit custom domains** experienced failures when submitting authentication forms \(sign-in, sign-up, MFA verification, device authorization, and email verification\). Users on affected domains saw **HTTP 500** errors on those form submissions, which blocked interactive login for the duration of the incident. This was a **partial outage**. It affected a small cohort of AuthKit custom domains provisioned through an earlier self-serve configuration path—**about 0.17%** of AuthKit custom-domain traffic was actually affected by the failure. Default AuthKit domains \(`*.authkit.app`\), the WorkOS API, the Dashboard, and **GET** requests \(page loads and most OAuth callback traffic\) on custom domains continued to work. The failure mode was specific to **server-action POST** requests used by AuthKit hosted UI forms. We first learned of the issue from a customer report at approximately **22:04 UTC** \(~55 minutes after impact began\). Engineering restored service by **22:30 UTC**. No evidence of unauthorized access, data exposure, or data corruption was found. ## Background AuthKit can be hosted on a customer’s own hostname \(a **custom domain**\). Traffic for those hostnames is terminated at our edge and forwarded to the AuthKit application with HTTP headers that identify the original host and browser origin. The AuthKit application relies on those headers so that browser security checks for interactive form submissions \(Next.js server actions\) succeed. Over time, WorkOS evolved how custom-domain traffic is routed at the edge—from earlier **per-domain** configuration toward shared, list-based edge rules and a newer application hosting path. During that migration, a set of **legacy per-domain header rules** remained in place for domains created in an earlier era. A later **zone-wide** edge rule normally overwrote those headers so the application always saw a consistent, CSRF-safe origin. While that zone-wide rule was enabled, the older per-domain rules appeared inert. ## What happened As part of decommissioning older edge configuration no longer needed for the current hosting path, the zone-wide header transform rule was **disabled** in production. That change unmasked the leftover per-domain rules for the older custom-domain cohort. For those domains, the edge continued to rewrite the browser `Origin` header to the tenant’s default AuthKit hostname, while the application still received the customer’s custom hostname as the forwarded host. AuthKit’s application layer correctly rejected the mismatched server-action POSTs as invalid \(CSRF / same-origin protection\), returning HTTP 500 before application business logic ran. **Scope** * Structurally exposed: AuthKit custom domains still carrying the legacy per-domain header configuration \(on the order of a few dozen hostnames\). * Observed traffic impact: a subset of those domains that received form POST traffic during the window \(about sixteen hostnames with material volume\). * Scale: roughly **2%** of AuthKit custom-domain traffic comes from potentially affected domains; **~3,100** failed server-action POSTs over the ~80-minute window. For affected domains, failure rates on form POSTs were often near **100%**. **Symptoms** * Users on affected custom domains could not complete interactive AuthKit forms \(sign-in and related flows\). * Default AuthKit hostnames and non-form traffic paths remained healthy. **What went wrong** The proximate trigger was disabling a shared edge header rule that had been **silently correcting** residual per-domain header rewrites left over from an earlier custom-domain design. The durable underlying condition was incomplete cleanup of that legacy configuration after the supported path moved to list-based rules—and the absence of automated detection for this failure shape. **Why it wasn’t caught sooner** Several detection gaps stacked: 1. Pre-change validation and canaries primarily covered **newer** custom domains \(and often **GET** probes\). This failure only appears on **server-action POSTs** for the legacy cohort, with no representation of this configuration in our validation environment. 2. Aggregate AuthKit error rates stayed relatively low because only a small share of traffic was affected, so fleet-wide thresholds did not page. As a result, detection depended on customer reports rather than an automated page. ## Remediation ### Completed since the incident * **Worker safety net** — Added application-edge protection that detects and drops stale per-domain origin values so legacy rules can be retired without recreating this mismatch, even if rule order differs. * **Staging validation path** — Reproduced the legacy custom-domain configuration in staging so further edge cleanup can be proven safe before production changes. ### In progress / planned * **Remove legacy per-domain edge rules** for AuthKit custom domains after staged validation, so shared transform rules are no longer load-bearing masks. * **Inventory residual edge configuration** tied to custom domains \(including orphaned rules\) so leftover resources have clear ownership and review. * **Per-customer / concentrated-failure detection** so a small set of fully down tenants is visible even when fleet-wide error rates look healthy. * **Stronger change gates** for catch-all / list edge-rule changes: same-phase inventory of earlier rules that can match the same hosts, and canaries that include **legacy-era domains** and **server-action POST** probes—not only GET checks on newer domains. * **Improved change visibility** for production edge-rule state \(today some zone rules are dashboard-managed without a PR trail\). We treat restoring the shared transform as necessary containment, not the durable end state. The combination of residue removal, better probes, and better detection is intended to prevent both recurrence of this exact mechanism and silent customer-total failures on a small hostname set. ## Closing A configuration cleanup on the AuthKit custom-domain edge path exposed leftover rules from an earlier design and blocked interactive sign-in for a limited set of custom domains for about eighty minutes. We restored service quickly once the issue was identified, and we are following through so legacy edge residue cannot remain load-bearing, so validation covers the domains and request types that actually break, and so concentrated authentication failures are visible to us before they are visible only to your users. We regret the impact on affected customers and their end users, and we appreciate the reports that helped us engage. If you have questions about whether your integration was in the affected cohort, please contact WorkOS support \([support@workos.com](mailto:support@workos.com)\).

    Resolved

  4. DigitalOceanSupport Portal Service Disruption

    From 08:06 UTC to 13:41 UTC, users may have experienced issues accessing the Support Portal, creating new support tickets, or reviewing existing support requests. Our Engineering team has confirmed the full resolution of the issue affecting the Support Portal following a fix from our upstream provider. We have verified that the Support Portal, ticket creation, and ticket review are now operating normally. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience and appreciate your patience.

    Resolved

Older incidents