Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Jul 12, 2026

  1. ZapierAI by Zapier

    The issue impacting AI by Zapier has been mitigated and systems are back to normal. Impact window: Approximately 19:00–22:20 UTC on July 11. We have automatically replayed the affected runs to recover previously failed executions. If you have any remaining questions or are still seeing issues, please contact support at https://zapier.com/get-help.

    Resolved

Jul 11, 2026

  1. PantheonDashboard Performance Issues

    This incident has been resolved. The Pantheon Dashboard is performing normally, and loading times remain completely stable.

    Resolved

  2. Hume AIEVI and TTS outage

    Issue seems to be resolved. We mitigated the issue by increasing capacity and adjusting timeout/retry handling to improve reliability under load. Service health was confirmed once automated checks and live sessions were consistently succeeding.

    Resolved

  3. NotionDegraded performance on loading pages and facing API errors

    The issue is currently resolved. Sharing the impact timeline in advance: - End of impact: ~8:10am PDT (15:10 UTC) - Timeline: the impact started from 7:34am PDT, severe (index.html/page-load failures) through ~8:03am PDT, and was fully recovered to baseline by ~8:10am PDT, and stable since (1h+)

    Resolved

  4. NeonES-2053073

    Further investigation has confirmed that this issue did not impact Database or project operations on the platform. We apologize for the inconvenience. Next Update No further updates are planned unless there is a material change in service status.

    Resolved

  5. GitHubDelays starting Actions runs

    On July 9, 2026, between 03:29 UTC and 13:39 UTC, GitHub Actions experienced delayed and failed job starts on GitHub-hosted runners. The incident was caused by an unhealthy state in a backend data service responsible for provisioning hosted runners, preventing runner acquisition for a subset of workloads. During most of the incident, approximately 8% of workflow runs on hosted runners were delayed by more than 5 minutes, while roughly 2% failed to start.<br /><br />At 13:39 UTC, we restored the health of the backend data replication system, allowing provisioning to recover and the accumulated workflow backlog to drain. Service performance then returned to expected levels. We are improving provisioning-service resiliency, workload distribution, and capacity balancing to reduce the likelihood and impact of similar incidents.

    Resolved

  6. DigitalOceanKubernetes Deployments in NYC1

    The issue affecting Kubernetes deployments in the NYC1 region has been resolved. Our investigation found intermittent DNS timeouts affecting a small number of DOKS clusters, with affected worker nodes running on shared-CPU Droplets. This is a documented limitation for latency-sensitive cluster DNS workloads such as CoreDNS. The affected clusters are currently functional. To reduce the risk of recurrence, we recommend running CoreDNS on non-shared/dedicated CPU node pools and using sufficient CoreDNS replicas. If you continue to see DNS failures or NodeNotReady events, please open a support ticket so we can investigate that cluster specifically.

    Resolved

  7. 1PasswordDevice Trust Outage

    # Incident Postmortem - Device Trust Authentication Outage **Date of Incident:** 2026-07-02 **Time of Incident \(UTC\):** 19:28 - 20:28 **Service\(s\) Affected:** This incident affected: USA/Global \(Device Trust Authentication Service, Device Trust Checks Server, Device Trust Portal, Device Trust API\) and Europe \(Device Trust Authentication Service, Device Trust Checks Server, Device Trust Portal, Device Trust API\). **Impact Duration:** 1 Hour ## Summary On July 2, 2026, 1Password Device Trust experienced an outage lasting approximately one hour, beginning at 19:28 UTC. During this window, all Device Trust services were unavailable and users were unable to authenticate. We have identified the root cause: an internal configuration change to our cloud load balancing infrastructure was applied more broadly than intended which temporarily made a required endpoint unreachable. This was not a security incident and did not involve any third party. The configuration has been corrected, service has been fully restored, and we have implemented additional safeguards to help prevent this type of issue from recurring. ## Impact on Customers * **All Device Trust services were completely unavailable during the 60-minute outage window. Users were unable to authenticate to any Device Trust service, including the authentication portal \(**[**auth.kolide.com**](http://auth.kolide.com)**\), admin console \(**[**app.kolide.com**](http://app.kolide.com)**\), API endpoints, and device server.** * **Number of Affected Customers \(approximate\):** 100% of Device Trust customers * **Geographic Regions Affected \(if applicable\):** US and EU ## What Happened? On July 2, 2026, A configuration change intended to improve the security configuration of an internal service was applied more broadly than intended, affecting a shared network routing component relied on by Device Trust. The change replaced the underlying network infrastructure supporting Device Trust. However, the traffic routing configuration continued to reference the previous infrastructure instead of the newly provisioned resources. As a result, Device Trust authentication endpoints became unreachable. Reverting the change restored connectivity and service. * **Timeline of Events \(UTC\):** * 19:28: Network configuration change applied; Network Load Balancers responsible for routing traffic for Device Trust services became unreachable * 19:30: Automated monitoring detects the outage and triggers alerts * 19:33: Incident is declared at [status.1password.com](http://status.1password.com) * 20:03: Root cause is identified as a load balancer configuration issue * 20:24: Load balancer is recreated with correct configuration * 20:26 Recovery is confirmed for [app.kolide.com](http://app.kolide.com) * 20:28 Recovery is confirmed for [auth.kolide.com](http://auth.kolide.com) * **Contributing Factors \(if any\):** * **Shared component scope not visible during review or validation.** The network infrastructure class affected by the change was shared across multiple services, but this dependency was not documented or surfaced in the change review process. As a result, the pre-production validation, while performed in a lower environment before production rollout, did not cover all dependent services, and the impact on Device Trust was not identified before the change reached production. * **Concurrent diagnostic signals slowed root cause identification.** At the time the outage began, unrelated certificate renewal errors surfaced alongside the network connectivity failure. Investigating both signals simultaneously delayed identification of the network configuration change as the root cause. ## How Was It Resolved? * **Resolution Steps:** Once the source of the outage was identified, engineers reverted the configuration change. The affected network infrastructure was recreated with its original, publicly accessible configuration, restoring routing to all Device Trust authentication endpoints. Service recovery was confirmed sequentially across affected services before the incident was closed. * **Verification of Resolution:** Following the revert, engineers verified that all Device Trust authentication endpoints were reachable and accepting traffic. Customer confirmation of resolution was subsequently received through the support team. ## What We Are Doing to Prevent Future Incidents * **Require explicit approval for destructive infrastructure changes.** We are implementing safeguards that require manual review and approval before changes that would destroy and recreate critical network infrastructure can be applied. This creates a forcing function to catch unintended scope before changes reach production. * **Improve scope visibility for shared infrastructure components.** We are documenting the dependencies between shared infrastructure components and the services that rely on them, and integrating this information into our change review process. Future changes to shared components will require an explicit assessment of downstream impact before deployment. * **Strengthen pre-production validation across shared dependencies.** We are updating our change promotion process to require validation against all services that share affected components — not only the intended target — before changes are promoted to production. * **Reduce single points of failure in traffic routing.** We are evaluating architectural changes to introduce redundancy into Device Trust's traffic routing configuration, so that a failure in any single component results in degraded service rather than a complete outage. ## Next Steps and Communication * No action is required from our customers at this time. We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding. Sincerely, The 1Password Team

    Resolved

Jul 10, 2026

  1. GitHubActions and Codespaces APIs experiencing partial failures

    On July 7, 2026, between 14:01 UTC and 16:17 UTC the Actions and Codespaces REST APIs were degraded and returned intermittent 500-class errors for a percentage of requests. Error rates peaked at approximately 8% of Actions runner API requests and 13% of Codespaces API requests, though retries were frequently successful. In-progress Actions runs and Codespaces were not impacted and continued successfully. This was due to a recent change that did not deliver the expected performance and, under certain conditions, caused downstream errors.<br /><br />We mitigated the incident by rolling back the change, after which the affected services recovered.<br /><br />We are working to improve the resilience of our services to these conditions and to strengthen our monitoring to reduce our time to detection and mitigation of issues like this one in the future.

    Resolved

  2. KlaviyoCatalog Webhook Processing Delays

    The repair to catch up on product update webhooks from the window of impact has been completed, and affected product data is up to date. This incident has been resolved.

    Resolved

Older incidents