Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Aug 10, 2026

  1. NeonIssue with starting computes in us-east-1

    Between 14:30 UTC and 14:57 UTC on August 10, 2026, a subset of customers in us-east-1 region may have experienced an unavailability and blocked compute suspension. Startup of new computes and scale from zero was not affected. The issue has been fully resolved, and all services are operating normally. We apologize for any inconvenience.

    Resolved

  2. GrafanaMetrics: Elevated Error Rates Reads/Writes

    Between 12:20 and 13:15 UTC, we experienced elevated error rates for metrics reads and writes due to a networking issue. Impact was limited to a subset of deployments in the prod-us-central-0 region. A fix has been applied and error rates have since recovered.

    Resolved

  3. MintlifySearch Service Disruption

    The issue affecting search functionality across documentation sites has been resolved. Search is now fully operational. The disruption was caused by a billing issue with a third-party service provider, which led to a temporary suspension of service. This has since been resolved and full service has been restored.

    Resolved

  4. BufferBuffer Login Disruption

    Things have remained stable following the fix, and we’re now marking this incident as fully resolved. Publishing success rates have returned to normal, and we’re no longer seeing reports of users being unexpectedly logged out. We’re sorry again for the disruption, and we appreciate your patience while our team worked to get things back to normal. Please reach out to us at hello@buffer.com if you're still experiencing issues.

    Resolved

  5. SnowflakeINC20000115

    Snowflake Engineering has completed the postmortem of this service incident. A detailed Root Cause Analysis \(RCA\) is available on the Snowflake Community site: [https://community.snowflake.com/s/article/INC20000115](https://community.snowflake.com/s/article/INC20000115)

    Resolved

  6. Box[Medium] Issues with File Requests

    We recently addressed an issue affecting metadata updates using File Request in Box. We would like to take the opportunity to further explain the issue and the steps we have taken to keep similar issues from happening in the future. Between 09:10 AM PDT on July 22, 2026 and 03:36 PM PDT on July 23, 2026, some users may have experienced delays in metadata updates while uploading files using File Request. The issue occurred when repeated attempts to process a subset of unsuccessful metadata events reduced available processing capacity and delayed other events. We restored the metadata functionality of File Requests by limiting unnecessary retries, reducing processing idle time, and improving the handling of failures that would not succeed when retried. In addition, we are improving retry behavior, monitoring, and response procedures for similar asynchronous processes. # Analysis At the time of the issue, the system automatically retried failed file processing. During an extended period of failures, these retries consumed most of the system's capacity, causing files to pile up in a queue. As processing slowed, more files encountered issues that retries could not resolve, further increasing the delay. We restored normal processing by temporarily reducing retry attempts, removing redundant retries, and allowing permanent failures to fail without being repeatedly processed. We continued monitoring the service before confirming that full functionality had been restored. # Corrective Actions The following corrective actions have been completed or are planned: * Completed changes to limit retries and stop repeatedly processing failures that cannot be resolved through retries. * Applying the same, improved retry handling to other asynchronous processes, preventing similar occurrences from happening to other File Request features. * Improving alerting and response procedures for processing backlogs and retained unsuccessful events to ensure that temporary issues can later be retried safely. We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. Sincerely, The Box Team

    Resolved

  7. BuildKiteLatency on Pipelines

    ## Service Impact Between 04:40 and 05:34 UTC on 3 August 2026, Buildkite stopped running recurring background work. Every background job process across the product was affected. The customer-visible impact was concentrated in Pipelines. During this window, scheduled builds did not start at their scheduled time, jobs waited longer than usual before being dispatched to an agent, in some cases by more than ten minutes, and pipeline uploads and inbound webhook processing were delayed. Job timeouts were not enforced, so a job that should have timed out kept running, and agents that had finished or gone away were not cleaned up promptly. Scheduled builds were not skipped. Every schedule that came due during the window ran once background scheduling recovered, up to around forty minutes late. A schedule that fires more often than that ran once on recovery rather than once for each interval it missed. No builds, jobs or queued work were lost. Work that was delayed was processed once the incident was resolved, and jobs that errored were retried automatically. The dashboard, the REST and GraphQL APIs, and agents already running jobs were not affected. ## Incident Summary At 04:33 UTC we merged a routine update to Rails, the web framework Buildkite runs on, taken to stay current with a security patch. Alongside Rails, that update pulled `connection_pool` from 2.5.5 to 3.0.2. The 3.0 release made that library's API keyword-only, which changes how every caller has to invoke it. Our review of the change covered our own code's use of `connection_pool` and updated it accordingly. It did not cover the other gems in our bundle that call the same library. One of those is Sidekiq, which runs our background jobs. From 04:40 UTC, as the deploy rolled out, three internal Sidekiq threads inside every worker process failed on startup and stopped. The processes themselves stayed healthy and kept working jobs that were already queued, so nothing crashed and no deploy failed. What stopped was the scheduling of recurring work, which is what triggers scheduled builds, dispatches queued jobs to agents, enforces job timeouts and cleans up finished agents. That scheduling currently runs through a single elected process for the whole background job fleet, so it stopped for every product and every customer at the same moment. Our monitoring alerted us at 04:48 UTC and we identified the change responsible at 04:59. Our first assessment was that there was no customer impact, because everything we could see was still moving. It took another ten minutes to recognise that recurring work had stopped across the whole product, at which point we published a status page and raised the severity. A fix pinning `connection_pool` back to 2.5.5 was merged at 05:10 and deployed ahead of the normal merge queue. Scheduling resumed at around 05:19 and the incident was resolved at 05:34. ## What contributed to this happening `connection_pool` is called both by our own code and by gems we depend on. Our review, and the automated review tooling that flagged the change on the pull request, both looked at how we call it and stopped there. Nothing in the dependency graph prevented the combination. Sidekiq 7.3.9, the version we run, permits any version of `connection_pool` from 2.3.0 upwards. Sidekiq 7.3.10 added an upper bound that would have refused 3.0, but we had not adopted it. The failure only appears when a real Sidekiq process starts up. Our test suite does not exercise that path, and our pre-production check verifies that the application boots rather than that background scheduling is still running. Because recurring work is scheduled by a single elected process for the whole fleet, one failure removed it for every customer at once rather than for a single group of customers. ## Changes we're making We have pinned `connection_pool` to a version compatible with the Sidekiq release we run. This shipped as part of resolving the incident. We are upgrading Sidekiq to 7.3.10, which declares an upper bound on `connection_pool` and so makes this combination impossible to resolve rather than something we have to remember to avoid. We are changing how recurring work is scheduled so that it is no longer driven by a single process for the whole fleet. Each group of shards will schedule its own recurring work independently. This would not by itself have prevented this incident, because the incompatible library was loaded by every process, but it removes a shared point of failure that can currently stop recurring work for every customer at once. This work is already in progress. We are updating our internal engineering guidance so that an upgrade moving a shared library to a new major version is reviewed against every consumer of that library, including the other libraries that depend on it, rather than only our own code.

    Resolved

Aug 9, 2026

  1. AivenDelayed VM Creation

    New VM creation has returned to normal across all cloud providers, including DigitalOcean. * VM provisioning is operating normally across all regions. * The upstream DigitalOcean incident has been resolved. * We are continuing to monitor the infrastructure to ensure stable provisioning. The incident is now resolved. Thank you for your patience.

    Resolved

  2. DigitalOceanAccount Registration, Droplets, and Related Services

    Our Engineering team confirms the full resolution of the issue affecting all impacted services, including new account registration, Droplet Creation, Reserved IPs, Automated Backups and Snapshots, Droplet Autoscale, DOKS clusters, GenAI services, the Droplet Console, and Database cluster creation. We apologize for the inconvenience. If you continue to experience any issues, please open a support ticket from within your account.

    Resolved

Aug 8, 2026

  1. ExpoIncreased EAS Update manifest error rates

    At around 4 PM PDT, we experienced a temporary spike in EAS Update manifest serving errors. Service reliability has returned to normal, and we are monitoring the situation.

    Resolved

Aug 7, 2026

  1. LiveKitA percentage of cross-region calls between US West and US Central not completing

    This incident has been resolved. Between 19:45 and 20:00 UTC, calls crossing the US West <-> US Central link experienced intermittent degradation in Room Service and SIP performance. The underlying network between the two regions had episodes of high packet loss during this time period. We rerouted traffic and performance has returned to baseline as of 22:00 UTC. We will follow up with a postmortem, which will include further details regarding scale and scope.

    Resolved

  2. KinstaService disruption at the Frankfurt (DE) data center

    The server has remained stable since the solution was implemented. The incident is resolved at this time. If you have any questions or are continuing to see any issues please do not hesitate to reach out to our Support team. We are available 24/7 and always happy to help.

    Resolved

  3. FactoryOpenAI Models Returning 431 Errors

    We deployed a backend fix for the errors affecting OpenAI models and have confirmed that error rates returned to normal. Users can resume normal usage; no Factory CLI or Desktop update or other action is required.

    Resolved

Older incidents