StripeElevated Authentication Requests
Between 11:40 UTC and 14:35 UTC we observed elevated Authentication Requests for card transactions in SCA scope. This issue has now been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
Between 11:40 UTC and 14:35 UTC we observed elevated Authentication Requests for card transactions in SCA scope. This issue has now been resolved.
Resolved
We continue to work with our DNS provider to discover the source of the issue. We currently believe approximately 0.37% of traffic was impacted by being unable to resolve DNS of the CDN that provides ClerkJS.
Resolved
All impacted services have now fully recovered.
Resolved
This incident has been resolved.
Resolved
The Test Engine backlog is now cleared and operating normally.
Resolved
This incident has been resolved.
Resolved
Latency has returned to normal operational bounds and this incident is now resolved. Twilio's Statuspage will continue to provide updates on the remaining impact to Romania, Serbia, Poland, Bosnia and Herzegovina, and Cabo Verde: https://status.twilio.com/incidents/shhrhbqxflt1
Resolved
This incident has been resolved.
Resolved
We are no longer seeing issues with Billing and Account Management and normal operation has been restored at this time.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The underlying issue was resolved.
Resolved
This incident has been resolved.
Resolved
All impacted services have now fully recovered.
Resolved
### Service Impact Between 02:29 and 02:48 UTC on May 1st 2026, the Buildkite dashboard, Agent API, REST API and GraphQL were severely degraded for the majority of customers. During this period, customers were unable to reliably load pages in the dashboard, and agents experienced errors when communicating with the Agent API. Service was restored for all customers by 02:48 UTC, with a small number of follow-up support cases handled in the hours afterward. ### Incident Summary At 02:29 UTC, a database migration was applied that renamed a column on the users table in our foundation database, which is a core, unsharded table referenced across the Buildkite platform. The migration ran before the new application code was deployed. During the rollout window, the application code already running in production still expected the original column name. Because database migrations run before application code is deployed, queries that loaded a user record began failing immediately across the product, and continued to fail until the column was manually restored. The impact was detected at 02:29 UTC by an automated alert, and several customers reached out to support within minutes. Initial investigation focused on preparing an emergency rollback, but at approximately 02:41 UTC it was identified that the change was an unsafe migration and that a rollback would not restore service. We then moved to manually recreate the renamed column on the users table. This change was applied at approximately 02:48 UTC, at which point error rates returned to normal levels across the affected services. Total customer-facing impact was 19 minutes. ### What contributed to this happening Three things came together to allow this change to ship without the safety pre-steps that would normally precede a destructive migration: The migration was generated by an LLM-assisted workflow, which added the `safety_assured` annotation as part of the generated change. `safety_assured` is intended as an explicit engineer acknowledgement that the safety implications of a destructive migration have been considered and prepared for not as a routine flag added by tooling. We use Danger \([https://danger.systems/ruby/](https://danger.systems/ruby/)\) to call out risky database migrations with specific guidance for certain operations such as index changes — but there was no specific guidance for `rename_column`. The generic migration warning that _did_ exist had, after dozens of routine migrations, become background noise. The migration was bundled into the same PR as the application code that depended on the renamed column. This made the schema change easier to overlook in review, and meant the test suite for the PR only ever exercised the new code against the new schema, never the previously-deployed code against the new schema. This combination meant the unsafe sequence reached production with no automated test or human review specifically focused on whether the _currently-running_ application could tolerate the schema change. ### Changes we're making This incident was caused by a database migration that was not adequately prepared for safe deployment, and exposed gaps in both our review process and our use of code generation tooling. We're making the following improvements to prevent similar incidents: Database migrations that are unsafe to apply against the currently-running application code will be flagged and blocked before they can be merged. We are still working through the exact mechanism for this \(one option under active discussion is requiring schema migrations to ship in separate pull requests from any application code that depends on them\), but the intent is the same: the test suite on a migration's PR exercises the post-migration schema against the application code already running in production — the exact intermediate state that exists between the migration running and any new code shipping — so that backwards-incompatibility shows up as a failing test on the migration PR itself, where it can be caught by branch protection rather than by a human reading a routine review comment. We're extending Danger, the automated review tool that comments on every pull request that touches a database migration, to specifically highlight \`rename\_column\` amongst the other higher-risk operations it already highlights, so that they stand out from routine migration changes during code review. We're updating our internal engineering guidance \([AGENTS.md](http://agents.md/)\) to make clear that protective constructs such as safety\_assured \(from [https://github.com/ankane/strong\_migrations](https://github.com/ankane/strong_migrations)\) are intended as engineer-acknowledged sign-offs and should not be added by code-generation tooling on an engineer's behalf. Finally, we're reviewing how we surface and document the required pre-steps \(such as adding ignored\_columns\) for breaking schema changes, so that the safe rollout sequence is harder to skip, particularly for changes to unsharded core tables where there is no opportunity for a phased rollout. The changes described above protect against the _test-detectable_ class of schema/code mismatches. There remains a class of failure where a destructive migration to an unsharded core table affects every customer at once with no opportunity for a staged rollout. Our longer-term work on **cell-based deployments,** splitting customer data across multiple foundation databases, is what will give us the ability to roll out schema changes incrementally and detect issues against a small subset of customers before they reach the full fleet.
Resolved
Latency and error rates have remained at normal levels for all services, post-implementation of our fix, and this incident is resolved. We will continue to monitor the overall system health until AWS reports full recovery. Please see https://health.aws.amazon.com/health/status for more details.
Resolved
Reddit Conversions API destination is operational.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We are no longer seeing issues with Metrics Alerting and Metrics Ingestion and normal operation has been restored at this time.
Resolved
This incident has been resolved.
Resolved
The issue has been resolved.
Resolved
From 03:08UTC to 03:14UTC we saw increased request error rates and webhook delivery latency increases. The issue is mitigated and systems are operating normally.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Between 05:23 PM (UTC -07:00) and 07:56 PM (UTC -07:00), HubSpot experienced degradation in portals in the U.S. . This was related to an AWS outage. HubSpot is now working properly and the incident has been resolved. HubSpot conducts a thorough review after each incident to understand the cause and prevent it from happening again. Learn more about HubSpot's commitment to reliability at www.HubSpot.com/reliability. The information on this page reflects our understanding of the incident and impact at the time of the update
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We have identified the root cause and are putting in place mitigations to ensure this does not happen in the future.
Resolved
We’ve mitigated the impact from AWS’s Availability Zone issue, and service has remained stable. We have not observed further node failures or customer impact, so we’re marking this incident as resolved. We’ll continue to monitor AWS’s recovery separately.
Resolved
Alright folks, all services are stable so we're closing this out. Thanks for your patience and apologies for the disruption today!
Resolved
Log-in and Matchmaking has recovered and we're no longer seeing any server delays. We'll continue to monitor these services and address any issues that arise.
Resolved
We see full recovery of the query subsystem. For a small number of metrics datasets, there might be data missing between 00:15 and 01:24 UTC. We're working on recovery with our infrastructure provider.
Resolved
This incident has been resolved
Resolved
This incident has been resolved.
Resolved
Between 20:03 UTC and 22:13 UTC on May 7, 2026, a subset of customers using Neon databases in AWS us-east-1 may have experienced issues with compute endpoints not scaling down to zero after periods of inactivity, as well as issues starting recently suspended computes. The issue has been fully resolved, and all services are operating normally. We apologize for any inconvenience this may have caused. If you require further details about this incident, please submit a support ticket.
Resolved
This incident has been resolved.
Resolved
All Klaviyo services have been fully restored and final recovery work is complete. We appreciate your patience throughout this incident. This incident has been resolved.
Resolved
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Resolved
DNS updates for ACME are confirmed healthy. Cluster updates are now proceeding as expected.
Resolved