RenderUpstream provider outage causing delays provisioning custom domains
Our upstream provider has resolved the issue and custom domain provisioning has recovered.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
Our upstream provider has resolved the issue and custom domain provisioning has recovered.
Resolved
This incident has been resolved.
Resolved
On May 20, 2026, between 16:00 UTC and 17:45 UTC, GitHub Actions customers experienced run start delays exceeding 5 minutes. Approximately 4.5% of all runs were delayed during the impact window, with scale set jobs disproportionately affected. 30% of scale set jobs were delayed and 4% failed to start entirely. <br /><br />The incident was caused by a misconfigured health check on an internal service that assigns jobs to runners. A brief latency spike in an upstream dependency triggered health check failures across several pods, removing them from service and concentrating load on the remaining capacity. The added load drove memory pressure that escalated into a cascading failure in one regional cluster, leaving it unable to self-recover. <br /><br />Responders mitigated the incident by scaling capacity in the healthy regional clusters and draining traffic away from the impaired one, after which run start latency recovered. To prevent recurrence, we are strengthening our health check configuration to avoid cascading failure scenarios and evaluating automated mitigations to rebalance traffic when a region is degraded.
Resolved
After a monitoring period, we can confirm that the Japanese-translated content in the Talkwalker Help Center has been fully restored and is now available. Thank you for your patience and understanding.
Resolved
## Summary LiveKit Cloud runs as a distributed realtime network, with data centers around the world interconnected via dedicated networking. Even with dedicated fiber, momentary disruptions between any two data centers can and do occur. To handle these blips, we have built a comprehensive set of resilience mechanisms that automatically reroute and relay traffic over alternate healthy paths. For example, if data centers A and B cannot reach one another cleanly but both can reach data center C, we use C as a relay so that traffic flows A to C to B. Under normal circumstances, network blips between our data centers are handled transparently there would be no visible impact. Between 2026-05-07 and 2026-05-19, a small number of these otherwise routine network blips did become customer-visible in our US regions. Affected sessions experienced a brief interruption to media \(under 5 minutes\) before recovering on a new path. ## Impact Five short windows of disruption were observed, each tied to a brief network blip between certain clusters: * 2026-05-07 17:16 UTC * 2026-05-12 05:00 UTC * 2026-05-12 17:00 UTC * 2026-05-13 19:00 UTC * 2026-05-15 07:30 UTC * 2026-05-19 15:01 UTC The majority of sessions traversed alternate paths normally and were not affected. A subset of sessions whose traffic happened to be relayed through a region in the specific failure state described below experienced a media interruption of up to ~5 minutes before re-routing onto a healthy path. ## Root Cause On 2026-05-06, a change was deployed that altered the relay process. The change introduced a subtle bug that required two conditions to occur simultaneously to manifest: 1. The relay region was under-utilized and had not cached a particular piece of state needed by the relay process. 2. A large volume of traffic was directed to that relay region in a very short window of time. When both conditions were present, the relay process would be stuck and would take minutes to fully catch up. During that time, neither endpoint of the relayed session could continue to receive media from the other. Because both conditions are narrow \(a cold-cache relay region absorbing a sudden burst of traffic\), the bug did not surface during pre-deploy testing, and it did not trigger on every network blip. It only manifested when a real network disruption happened to redirect a sufficiently large burst of traffic to a relay region that had not warmed its cache. Once that occurred on a given relay, sessions flowing through it stalled until traffic shifted off that path. We tracked down the root cause on 2026-05-20, and a fix has been fully rolled out across the fleet. ## Corrective Actions & Prevention * **Improved integration tests for relay locks under load.** We are adding tests that exercise the relay process with cold caches and sudden traffic bursts, so this class of contention is caught before deploy. We are also auditing related lock paths to ensure they behave correctly under similar load profiles. We sincerely apologize for the disruption to customers whose sessions were affected during these windows. Thank you for your patience, and we welcome any additional feedback from customers who were impacted.
Resolved
We are no longer seeing issues with CSE Processing Pipeline and normal operation has been restored at this time.
Resolved
This issue is now resolved.
Resolved
This incident has been resolved.
Resolved
A SITUAÇÃO QUE ESTAVA ACONTECENDO na INTEGRAÇÃO COM A GUPSHUP (BSP) do RD CONVERSAS foi resolvida com sucesso. Caso você note que sua conta ainda contém alguma inconsistência, por favor, entre em contato com nosso Time de Suporte através da nossa Central de Ajuda "através da nossa Central de Ajuda": http://ajuda.rdstation.com/..
Resolved
We are no longer seeing issues with Data Indexing and normal operation has been restored at this time.
Resolved
All deployments have been automatically repaired, and HTTP actions are working across the entirety of Convex's cloud. Our team will follow up with a postmortem soon.
Resolved
ClickHouse Cloud no longer supports AWS me-central-1 "AWS me-central-1": https://health.aws.amazon.com/health/status?eventID=arn:aws:health:me-central-1::event/MULTIPLE_SERVICES/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE_5E6B8_EF2498889B5. Customers who were running services in this region have been migrated to their region of choice, and the region is no longer available for selection. No further updates will be posted for this region.
Resolved
Latency for sentry.io has returned to normal.
Resolved
Earlier today, some US customers may have seen degraded or incomplete search results for a number of hours. This issue has now been resolved, and search should be working normally again. The impact was caused by backed-up event processing, which has since been cleared. We’re sorry for the disruption and are continuing follow-up work to help prevent similar issues.
Resolved
### **Intermittent API Errors and Background Processing Delays** **Incident Overview** On May 7, 2026, an infrastructure event in [AWS US-EAST-1](https://health.aws.amazon.com/health/status?eventID=arn:aws:health:us-east-1::event/EC2/AWS_EC2_OPERATIONAL_ISSUE/AWS_EC2_OPERATIONAL_ISSUE_FFA94_A4DC515D30F) \(availability zone use1-az4\) caused degraded performance in Amazon DynamoDB, a component of Chargebee’s infrastructure. Rather than a sustained outage, this manifested as two brief windows of elevated API errors — approximately 3:00–3:15 AM UTC and 4:15–4:50 AM UTC on May 8, 2026, while the rest of the period remained largely stable. Approximately 1.5% of active sites experienced intermittent API failures or background job delays during these windows, with actual impact limited to approximately 15 minutes across the event. All affected background jobs were successfully rescheduled and completed. **What Happened** AWS experienced a thermal event in a single availability zone \(use1-az4\) in US-EAST-1, which impaired a subset of DynamoDB storage partitions. While the AWS-side infrastructure recovery took several hours, the customer-visible impact on Chargebee was limited to intermittent error spikes rather than continuous degradation, reflecting our platform’s ability to absorb partial infrastructure failures. **What We Are Doing** All API errors during this incident were traced to DynamoDB latency caused by the AWS internal AZ-level impairment. Given DynamoDB’s role in our business-critical workflows, we are evaluating highly available alternatives and fallback architectures to reduce the impact of similar infrastructure events. --- We recognise that even intermittent disruption has real consequences for your operations. The errors that spiked during this incident were brief, and our systems absorbed the majority of the AWS recovery window. We remain committed to further reducing Chargebee’s exposure to external infrastructure events
Resolved
Synthetics browser monitors running in Europe-Germany region (GCP europe-west3) for some of the users were experiencing delayed executions results and some of them being stopped. The issue affected both Elastic Cloud Hosted and Elastic Cloud Serverless in that region and other regions were not impacted.
Resolved
Our team has resolved the issue impacting some partners trying to navigate in Workpapers. We apologise for any inconvenience this has caused. A product release caused Workpapers to become unavailable, and a rollback restored service.
Resolved
This incident has been resolved.
Resolved
We have applied a fix and are seeing recovery across our systems
Resolved
This issue is now resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Snowflake Engineering has completed the postmortem of this service incident. A detailed Root Cause Analysis \(RCA\) is available on the Snowflake Community site: [https://community.snowflake.com/s/article/INC0157230](https://community.snowflake.com/s/article/INC0157230)
Resolved
The incident has been resolved.
Resolved
This incident has been resolved.
Resolved
All impacted services have now fully recovered.
Resolved
# Mixpanel RCA: Transient Data Access Issue, May 14, 2026 ## Summary On Thursday, May 14, 2026 at approximately 2:30 PM PT, a routine but infrequent cleanup operation in Mixpanel's storage system mistakenly removed a portion of production data files in addition to the unused files it was intended to remove. Some customers experienced query errors during the hours that followed. We detected the issue within minutes, deployed mitigations the same evening that returned query success rates and latency to normal, and restored the affected files from backup by 5:15 PM PT on Friday, May 15. Mixpanel's ingestion pipeline was not affected and no event data was lost in transit. ## What happened This incident was triggered by a storage cleanup procedure that runs periodically to remove files no longer referenced by Mixpanel's metadata. The procedure was more involved than usual: it followed a recent enhancement to our file storage strategy that left a set of unused files behind in our storage backend, and addressing them required extending our standard cleanup approach to cover a new code path. As part of executing this extended cleanup, an engineer generated the list of files to delete using a SQL query whose date filter was not strictly earlier than the reference snapshot it was being compared against. As a result, a small set of legitimate production files that had been written in the gap window between the snapshot and the filter date were incorrectly classified as unused and removed. The deletion ran for roughly half an hour before internal alerting caught the resulting query failures and the operation was stopped. The trigger was operator error against an ambiguous runbook, not a defect in the live serving path or in our ingestion pipeline. ## Customer impact Impact unfolded in two phases. The first phase ran from Thursday at approximately 2:30 PM PT until 8:11 PM PT — roughly five and a half hours. During this window, customers across the platform may have seen slower or failed queries when their requests touched files that had been deleted. The breadth and severity varied by project depending on which data each query touched. By 8:11 PM PT, mitigations had fully rolled out — queries automatically retried against an alternate availability zone, and a fallback path was put in place to serve missing files from a backup datastore. After this point, query success rate and latency returned to normal. The second phase lasted from 8:11 PM PT Thursday through approximately 5:15 PM PT Friday, May 15. During this window, fewer than 2% of customers were still affected — specifically, those whose deleted files had not yet been fully restored from backup. The vast majority of these files were recovered by Friday afternoon. A small number of projects \(under 30\) had files that could not be fully recovered from backup, and we are following up with those accounts directly. ## Timeline \(Pacific Time\) * May 14, 2:30 PM — Cleanup operation begins * May 14, 3:11 PM — Internal alerting flags query failures; the cleanup operation is stopped within minutes * May 14, 4:07 PM — Status page banner posted * May 14, 4:45 PM — Mitigation deployed: queries automatically retry against an alternate availability zone * May 14, 7:12 PM — Mitigation deployed: queries fall back to a backup datastore for missing files * May 14, 8:11 PM — Query success rate and latency fully restored to normal levels * May 15, 5:15 PM — File restore from backup complete; status page banner resolved ## Why this happened Several contributing factors lined up. The runbook for this cleanup procedure had ambiguous wording around the ordering and timing of its inputs. It had been recently authored to handle the new file-storage code path and had not gone through a formal review before being used. Our cleanup tooling did not programmatically enforce the safety invariant that the date filter must be strictly before the reference snapshot. That invariant lived only in operator-authored SQL. The extended cleanup was being executed in parallel across two storage layers by two different engineers, which increased the room for error. ## What we're doing to prevent recurrence We have already made or have actively in flight the following changes. We are adding programmatic safeguards to our cleanup tooling so that an input set whose date filter is not safely before the reference snapshot is rejected before any deletion occurs, along with a reconciliation step that flags any production-referenced file before deletion proceeds. Destructive cleanup operations will now run in phased stages, starting with internal projects and pausing for a holding period before any broader execution. Destructive storage operations now require a second engineer to sign off on the exact deletion set and to be present during execution, matching the practice we already follow for database migrations. We have updated the cleanup runbook with explicit guidance on input timing, required safety buffers, and an enforced review process for any runbook covering a destructive operation. Longer term, we are working to eliminate the manual portion of this cleanup procedure entirely and route it through our existing automated cleanup infrastructure, so the class of failure that produced this incident is no longer reachable through human input. ## Closing Reliability and data integrity are foundational to the trust our customers place in Mixpanel, and we recognize the impact this incident had on the teams who rely on us. We are sorry for the disruption. If you have questions about how this incident may have affected a specific project, please reach out to your account team or Mixpanel Support.
Resolved
This incident has been resolved.
Resolved
All impacted services have now fully recovered.
Resolved
The incident is now resolved and the system is fully operational.
Resolved
On May 13, 2026, between 14:31 and 16:03 UTC, the Code Scanning service experienced processing delays and 12% of check runs took over 15 minutes to complete. The delays were caused by replication lag due to an internal database migration, resulting in insufficient worker capacity for our high rate of job enqueues. <br /><br />We mitigated the impact by scaling our processing workers by 34%. Code Scanning results returned to normal processing times after the mitigation was applied. <br /><br />The capacity increases are permanent, and we are looking into more ways to decrease the load on our workers to help prevent this in the future.
Resolved
On May 12, 2026, between 13:41 and 17:43 UTC, some services experienced delays in processing. For the Code Scanning service, 53% of check runs took over 15 minutes to complete. Additionally, notifications took an average of 22 minutes to be delivered and Slack integration webhooks took an average of 20 minutes to be delivered. The delays were caused by replication lag due to an internal database migration, resulting in insufficient worker capacity for our high rate of job enqueues. <br /><br />We mitigated the impact by scaling our processing workers to handle the increased load. All services returned to normal processing times after the mitigation was applied. <br /><br />We are working to create dedicated worker pools for some of our high usage shared queues to help prevent this in the future.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Branch creation continues to be stable.
Resolved
The issue is resolved, reporting data is up to date.
Resolved
Approximately between 3:04PM ET to 3:07PM ET, the Kernel API was unavailable due to a misconfiguration. Remediation was executed and restored service by 3:07PM ET.
Resolved
The issue affecting Miro AI has been fully resolved. We apologize for the inconvenience caused and appreciate your understanding. If you continue to experience any issues, please reach out to our support team.
Resolved
The issue affecting Miro AI has been fully resolved. We apologize for the inconvenience caused and appreciate your understanding. If you continue to experience any issues, please reach out to our support team.
Resolved
The issue affecting Miro AI has been fully resolved. We apologize for the inconvenience caused and appreciate your understanding. If you continue to experience any issues, please reach out to our support team.
Resolved