LiveKitInvestigating alerts for increased timeout error rates on ListParticipants APIs
Error rates on ListParticipants API requests recovered at 18:58 UTC and have remained stable since.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
Error rates on ListParticipants API requests recovered at 18:58 UTC and have remained stable since.
Resolved
Access to the Axiom Console through Github is available as well as all other authentication methods. Users might experience a one-time email verification step for their Primary Github email. The Primary email address should be configured as the email used with an Axiom account. Additional verified emails can be used to access separate Axiom accounts.
Resolved
This incident has been resolved
Resolved
On August 11, 2026, between 14:00 UTC and 16:00 UTC the GraphQL API service was degraded and customers in saw higher than normal timeouts. On average, the timeout rate was 0.06% and peaked at 0.14% of requests routing to the service. <br /><br />This was due to increased utilization at one of our sites which caused resource contention across our dependencies, leading to an increase in timeouts for GraphQL requests. We mitigated the incident by increasing capacity to alleviate the capacity bottleneck. <br /><br />We are working to improve our monitoring so that we can proactively reduce the impact of high consumption requests in addition to scaling up; Additionally, we will improve our time to detection and mitigation of issues like this one in the future.
Resolved
Sessions were temporarily extremely slow to start or be resumed.
Resolved
Campaign scheduling has returned to normal operation. If you scheduled a campaign between 1:15 PM and 2:00 PM US/Eastern on August 12, 2026, we recommend checking its status and rescheduling if needed. This incident has been resolved.
Resolved
On July 29, 2026, between 19:45 UTC and 21:51 UTC, users of the Fable 5 model in GitHub Copilot experienced increased request failures and latency. The average failure rate across all Copilot requests was 0.006%, while failures for Fable 5 peaked at 21%. Other models remained available. This was caused by degradation of an upstream model provider.<br /><br />The affected endpoint recovered, and we monitored the service until error rates and latency returned to normal levels. We are working to add endpoint redundancy to mitigate similar provider issues in the future.
Resolved
The issue impacting customers who use GitHub as their VCS while GitHub was undergoing a service degradation (https://www.githubstatus.com/incidents/76t89hbfb09h) has now been resolved. GitHub has resolved the underlying issue and the affected functionality has returned to normal. We thank you for your patience.
Resolved
On August 1, 2026, between 17:47 UTC and 18:20 UTC, users of the Fable 5 model in GitHub Copilot experienced increased request failures and latency. The average failure rate across all Copilot requests was 0.007%, while failures for Fable 5 peaked at 5.6%. Other models remained available. This was caused by degradation of an upstream model provider.<br /><br />The affected endpoint recovered, and we monitored the service until error rates and latency returned to normal levels. We are working to add endpoint redundancy to mitigate similar provider issues in the future.
Resolved
Documentation site now loads after refreshing.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
On 2026-08-03, between 06:52 and 11:25 UTC, some GitHub Copilot users experienced errors when using chat and agent features. Requests to list the available models failed, and because every chat or agent interaction begins by retrieving the list of models, affected users saw their requests fail. On average about 3% of these model-listing requests failed during the incident (roughly 97% succeeded), but failures were significantly higher during peak-traffic periods, at times approaching 100% for the affected internal lookups. Approximately 4,066 users were affected in a single 60-minute window, concentrated among IDE-based clients. The underlying AI models themselves remained healthy throughout.<br /><br />The incident was caused by an increase in how often clients requested the model list, which pushed an internal user-authorization lookup past a rate limit; the rate-limited responses were surfaced to users as errors. We mitigated the impact by increasing how long Copilot caches that authorization lookup, which reduced load on the internal service, and we have additional capacity and rate-limit changes in progress. To prevent recurrence we are improving monitoring for this class of failure, adjusting cache and rate-limit settings, and coordinating with client teams on request patterns.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
The backlog has been processed
Resolved
API performance has returned to expected levels.
Resolved
The issue is resolved now
Resolved
The issue has been successfully resolved. Thank you for your patience
Resolved
Elasticsearch 9.5.1 has been released and contains the fix for this issue. Customers running 9.5.0 should upgrade to 9.5.1. At this time we are considering this issue resolved and will be providing no further updates.
Resolved
We are pleased to inform you that the incident affecting our Barbican offer has been resolved. <b>Start time :</b> 07/08/2026 09:41 UTC <b>End time :</b> 11/08/2026 11:11 UTC <b>Root Cause :</b> A service disruption occurred due to an unexpected underlying infrastructure malfunction. We apologize for any inconvenience caused and appreciate your understanding.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
This incident has been resolved
Resolved
For the last week we have been internally testing our agent traffic running on our new EKS infrastructure successfully. At 23:01 11 August 2026 UTC we started routing <1% of agent traffic for a small subset of our customer to this EKS infrastructure. Within that subset of customers, any that were using an agent version < 3.97.0 saw an error "failed to decode JSON response: invalid character '<' looking for beginning of value". These older agents crashed once they received that error, which for some customers triggered repeated agent cycling. We reverted the routing to EKS infrastructure at 03:25 UTC after being alerted to the issue.
Resolved
This incident is resolved. Between 00:20 and 04:15 UTC on 12 August, some HTTP Actions returned a 404 response. This affected requests that reached Convex through an intermediate proxy or CDN, because those requests carry an `X-Forwarded-Host` header. A change to our edge configuration made it read that header instead of the request's own hostname, so it looked for the wrong deployment. We reverted the change in all regions and error rates are back to their normal level. No data was lost, and requests that did not carry that header were unaffected. We are sorry for the disruption, and are improving our testing and monitoring of edge configuration changes to catch issues like this before they reach production.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
The issue has been fully resolved and all services are now back to normal operation. We apologize for any inconvenience caused and thank you for your patience.
Resolved
All impacted services have now fully recovered.
Resolved
all our tests are back to normal
Resolved
All systems remain stable during the monitoring period after the mitigation was implemented. The incident response has been stood down.
Resolved
Minimax M3 serving is fully restored.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved