RadarElevated API latency and errors
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The processing of in-app events is back to normal and the issue is now fully resolved. Thank you again for your patience.
Resolved
This incident has been resolved.
Resolved
Llama 3.1 8B is having intermittent timeouts has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved. Please contact GOV.UK Technical Support through the usual means, if you see anything wrong.
Resolved
**Dates:** Start Time: Thursday, December 12, 2024 at 19:24 UTC End Time: Thursday, December 12, 2024 at 20:54 UTC Duration: 1 hour and 30 minutes **What happened:** Log lines submitted to Mezmo for ingestion into Log Analysis were never made available in our WebUI for Searching, Graphing, and Timelines – neither during the incident, nor afterwards. This affected a significant number of accounts. Log lines were still passed through Pipeline and made available at all times within Live Tail. The log data still triggered both Telemetry Pipeline and Log Analysis based alerts, and were still archived in both places. **Why it happened:** A single pod within our indexing service ran out of disk space. This was due to a sudden increase in the volume of log lines sent to Mezmo by accounts that happened to be assigned to this pod. Our pods are configured to limit how much disk space is available for writing new data; these limits should have prevented any pod from running out of disk space, in any scenario. After the incident, we discovered that the limits had been configured incorrectly, which explains why it was possible for this pod to run out of disk space. Our service is designed to tolerate the loss of a single pod within its indexing service without any widespread impact to customers. Instead, for reasons still under investigation, the impact expanded to many other pods. Our service is also designed to retain submitted log lines, even if the indexing portion of our service cannot process them immediately; these log lines can be indexed later, when the service is functional again. In this incident, however, the log lines were never indexed. The reason for this failure is also under investigation. **How we fixed it:** We restarted the pod that had run out of disk space. It immediately had enough free disk space to accept new log lines for indexing. All other indexing pods also returned to a normal operational state. **What we are doing to prevent it from happening again:** We have properly configured our pods to prevent them from running out of disk space. This step alone should prevent any recurrence of the same problem. We have updated our monitoring to send high priority alerts when any indexing pod is in danger of running out of disk space. These alerts were in place before, but set to “low” priority; they did not come to our attention in time to prevent the incident. We will continue to actively investigate why the impact spread to other indexing pods and why log lines were not retained for indexing in the future.
Resolved
This incident has been relsolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We're all set! PandaDoc is working properly and the incident has been fully resolved. If you continue to experience issues with this, please contact our Support team (https://support.pandadoc.com/en/articles/9715061-how-to-contact-pandadoc-support).
Resolved
Between 9:11 UTC and 9:42 UTC on December 18th, some customers in the US regions may have experienced issues accessing the UI. We have investigated and resolved this service interruption and impacted services have returned to normal operations.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Our team has identified the issue in Schedule API between 15:30 UTC and 17:00 UTC. We saw performance degradation and 5XX errors in response. Faulty deployment has been reverted quickly in the USA region and rapid recovery is seen. We are monitoring the system for a full recovery right now. The Schedule API is up and running again without any data loss.
Resolved
The purge queue and purge time has been stable since the last provided update.
Resolved
We are marking this issue as resolved because current users can continue to use the Anyscale Platform without any login restrictions. However new account creation will be restricted until early next week.
Resolved
Between 1:06pm ET and 1:50pm ET customers experienced issues with context search. This issue has been resolved and we are posting this retroactive incident.
Resolved
# Timeframe December 9th, 11:08 AM UTC - December 10th, 6:38 AM UTC # Impact A small percentage of image requests served by a single machine in the EU region experienced quality degradation. # Root Cause A hardware failure on one of the rendering machines resulted in images with artifacts being delivered to users. # Contributing Factors An absence of automated testing for faulty renders across active machines caused assets to be cached undetected. The lapse between when the issue was detected and when the issue began culminated in an unknown number of renders that needed to be purged from the cache. Along with inadequate cache tooling for purging affected images, these factors caused the issue to fully resolve in a longer timeframe than expected. # Corrective Actions To prevent this issue from happening in the future, we will: * Investigate and implement automated image quality testing for active rendering machines * Replace and refresh machines on a more frequent basis to minimize impact of single machine failures * Upgrade cache tooling for targeted issue resolution
Resolved
This incident has been resolved.
Resolved
This incident has been resolved. Impacted timeframe: 8:02 PM to 8:22 PM PST
Resolved
This caused numerous Front AI features to be unavailable, including AI Answers, AI Tagging (beta), Suggested Replies(beta), AI Compose, and AI Translate (beta). Additionally, the AI Tagging failures blocked subsequent rules from being run.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
**Dates:** Start Time: Monday, December 9, 2024 at 17:32 UTC End Time: Monday, December 9, 2024 at 20:28 UTC Duration: 2 hours and 56 minutes **What happened:** For some accounts, data sent to Mezmo Pipelines was slow to be processed and sent onwards to their destination, or was not processed at all for the duration of the incident. **Why it happened:** We released a new version of the agent \(3.10.1\) that improves how logs lines are sent to the Mezmo service for ingestion. Most applications add newly written log lines by a process known as “appending”; by contrast, a small number use a process called “truncation”. The new agent version has improved its ability to handle log lines added to logs using truncation, particularly when log lines are written frequently. Many accounts do not monitor any logs that use truncation. A few accounts do, but they write new log lines infrequently. However, a handful of customer accounts have applications that write log lines using truncation at a very high frequency. When these accounts upgraded to agent 3.10.1, there was a very large increase in the volume of data sent to the Mezmo Pipeline service. The increase was detected by our monitoring. It also caused some pods in some Vector partitions to crash, which affected all accounts on the same partition. Data was still ingested and cached within our service, but it was not being quickly processed or sent on to destinations for all accounts. **How we fixed it:** We temporarily paused the processing of newly ingested data, which stopped pods from crashing. We then identified the accounts \(approximately 20\) that were sending us an increased volume of data and moved them to a newly created Vector partition on a newly commissioned node. This allowed the accounts on all other Vector partitions to function normally again; we restarted Pipeline processing on their pods and data began to flow to destinations again. For the remaining affected accounts, we applied exclusion rules within our service to remove redundant data and thereby reduce the overall volume of data. We also contacted the owners of the handful of accounts sending log lines at very high volumes and helped them apply exclusion rules within their locally running Mezmo agents. With these changes, the overall volume of ingested data was reduced and again able to be processed and sent to destinations. After monitoring these accounts and seeing no ill effects, we moved them back to the general pool of Vector partitions. **What we are doing to prevent it from happening again:** We discovered our Vector partitions are configured to use more CPU cores than necessary; under high load, the increased CPU usage causes pods to crash. Accordingly, we will limit the number of cores available to Vector per node. We will rebalance the number of accounts assigned to Vector partitions, aiming to assign less accounts to each one. This will reduce the impact of any similar incidents in the future. We will explore how to rate limit data sent to Pipelines by individual agents in the future. Rate-limiting will prevent any impact in a similar incident in the future.
Resolved
We’ve identified the root cause, deployed, and validated the fix has worked. We’re closing this incident. Thanks so much for bearing with us while we investigated and resolved this! We’re sorry for the trouble as well. If you don’t think the incident is resolved, or you’re still seeing problems on your end please raise a ticket through our Help Center. Thanks!
Resolved
This incident has been resolved.
Resolved
Retroactive update - Dash Answers was unavailable for some users beginning around 11:30pm UTC on December 11th. This was due to an OpenAI outage. We are back up and running!
Resolved
This incident has been resolved.
Resolved
The issue is now resolved. The data in the Protect360 Post Attribution and Organic In-apps reports is back to normal.
Resolved
A fix was released for the Card Repeater Power-up, and the repeats and new ones are working as intended.
Resolved
The issue is resolved: Yesterday's and today's LTV data is available in our overview dashboard and pull API aggregated reports.
Resolved
This incident has been resolved.
Resolved