Increased Rate of LLM Failure Errors on Agent Platform Calls

Resolved

Started

Lasted 92h 46m

View official incident ↗

Updates

  1. Resolved

    Remediation measures have been fully rolled out, restoring failure rates to baseline. The root cause has been addressed, and the issue is now fully resolved.

  2. Monitoring

    Remediation measures have rolled out and instances of failures have returned to baseline. We are continuing to investigate further measures that may be necessary to completely mitigate the core issue.

  3. Identified

    Some Agent Platform conversations utilizing RAG are currently affected by RAG latency increases, in some cases resulting in slower RAG performance or intermittent call failures. Our team is currently working to resolve this.

  4. Monitoring

    We have identified the direct cause of the RAG failures, and are monitoring performance. At the moment, instances of call failures have returned to baseline. We are investigating further measures that may be necessary to completely mitigate the core issue.

  5. Investigating

    Some Agent Platform conversations utilizing RAG are currently affected by RAG latency increases, in some cases resulting in slower RAG performance or intermittent call failures. Our team is currently working to resolve this.