Increased processing pipeline latency

Resolved

Started

Lasted 7h 50m

View official incident ↗

Updates

  1. Resolved

    This incident was resolved. Rollbar processing pipeline latency (measured at p99) was elevated above our 30-second p99 SLO today during the following windows: 02:02AM - 02:35AM PDT: 30s - 7.0 minutes latency 06:02AM - 07:03AM PDT: 30s - 3.9 minutes latency 08:00AM - 09:02AM PDT: 30s - 37.2 minutes latency 09:02AM - 09:52AM PDT: 30s - 23.1 minutes latency p50 stayed at SLO (~5s) during the entire time period, and no data was lost. The initial cause analysis is a particular traffic pattern. We're investigating the root cause to ensure the system is resilient to this traffic pattern going forward.

  2. Investigating

    Rollbar processing pipeline latency (measured at p99) was elevated above our 30-second p99 SLO today beginning at 02:02AM PDT.