Resolved incident

Ingest delays

Scout APM · Major · Started

Duration
36h 32m
Severity
Major
Status
Resolved

Summary

Major · Resolved

This incident has been resolved.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    Scout APM

    We are currently investigating this issue.

  2. Monitoring

    Scout APM

    A fix has been implemented and we are monitoring the results.

  3. Identified

    Scout APM

    The initial fix was unsuccessful, certain accounts are now substantially delayed.

  4. Monitoring

    Scout APM

    An alternate approach has been applied, we are watching.

  5. Identified

    Scout APM

    It has been a long day with kafka. We continue to experience instability, causing lag and dropped payloads.

  6. Identified

    Scout APM

    Throughput has improved although behavior of individual partitions remains a problem and is still causing delays in some cases.

  7. Identified

    Scout APM

    We are not to full resolution yet.

  8. Monitoring

    Scout APM

    Zookeeper corruption has been rooted out. Things appear healthier and catching up in all cases.

  9. Resolved

    Scout APM

    This incident has been resolved.