Delayed Metrics

Resolved

Started

Lasted 45h 17m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Identified

    Almost all metrics have caught up aside from a small handful in sin and syd. We're continuing to monitor and expect these to complete in the next few hours.

  3. Identified

    Backlogged metrics are still being processed. We're bringing extra processing capacity online to speed up the process.

  4. Identified

    Backlogged metrics are still being processed and ingestion delays persist for some customers

  5. Identified

    Backlogged metrics are still being processed and ingestion delays persist for some customers, but we’re continuing to see gradual improvement as the backlog continues to drain.

  6. Identified

    Backlogged metrics data is still being processed, continue to monitor the situation while the system catches up.

  7. Identified

    Metric ingestion is still delayed for some customers. We’re seeing gradual improvement and continue to monitor the situation while the system catches up.

  8. Identified

    We are still in the process of increasing the metrics cluster throughput to catch up with metric backlog.

  9. Identified

    We are continuing to see delayed metrics due to resource contention on a subset of metrics ingestion hosts, and we are working to rebalance ingestion traffic and reduce the backlog.

  10. Identified

    We are continuing to see delayed metric exports from a number of hosts. Users will see delayed or missing metrics for machines on impacted hosts at this time.

  11. Identified

    We've identified an issue on multiple hosts causing delayed metric ingestion into our hosted fly-metrics.net dashboards. We are working on a fix.

  12. Investigating

    We are investigating issues with customer facing metrics in the fly-metrics.net dashboard. Users may see delayed or missing metrics at this time.