Resolved incident

Delays in SLO, Service Maps processing

Honeycomb · Major · Started

Duration
25h 24m
Severity
Major
Status
Resolved

Summary

Major · Resolved

This incident has been resolved. At no point did we lose any customer data that hit our load balancers. Querying has been stable for hours, and all features that were degraded are functional.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Monitoring

    Honeycomb

    Due to constrained EC2 instance capacity in the AWS us-east-1 region, we are choosing to allocate the capacity we have to incoming events and telemetry. As such, customers may see over 5 minute delays in our processing of - SLOs - Service Maps We do not expect a degradation of our core ingest -> query flow, and we do not expect triggers to be impacted

  2. Monitoring

    Honeycomb

    SLO processing has recovered. Service Maps continues to be degraded but historical data can be queried.

  3. Monitoring

    Honeycomb

    The networking issues in us-east-1 is affecting our query engine - customers may see errors or delays when running queries as a result.

  4. Monitoring

    Honeycomb

    We’re currently observing that querying is seeing signs of recovery. We are continuing to monitor the situation

  5. Monitoring

    Honeycomb

    AWS is starting to show signs of recovery. Querying remains partially impacted and may take longer to return results. We are continuing to monitor the situation.

  6. Monitoring

    Honeycomb

    Service maps are up and running and errors have resolved. Querying, triggers and SLOs are continuing to improve and customers may see delays as functionality continues to improve. We are continuing to monitor the situation.

  7. Monitoring

    Honeycomb

    Honeycomb core functionality is operational. Service maps are up and running and errors have resolved. Querying, triggers and SLOs are continuing to improve and customers may see delays as functionality continues to improve. This is our final update for the night unless the situation degrades. We will continue to monitor the situation.

  8. Resolved

    Honeycomb

    This incident has been resolved. At no point did we lose any customer data that hit our load balancers. Querying has been stable for hours, and all features that were degraded are functional.