Ingest issues for some customers

Event timeline

How this incident unfolded, from the official status page.

1. Identified

   7 months ago

   Honeycomb

   We have confirmed reports of some teams getting issues ingesting their data. We have identified a probable source behind this behavior and are currently trying to correct it.

2. Identified

   7 months ago

   Honeycomb

   We have further narrowed down that only teams on Classic environments are currently seeing issues. We are figuring out mitigation mechanisms at the moment.

3. Identified

   7 months ago

   Honeycomb

   We've identified a manual fix and are deploying that by hand to each affected environment, while also investigating a systemic solution.

4. Monitoring

   7 months ago

   Honeycomb

   We've applied the manual fix to nearly all affected datasets (less than 1% remaining). The systemic solution is still in progress.

5. Monitoring

   7 months ago

   Honeycomb

   The manual fix has been applied to all environments.

6. Monitoring

   7 months ago

   Honeycomb

   We are continuing to work on the systemic solution and monitor. We'll leave the incident open until we're able to confirm the issue won't recur.

7. Resolved

   7 months ago

   Honeycomb

   We can confirm that the fix was applied uniformly to all impacted and potentially impacted environments; no teams should see ingest errors related to this issue anymore.