For some customers, newly submitted logs are only appearing in the UI after long delays

Resolved

Minor impact

Started

Lasted 110h 34m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Identified

    We have resumed all services and functionality. We are continuing to monitor the situation.

  3. Identified

    As we continue working on this issue, we have temporarily scaled down live-tail and alerting. Some customers may not see live tail and receive new alerts. New logs still appear in the UI with delays.

  4. Identified

    We have resumed live tail and alerting now. Some users may still experience delays with searching and archiving. We are continuing to monitor the situation.

  5. Identified

    As we continue working on this issue, we have temporarily scaled down livetail and alerting. Some customers may not see live tail and receive new alerts. New logs still appear in the UI with delays.

  6. Identified

    We are still experiencing some delays, new logs are continuing to be processed and our engineers are taking steps to mitigate the impact.

  7. Identified

    We are still experiencing some delays, new logs are continuing to be processed. Investigations are ongoing.

  8. Identified

    New logs are still appearing in the UI with delays for some customers. We are continuing to investigate.

  9. Identified

    Delays are still being experienced by some customers. We continue to work towards a solution.

  10. Identified

    For some customers, newly submitted logs are being ingested successfully but are only appearing in our UI after long delays. Our engineers have identified the cause and are working towards a solution.