Past incident

Delays in build logs appearing in the UI

Netlify · Minor · Started

Duration
Unavailable
Severity
Minor
Status
No longer listed as active

Summary

Minor · No longer listed as active

We have completed processing the Build Logs backlog. All customers should now have a consistent experience when viewing both historical and new Build Logs in the UI. We believe customer impact has ended. We will continue to closely monitor the situation for any signs of recurrence.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    Netlify

    We are investigating an issue causing delays in build logs appearing in the Netlify UI. Our team is actively investigating and will provide updates as we learn more.

  2. Investigating

    Netlify

    We are continuing to investigate delays in build logs appearing in the Netlify UI. Our team is working to identify the cause and will provide further updates as information becomes available.

  3. Investigating

    Netlify

    We are continuing to investigate delays in build logs appearing in the Netlify UI. We are still actively investigating and will provide further updates as information becomes available.

  4. Investigating

    Netlify

    We are continuing to investigate the delays affecting build log availability in the Netlify UI. We will share further updates as more information becomes available.

  5. Investigating

    Netlify

    We have identified an ongoing issue with an upstream service provider that may be contributing to delays in build logs appearing in the Netlify UI. We are in contact with the provider and are continuing to monitor the situation. We will provide further updates as more information becomes available.

  6. Investigating

    Netlify

    We continue to experience delays in build logs appearing in the Netlify UI. We remain in contact with our upstream service provider as the investigation continues. We will provide further updates as more information becomes available.

  7. Identified

    Netlify

    We've identified the cause of the delays in build logs appearing in the Netlify UI as an issue with an upstream service provider, who is working on a fix

  8. Identified

    Netlify

    We continue to work with our upstream service provider on the delays in build logs appearing in the Netlify UI.

  9. Monitoring

    Netlify

    The upstream provider applied a fix at 3:30 UTC and we are monitoring system health. We are able to ingest build logs at full throughput but we have hours of backlog that we need to process.

  10. Monitoring

    Netlify

    We are still monitoring the build log ingestion backlog. We will post our next update at 7:30 UTC, or sooner if anything changes.

  11. Monitoring

    Netlify

    Build log processing continues at full throttle and we estimate backlog to be cleared by 9:00 UTC. We’ll post next update at 8:45 UTC.

  12. Monitoring

    Netlify

    As of 08:45 UTC, we are still processing the backlog of build logs at full throttle. Some partitions have completed backlog processing, but others still have backlog pending. As a result of this partial processing, customers may experience intermittent availability until the full backlog is processed. Some build logs may be available, and others not. Real-time build logs should be working as expected. The new ETA for backlog processing is 10:00 UTC. We will provide our next update at 10:00 UTC.

  13. Monitoring

    Netlify

    We have completed processing the Build Logs backlog. All customers should now have a consistent experience when viewing both historical and new Build Logs in the UI. We believe customer impact has ended. We will continue to closely monitor the situation for any signs of recurrence.