Resolved incident

Images in Whitehall intermittently rendering

GOV.UK · None · Started

Duration
140h 36m
Severity
None
Status
Resolved

Summary

None · Resolved

We have pushed up a fix to draft-frontend and are declaring the incident resolved.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    GOV.UK

    We are aware of intermittent issues with the rendering of images, both in Whitehall's Admin UI and in the draft 'preview'. We believe this affects drafts only, and that there are no issues viewing images once the document is published. Meanwhile, we are investigating the underlying cause as a matter of priority.

  2. Identified

    GOV.UK

    We've identified the root cause of the issue: draft documents with more than one image can fail to pass authentication when requesting those images from Asset Manager, due to the random order in which the authentication 'callbacks' can happen. The issue does resolve itself when the document is published (the images become live and no authentication is required). We're working on a fix. In the meantime, a workaround for publishers trying to preview images is to load an asset in a new tab in their browser (e.g. https://draft-assets.publishing.service.gov.uk/media/5e59279b86650c53b2cefbfe/placeholder.jpg). This reliably sets up the session cookie, and subsequent reloads of the draft document should now load images as normal.

  3. Identified

    GOV.UK

    We've shipped a change to Whitehall to minimise the impact of this issue. Publishers using Whitehall normally should now be able to edit documents and see the images in the Images tab - and on the document preview - with no issues. The intermittent image-rendering continues to be a problem when clicking on a preview URL (for example, a URL shared by a colleague) and not navigating to it directly from Whitehall. We've identified a fix for that and hope to have an update tomorrow.

  4. Identified

    GOV.UK

    We've successfully tested the fix for draft-frontend and are finalising its implementation. We'll provide another update tomorrow.

  5. Monitoring

    GOV.UK

    Reports of users struggling to load images have dried up since Tuesday. We believe the Whitehall fix has largely addressed the issue for most users. We have a more permanent fix for Frontend almost ready to go, but will defer deploying it until Monday so that its rollout can be properly monitored.

  6. Resolved

    GOV.UK

    We have pushed up a fix to draft-frontend and are declaring the incident resolved.