Quality degradation for a small portion of rendered images in the EU

Event timeline

How this incident unfolded, from the official status page.

1. Postmortem

   almost 2 years ago

   imgix

   # Timeframe December 9th, 11:08 AM UTC - December 10th, 6:38 AM UTC # Impact A small percentage of image requests served by a single machine in the EU region experienced quality degradation. # Root Cause A hardware failure on one of the rendering machines resulted in images with artifacts being delivered to users. # Contributing Factors An absence of automated testing for faulty renders across active machines caused assets to be cached undetected. The lapse between when the issue was detected and when the issue began culminated in an unknown number of renders that needed to be purged from the cache. Along with inadequate cache tooling for purging affected images, these factors caused the issue to fully resolve in a longer timeframe than expected. # Corrective Actions To prevent this issue from happening in the future, we will: * Investigate and implement automated image quality testing for active rendering machines * Replace and refresh machines on a more frequent basis to minimize impact of single machine failures * Upgrade cache tooling for targeted issue resolution

2. Resolved

   almost 2 years ago

   imgix

   A small percentage of image requests served by a single machine in the EU region experienced quality degradation between December 9th, 11:08 AM UTC - December 10th, 6:38 AM UTC