CDN configuration sync delays

Resolved

Minor impact

Started

Lasted 28h 28m

View official incident ↗

Updates

  1. Resolved

    The fix has been deployed to all nodes. We will be readding the missing nodes to the pool today.

  2. Monitoring

    A fix is being staged on a subset of affected nodes. Impacted nodes are since identifying the issue isolated from CDN traffic and only receiving test traffic. Further updates will be provided after testing concludes and deployment to all affected nodes is complete.

  3. Identified

    We've been observing 25 to 30 nodes in various PoPs around the world exhibiting constant delays with syncing their configuration since yesterday evening. We've depooled all impacted nodes as a temporary workaround and working on a permanent fix.