Increased message delays in US and EU due to DNS resolution errors

Event timeline

How this incident unfolded, from the official status page.

1. Identified

   12 days ago

   SparkPost

   We are investigating an increase in delayed (deferred) messages in both our US and EU regions, beginning at approximately 14:40 UTC on October 11. A subset of our internal DNS resolvers are intermittently failing to resolve recipient domains, causing some delivery attempts to be temporarily deferred. Affected messages are not being dropped or bounced. Retries are succeeding as they are routed to healthy resolvers, so impacted messages are being delivered with additional latency. Customers may observe an elevated number of delay events in webhooks, event logs, and reporting during this time. We have identified the cause as a DNSSEC validation issue on a portion of our resolver fleet related to a scheduled update of the internet's root DNS signing key, and are applying a fix. No action is required from customers. We will provide an update within 60 minutes or as soon as we have more information.

2. Identified

   12 days ago

   SparkPost

   We are continuing to rollout the new root signing keys from ICANN. We continue to see recovery across our global clusters.

3. Identified

   12 days ago

   SparkPost

   We are continuing to work on a fix for this issue.

4. Monitoring

   12 days ago

   SparkPost

   A fix has been implemented and we are monitoring the results.

5. Resolved

   12 days ago

   SparkPost

   This incident has been resolved.