Increased error rate sending and receiving messages on email-based channels

Resolved

Major impact

Started

Lasted 6h 26m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Monitoring

    Customers in us-west-2 and eu-west-1 regions have fully recovered. We are still working through the incoming message backlog on us-west-1.

  3. Monitoring

    The fix has been deployed and we are seeing email delivery services have recovered. We're monitoring and catching up on backlog so there might be residual delays.

  4. Identified

    We have validated the fix and are in the process of doing a manual deploy to circumvent the current Docker outage.

  5. Identified

    We are validating the fix in our staging environment.

  6. Identified

    We have identified the cause of the issue and are working on a fix. The issue affects some but not all channels.

  7. Investigating

    We are continuing to investigate this issue.

  8. Investigating

    We are currently investigating this issue.