Increased error rate sending and receiving messages on email-based channels

Event timeline

How this incident unfolded, from the official status page.

1. Investigating

   about 2 years ago

   Front

   We are currently investigating this issue.

2. Investigating

   about 2 years ago

   Front

   We are continuing to investigate this issue.

3. Identified

   about 2 years ago

   Front

   We have identified the cause of the issue and are working on a fix. The issue affects some but not all channels.

4. Identified

   about 2 years ago

   Front

   We are validating the fix in our staging environment.

5. Identified

   about 2 years ago

   Front

   We have validated the fix and are in the process of doing a manual deploy to circumvent the current Docker outage.

6. Monitoring

   about 2 years ago

   Front

   The fix has been deployed and we are seeing email delivery services have recovered. We're monitoring and catching up on backlog so there might be residual delays.

7. Monitoring

   about 2 years ago

   Front

   Customers in us-west-2 and eu-west-1 regions have fully recovered. We are still working through the incoming message backlog on us-west-1.

8. Resolved

   about 2 years ago

   Front

   This incident has been resolved.