Updates
Postmortem
The postmortem is posted here - [https://blog.mailchannels.com/mailchannels-service-disruption-all-the-gory-details/](https://blog.mailchannels.com/mailchannels-service-disruption-all-the-gory-details/)
Resolved
All of the metrics, mainly SMTP connection delay and SMTP round trip delivery time, continue to be normal. If you are still encountering any issues, please reach out to support@mailchannels.com. We will be updating the port-mortem here shortly.
Monitoring
All of our metrics, mainly SMTP connection delay and SMTP round trip delivery time, have returned to normal as of 18:00 UTC. This means that if there are any emails backed up in the server queues, they should start delivering. Customers may continue to receive complaints from users about delivery delays because messages may still be leaving the queue as per the exponential back-off rules of the SMTP servers. However, new messages introduced into the service will be delivered immediately. One option is to flush the queues. If you are still encountering any issues, please reach out to support@mailchannels.com
Investigating
The root cause has been identified as a failure of certain auto-scaling functionality within the service that failed to adapt service scale to a surge of incoming email. We will provide a more detailed analysis when this incident is fully resolved.
Investigating
Legitimate emails backed up in our queues have been delivered. We are still seeing increased latency at the ingress side. We are continuing to investigate.
Investigating
Message delivery is slow for a small percentage of traffic. We are trying to clear out all the messages in our queues as fast as possible.
Investigating
We are continuing to investigate the issue.
Investigating
We are continuing to investigate the issue.
Investigating
Severity Level: 3 We are currently investigating higher than normal connection latency that might have an impact on email delivery services.