Resolved incident

High SMTP Connection Latency

MailChannels · Minor · Started

Duration
13h 54m
Severity
Minor
Status
Postmortem

Summary

Minor · Postmortem

The postmortem is posted here - [https://blog.mailchannels.com/mailchannels-service-disruption-all-the-gory-details/](https://blog.mailchannels.com/mailchannels-service-disruption-all-the-gory-details/)

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    MailChannels

    Severity Level: 3 We are currently investigating higher than normal connection latency that might have an impact on email delivery services.

  2. Investigating

    MailChannels

    We are continuing to investigate the issue.

  3. Investigating

    MailChannels

    We are continuing to investigate the issue.

  4. Investigating

    MailChannels

    Message delivery is slow for a small percentage of traffic. We are trying to clear out all the messages in our queues as fast as possible.

  5. Investigating

    MailChannels

    Legitimate emails backed up in our queues have been delivered. We are still seeing increased latency at the ingress side. We are continuing to investigate.

  6. Investigating

    MailChannels

    The root cause has been identified as a failure of certain auto-scaling functionality within the service that failed to adapt service scale to a surge of incoming email. We will provide a more detailed analysis when this incident is fully resolved.

  7. Monitoring

    MailChannels

    All of our metrics, mainly SMTP connection delay and SMTP round trip delivery time, have returned to normal as of 18:00 UTC. This means that if there are any emails backed up in the server queues, they should start delivering. Customers may continue to receive complaints from users about delivery delays because messages may still be leaving the queue as per the exponential back-off rules of the SMTP servers. However, new messages introduced into the service will be delivered immediately. One option is to flush the queues. If you are still encountering any issues, please reach out to support@mailchannels.com

  8. Resolved

    MailChannels

    All of the metrics, mainly SMTP connection delay and SMTP round trip delivery time, continue to be normal. If you are still encountering any issues, please reach out to support@mailchannels.com. We will be updating the port-mortem here shortly.

  9. Postmortem

    MailChannels

    The postmortem is posted here - [https://blog.mailchannels.com/mailchannels-service-disruption-all-the-gory-details/](https://blog.mailchannels.com/mailchannels-service-disruption-all-the-gory-details/)