Resolved incident

Delays in sending and receiving Gmail messages [gmail]

Front · Major · Started

Duration
3h 47m
Severity
Major
Status
Resolved

Summary

Major · Resolved

All channels should be back to normal now.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    Front

    We're observing problems sending and receiving some messages on Gmail channels in the us-west-2 region. We're working on identifying the root cause.

  2. Investigating

    Front

    We are continuing to investigate this issue, which seems to be related to Gmail's rate limiting mechanisms. We're seeing some impact across multiple regions.

  3. Investigating

    Front

    We are continuing to see message delays, and are rescaling our internal queues globally to help mitigate the issue.

  4. Investigating

    Front

    We are beginning to see a gradual increase in successful deliveries. We are continuing to monitor the situation, as well as replaying deliveries that have been delayed by the original incident.

  5. Identified

    Front

    We are on the recovery path now and are focused on draining the backlog of delayed messages.

  6. Monitoring

    Front

    We are continuing to drain the backlog and will be doing so for the next couple hours.

  7. Monitoring

    Front

    Continuing to drain the backlog

  8. Monitoring

    Front

    Some channels are still experiencing delays as we import the backlog of messages received while Gmail was experiencing an outage

  9. Resolved

    Front

    All channels should be back to normal now.