Delays in sending and receiving Gmail messages [gmail]

Resolved

Major impact

Started

Lasted 3h 47m

View official incident ↗

Updates

  1. Resolved

    All channels should be back to normal now.

  2. Monitoring

    Some channels are still experiencing delays as we import the backlog of messages received while Gmail was experiencing an outage

  3. Monitoring

    Continuing to drain the backlog

  4. Monitoring

    We are continuing to drain the backlog and will be doing so for the next couple hours.

  5. Identified

    We are on the recovery path now and are focused on draining the backlog of delayed messages.

  6. Investigating

    We are beginning to see a gradual increase in successful deliveries. We are continuing to monitor the situation, as well as replaying deliveries that have been delayed by the original incident.

  7. Investigating

    We are continuing to see message delays, and are rescaling our internal queues globally to help mitigate the issue.

  8. Investigating

    We are continuing to investigate this issue, which seems to be related to Gmail's rate limiting mechanisms. We're seeing some impact across multiple regions.

  9. Investigating

    We're observing problems sending and receiving some messages on Gmail channels in the us-west-2 region. We're working on identifying the root cause.