Function processing delays

Resolved

Started

Lasted 1h 7m

View official incident ↗

Updates

  1. Resolved

    The incident is now resolved and the system is full operational. Delays began around 14:03 UTC caused by an issue publishing to a new Kafka topic. Recovery began around 14:40 UTC as the team was able to isolate this topic. The consuming services which schedules new function runs was scaled out and began to consume the backlog. No events were dropped during this incident, all functions were scheduled, but with delays during this time window. No action is needed for manual recovery.

  2. Monitoring

    New function execution has resumed and nearly caught up from the incurred backlog. Wait for event and cancel on handlers are still in a backlog and are catching up with processing. No events were dropped during this issue, all functions will be executed, but with delays. The root cause has been confirmd and changes are underway to prevent reoccurrence.

  3. Identified

    We have identified the cause of the issue - the team has deployed a fix and is scaling to catch up on function delays.

  4. Investigating

    We are actively investigating an issue with function execution that is leading to delays starting function runs after events are received. We will provider further updates as we identify the cause and resolve the issue.