Updates
Resolved
The topics fully drained and everything is back to baseline and operational again. No data was lost.
Monitoring
The fix is deployed and the partition is now draining.
Identified
We have a fix prepared but due to an github outage https://www.githubstatus.com/ we are waiting for actions to run to merge and resolve the issue.
Identified
The problem was a poison pill message clogging the queue on one partition. Invocations on that particular partition see some delay whilst the rest of the system is completely healthy. No data was lost and delays on that partition is the only impact. We prepared a PR and are merging the fix.
Investigating
We’re investigating issues with realtime destinations. Some events may be delayed.