Delayed updating of escalation status when using PagerDuty

Event timeline

How this incident unfolded, from the official status page.

1. Investigating

   almost 4 years ago

   incident.io

   We're seeing a specific PagerDuty endpoint that we use to fetch escalation log entries fail, which is preventing us from updating escalations with recent data. Creating escalations via `/inc escalate` is still working, and PagerDuty auto-create is unaffected. But the escalation message presented to the channel after `/inc escalate` may show stale data, even while the escalation has been ack'd.

2. Resolved

   almost 4 years ago

   incident.io

   We're no longer seeing errors from the PagerDuty endpoint and have confirmed escalations are polling successfully, meaning escalation messages (created in response to `/inc escalate`) are working once more. We've followed up with PagerDuty to understand what caused this, and improved our retrying logic to make us more robust to future flakiness.