Delayed Notifications (DataDog, Slack, Webhooks, Pagerduty, etc)

Resolved

Minor impact

Started

Lasted 1h 11m

View official incident ↗

Updates

  1. Resolved

    DataDog ingestion performance issues have been resolved. Follow the DataDog status updates for more information https://status.datadoghq.com/

  2. Monitoring

    We have implemented a workaround to prevent slow DataDog API requests from delaying other notifications. We will continue to monitor until the DataDog performance issue is resolved.

  3. Identified

    We have identified an issue with the DataDog ingestion API. (See: https://status.datadoghq.com/ ) This is causing events sent to DataDog to time out and slowing down our notification queuing system. We are monitoring the delay and will deploy workarounds as needed. Customers using the DataDog integration may see data loss.