Resolved incident

Delayed Notifications (DataDog, Slack, Webhooks, Pagerduty, etc)

Runscope · Minor · Started

Duration
1h 11m
Severity
Minor
Status
Resolved

Summary

Minor · Resolved

DataDog ingestion performance issues have been resolved. Follow the DataDog status updates for more information https://status.datadoghq.com/

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Identified

    Runscope

    We have identified an issue with the DataDog ingestion API. (See: https://status.datadoghq.com/ ) This is causing events sent to DataDog to time out and slowing down our notification queuing system. We are monitoring the delay and will deploy workarounds as needed. Customers using the DataDog integration may see data loss.

  2. Monitoring

    Runscope

    We have implemented a workaround to prevent slow DataDog API requests from delaying other notifications. We will continue to monitor until the DataDog performance issue is resolved.

  3. Resolved

    Runscope

    DataDog ingestion performance issues have been resolved. Follow the DataDog status updates for more information https://status.datadoghq.com/