Resolved incident

Delayed Monitors Notifications

Datadog US1 · Critical · Started

Duration
14h 1m
Severity
Critical
Status
Resolved

Summary

Critical · Resolved

This incident has been resolved.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    Datadog US1

    We are investigating delays in Monitors Notifications, which began at 23:39 UTC.

  2. Investigating

    Datadog US1

    We are investigating increased latency across multiple products which began at 23:39 UTC. As a result of this issue, some users may see delays in data across the platform.

  3. Investigating

    Datadog US1

    Due to upstream provider issues, we are also continuing to see unavailability of telemetry data coming from AWS into Datadog. https://health.aws.amazon.com/health/status

  4. Identified

    Datadog US1

    We have identified upstream provider issues and are continuing to experience delays in processing data across multiple products. We are working on a fix. https://health.aws.amazon.com/health/status

  5. Identified

    Datadog US1

    We have identified upstream provider issues and are continuing to experience delays in processing data across multiple products. We are working on a fix. https://health.aws.amazon.com/health/status

  6. Identified

    Datadog US1

    We have identified due to upstream provider issues, we are continuing to see unavailability of telemetry data coming from AWS into Datadog. We are continuing to work on a fix. https://health.aws.amazon.com/health/status

  7. Identified

    Datadog US1

    We have identified due to upstream provider issues, we are continuing to see unavailability of telemetry data coming from AWS into Datadog. We are continuing to work on a fix. https://health.aws.amazon.com/health/status For metrics we are still seeing delays in distribution metrics. Counts, rates and gauge metrics are being processed normally for most customers

  8. Identified

    Datadog US1

    We have identified upstream provider issues and are continuing to experience delays in processing data across multiple products. We are continuing to work on a fix. https://health.aws.amazon.com/health/status For metrics, distribution metrics and point metrics are being processed normally for all customers.

  9. Identified

    Datadog US1

    We have identified upstream provider issues and are continuing to experience delays in processing data across multiple products. We are continuing to work on a fix. https://health.aws.amazon.com/health/status For metrics, distribution metrics and point metrics are being processed normally for all customers.

  10. Identified

    Datadog US1

    We have identified upstream provider issues and are continuing to experience delays in processing data across multiple products. We are continuing to work on a fix. https://health.aws.amazon.com/health/status For metrics, distribution metrics and point metrics are being processed normally for all customers.

  11. Identified

    Datadog US1

    We have identified upstream provider issues. Metrics, Logs, APM and RUM data are being processed normally for all customers. Alerting is also functional. There are still processing delays for CI visibility. We are monitoring the situation. https://health.aws.amazon.com/health/status

  12. Monitoring

    Datadog US1

    We have identified upstream provider issues. Metrics, Logs, APM, RUM and CI visibility data are being processed normally for all customers. Alerting is also functional. We are monitoring the situation. https://health.aws.amazon.com/health/status

  13. Resolved

    Datadog US1

    This incident has been resolved.