Resolved incident

FME - Some customers are experiencing delays in scheduled exports of impressions

Harness · Minor · Started

Duration
24h 23m
Severity
Minor
Status
Postmortem

Summary

Minor · Postmortem

**Incident Summary** Harness FME \(Feature Management & Experimentation\) faced substantial delays in scheduled impressions data exports on Jul 2, 2026. This issue stemmed from performance degradation within Tinybird's infrastructure, affecting query execution crucial to our export pipeline. **Root Cause** The core issue was a temporary **performance degradation on the provider's infrastructure**. This directly impacted the metadata retrieval endpoint, which failed to return job IDs intermittently. Consequently, completed export jobs underwent repeated retries, leading to a backlog in the export queue. **Impact** * The export backlog accumulated, peaking at more than 6 hours behind the planned schedule. * No data loss was detected during the incident. ‌ **Mitigation** 1. **Adjusted Temporal workflows:** Extended timeouts from ~30 to ~50 minutes and increased polling attempts from ~180 to ~300. 2. **Expanded resource capacity.** 3. **Traffic redirection:** Systematically shifted export workloads over to an optimized in-house cluster. 4. **Manual intervention:** Terminated stalling workflows stuck in retry loops from failed metadata lookups. ‌ **Next Steps** Complete the migration of all remaining export jobs to the internal analytics cluster \(In progress\).

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Monitoring

    Harness

    A fix has been implemented and we are monitoring the results.

  2. Monitoring

    Harness

    We are seeing the backlogged items worked through and continuing to monitor results.

  3. Monitoring

    Harness

    We are continuing to monitor results while the backlog is worked through.

  4. Resolved

    Harness

    This incident has been resolved.

  5. Postmortem

    Harness

    **Incident Summary** Harness FME \(Feature Management & Experimentation\) faced substantial delays in scheduled impressions data exports on Jul 2, 2026. This issue stemmed from performance degradation within Tinybird's infrastructure, affecting query execution crucial to our export pipeline. **Root Cause** The core issue was a temporary **performance degradation on the provider's infrastructure**. This directly impacted the metadata retrieval endpoint, which failed to return job IDs intermittently. Consequently, completed export jobs underwent repeated retries, leading to a backlog in the export queue. **Impact** * The export backlog accumulated, peaking at more than 6 hours behind the planned schedule. * No data loss was detected during the incident. ‌ **Mitigation** 1. **Adjusted Temporal workflows:** Extended timeouts from ~30 to ~50 minutes and increased polling attempts from ~180 to ~300. 2. **Expanded resource capacity.** 3. **Traffic redirection:** Systematically shifted export workloads over to an optimized in-house cluster. 4. **Manual intervention:** Terminated stalling workflows stuck in retry loops from failed metadata lookups. ‌ **Next Steps** Complete the migration of all remaining export jobs to the internal analytics cluster \(In progress\).