Resolved incident

Test Runs Slow

Runscope · Major · Started

Duration
6h 12m
Severity
Major
Status
Resolved

Summary

Major · Resolved

All systems have been operating normally since 7:30 PDT when our datastore service was restored.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    Runscope

    We're investigating a slowdown in processing test runs.

  2. Investigating

    Runscope

    We are investigating a performance issue with our primary datastore for test results. This is causing test runs to slow down and results to not load or load very slowly in the dashboard and API. We are still investigating the full extent of the issue.

  3. Identified

    Runscope

    Our primary service provider has acknowledged an issue with the datastore service causing the issue. They are investigating and working on a resolution. We are working to add work arounds for the performance issue as much as possible. Test run processing is very limited at this time.

  4. Identified

    Runscope

    We are seeing improvements in response times from the test results datastore allowing some test runs to resume. Capacity is still significantly reduced. We are continuing to attempt to mitigate the issue with workarounds and additional capacity while we wait for our service provider to resolve the underlying issue.

  5. Monitoring

    Runscope

    Test run throughput has improved to near normal levels as performance of the datastore service has been restored. We are still monitoring closely for regressions in performance but all queues have caught up for the time being.

  6. Resolved

    Runscope

    All systems have been operating normally since 7:30 PDT when our datastore service was restored.