Slow and expired test runs

Event timeline

How this incident unfolded, from the official status page.

1. Investigating

   almost 9 years ago

   Runscope

   We’re investigating an issue with slow and expiring test runs. We will provide more details as they become available.

2. Investigating

   almost 9 years ago

   Runscope

   We are continuing to investigate the cause of the delayed and expired test runs. We have identified the affected subsystem, but are continuing work to identify the root cause of the issue.

3. Investigating

   almost 9 years ago

   Runscope

   We have deployed some an update to the system to attempt to mitigate the impact of the issue. You may see some improvement, though it may be temporary while we continue to work on a permanent resolution. We currently do not have an ETA for resolution, but we are working to resolve this incident as quickly as possible.

4. Investigating

   almost 9 years ago

   Runscope

   Test run throughput has improved significantly, but volatility remains as we continue work to mitigate the original issue and the ensuing side effects. You will likely see 'Expired' test runs as we continue to process the backlog of work that was accumulated during the initial slowdown.

5. Investigating

   almost 9 years ago

   Runscope

   We are continuing to implement additional measures to attempt to return test run throughput to full capacity. We are seeing marginal ongoing improvements while we try to resolve the issue completely.

6. Monitoring

   almost 9 years ago

   Runscope

   We've deployed some additional measures that have restored test run throughput to near normal levels for newly-initiated test runs. You will see an increased number of 'Expired' test runs as the workers catch up. We are continuing to work on overall stability while monitoring for any regressions or additional hot spots.

7. Resolved

   almost 9 years ago

   Runscope

   All systems are operating normally.