Queued Test Runs and Requests

Resolved

None impact

Started

Lasted 5h 20m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Monitoring

    We have implemented additional capacity and other improvements to stabilize the system. Performance at this time has been restored to normal levels as of 8:25am PT. We are continuing to investigate the cause of the original issue and the elevated number of expired tests over the past hour or so.

  3. Identified

    We have identified the malfunctioning component and have taken steps to restore functionality. Test run throughput is recovering, though test runs that were queued for over 10 minutes will show as 'Expired' or 'Runscope System Error'. We are continuing to work on fully recovering from the incident.

  4. Investigating

    We are continuing to work on resolving the queuing issues and hope to have an update shortly.

  5. Investigating

    We're looking into an issue causing test runs to be queued for longer-than-normal amounts of time. Will update when we know more.