Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Oct 12, 2020

  1. RunscopeInitial Scripts Broken

    There was a 7-minute period around 18:05 UTC where initial scripts defined in an environment were not being processed correctly causing some tests to fail. This was caused by a failed deploy that was rolled back as quickly as possible, resolving the issue. No further issues are expected.

    Resolved

  2. RunscopeIncorrect URL Character Encodings

    We have rolled back a deploy that was causing request URLs in tests to be double encoded or encoded improperly between 22:08 and 22:51 UTC. This caused tests to fail for some customers. We are adding additional precautions to prevent this error from recurring.

    Resolved

  3. RunscopeTest Runs Slow

    All systems have been operating normally since 7:30 PDT when our datastore service was restored.

    Resolved

  4. RunscopeTest Run Delays

    There was a delay in processing test runs for approximately 6 minutes at 11:24am UTC due to an internal queuing system encountering an error. The error was resolved and test runs resumed running normally.

    Resolved

  5. RunscopeTest Run Performance issues.

    New hosts are fully operational and system performance has consistently been at normal levels. We are continuing to monitor for any regressions in performance or reliability.

    Resolved

  6. RunscopeTest Run Delays

    The additional capacity is fully in place and tests are running normally.

    Resolved

  7. RunscopeTest Run Issues

    The capacity improvements have stabilized the system. The issue stemmed from a large influx of traffic which overwhelmed a system in an unexpected way. Our fixes this morning had unintended side effects which led to the second incident. We don’t expect further issues but will update the status page if there are any recurrences or regressions.

    Resolved

  8. RunscopeTest Run Issues

    The system has been stable for over 12 hours. We are continuing work on adding additional performance and stability enhancements, but the main cause of the downtime yesterday has been resolved. If you have any questions, please feel free to contact us at help@runscope.com.

    Resolved

  9. RunscopeFalse PagerDuty Alerts

    **Root cause of the problem** As part of BlazeMeter/Runscope’s business continuity plans, BlazeMeter/Runscope engineering is preparing redundant stage and DR environments in other locations/providers to ensure business and data continuity. As part of this initiative, they were performing simulation testing in their redundant staging environment with redundant test schedules dataset from production backups. But unfortunately, the simulation test missed turning off the integrations associated with those test schedules, hence, the staging schedules simulation triggered failure notifications to customer’s configured alert/notification mechanism associated with it. ‌ **What we are doing to avoid recurrences in future:** 1. We immediately stopped simulation tests from stage to prevent any further false alerts going to customers 2. We revisited our simulation and data redundancy plan and started employing additional data validation, cleansing and sanitization on the staging simulation dataset. 3. Working on employing outbound gateway to control all egress traffic in staging to be able to control these alerts and notification.

    Resolved

Older incidents