503 errors in the UserVoice admin console

Postmortem

Major impact

Started

Lasted 5h 33m

View official incident ↗

Updates

  1. Postmortem

    On October 19th between 10:00 and 11:30 PDT UserVoice experienced two approximately 10 minute infrastructure outages that caused site-wide outages and system unavailability. **Business Impact** During the outage end users and admins would have been unable to load or interact with UserVoice sites or widgets. Email would have been delayed, but no emails were lost. **Root Cause** UserVoice uses an in-memory data-store cluster \(Redis\) to handle asynchronous job management and transient data storage.  A recent change to one of the libraries that use this service caused a very sudden increase in its usage. The sudden usage increase caused a system failure and prevented failover to like-sized standby services. **What we are Doing to Prevent This** * Increased sizing of our Redis cluster and added additional alerting to allow us to more quickly detect usage spikes * Fixed the library that wasn’t properly interacting with Redis

  2. Resolved

    This incident has been resolved.

  3. Monitoring

    A fix has been implemented and we are monitoring the results.

  4. Investigating

    We are currently investigating 503 errors on UserVoice web portals and the admin console. This also affects the API and widgets.