Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Apr 16, 2019

  1. HoneybadgerSourcemap upload issues

    We have determined that some of the sourcemap traffic was being incorrectly routed to a staging server, which was encountering errors. This routing problem has been fixed.

    Resolved

Apr 2, 2019

Mar 28, 2019

Mar 21, 2019

Mar 10, 2019

  1. CodaPeriodic automations not firing

    We've resolved the issue. Periodic automations should start executing now and catch up shortly. Sorry for the trouble.

    Resolved

Feb 26, 2019

Feb 6, 2019

  1. StreamElevated API Errors on US-EAST

    We were experiencing an elevated level of API errors on our us-east region. This incident lasted from 2:11PM UTC to 2:18PM.

    Resolved

Feb 2, 2019

Jan 31, 2019

Jan 30, 2019

Jan 15, 2019

  1. CodaCDN under maintenance

    New CDN is accessible worldwide and site should be fully operational for most users until propagation completes.

    Resolved

Jan 13, 2019

Jan 9, 2019

Jan 2, 2019

Jan 1, 2019

Dec 10, 2018

  1. Webhooks.ioWebhooks.io Outage

    This incident has been resolved. We have monitored traffic and all communication seems to be running stable for the last 4 hours.

    Resolved

Dec 7, 2018

Nov 22, 2018

  1. StreamEU-WEST API downtime

    Due to an operation mistake, API service between 12:56PM UTC and 12:58PM UTC API had very high error rate on the Europe West region. The problem is mitigated and resolved. Detailed API error rate over time: 12:56PM 78% 12:57PM 93% 12:58PM 4%

    Resolved

Nov 16, 2018

  1. UserVoice503 errors

    On November 13th between 13:38 and 14:14 PT, UserVoice experienced a networking infrastructure issue that caused a sitewide outage and system unavailability. **Business Impact** * During the outage end users and admins would have been unable to load or interact with UserVoice sites, widgets or the API. * Email would have been delayed, but no emails were lost. **Root Cause** In the process of cleaning up unused resources in the UserVoice infrastructure an old kubernetes cluster was removed from production. The automated cleanup of this cluster unintentionally removed a networking firewall rule that allowed our active application cluster to communicate with our backend infrastructure. Initial debugging was incorrectly focused around in-cluster symptoms and we did not immediately determine proper cause of the issue. Manual restoration of a proper firewall rule allowed the service to be fully restored. **What we are Doing to Prevent This** * Proper failover firewall rules are now being controlled via our infrastructure-as-code system preventing automated cleanup of old rules. * Infrastructure cleanup tasks will be scheduled during maintenance windows going forward. We didn’t meet our own or your expectations for using UserVoice with this outage. We do apologize for the pain points this caused for you and your team. If you have any questions or concerns, please reach out and let me know. Claire Talbott Support Manager claire.talbott@uservoice.com

    Resolved

Nov 8, 2018

Nov 2, 2018

  1. UserVoiceDatabase timeouts

    On Friday 11/2/18 from 5AM to 5:13AM PDT, UserVoice experienced downtime. ## Business Impact During the time of the incidents end users and admins would have seen 500 errors. They wouldn’t have been able to load the admin console, use the API, interact with ideas on the front end or use the widget or Contributor Sidebar. Email would have been delayed, but no emails were lost. ## Root Cause We saw an issue similar to last Friday’s incident where one of the servers in Uservoice's database cluster experienced an application stall event.  This caused a pause in database writes. Our engineering team manually removed the affected node to allow the cluster to resume operation. ## What we are Doing to Prevent This Our team has been focused, since last week, on finding the root issue that is caused one of our database clusters to stall. This work is still in progress. Once the root issue is identified, we will be implementing a fix and updating this report with the information discovered. In the meantime, we have put increased alerting in place so that should the issue repeat, we will identify it immediately. We understand UserVoice being down is an interruption to you and your team, and impacts your workflows! We take this downtime seriously, and are all hands on deck to get this issue fully addressed, so we can prevent it happening again. If you have any questions or feedback for us about the incident please don’t hesitate to contact me at [claire.talbott@uservoice.com](mailto:claire.talbott@uservoice.com). Claire Talbott Support Manager

    Resolved

Oct 31, 2018

  1. UserVoiceErrors in admin console and web portals

    On October 26th between 10:20 and 10:35 PDT UserVoice experienced an infrastructure issue that caused intermittent system unavailability. ## Business Impact During the outage end-users and admins may have been unable to load or interact with UserVoice sites or widgets. ## Root Cause One of the servers in Uservoice's database cluster experienced an application stall event.  Due to misconfiguration of our cluster, this caused a pause in database writes. Our engineers were required to manually remove the affected node to allow the cluster to resume writing. ## What we are Doing to Prevent This We have updated our database cluster configuration to more aggressively monitor and remove cluster members that are non-performant. This caused an interruption for your team in UserVoice and your end users trying to view and submit feedback. We are sorry for the pain point this caused. We have already put improvements in place to prevent this type of issue from happening again. If you have any questions, don’t hesitate to reach out at [claire.talbott@uservoice.com](mailto:claire.talbott@uservoice.com). Claire Talbott Support Manager

    Resolved

Oct 23, 2018

  1. UserVoice503 errors in the UserVoice admin console

    On October 19th between 10:00 and 11:30 PDT UserVoice experienced two approximately 10 minute infrastructure outages that caused site-wide outages and system unavailability. **Business Impact** During the outage end users and admins would have been unable to load or interact with UserVoice sites or widgets. Email would have been delayed, but no emails were lost. **Root Cause** UserVoice uses an in-memory data-store cluster \(Redis\) to handle asynchronous job management and transient data storage.  A recent change to one of the libraries that use this service caused a very sudden increase in its usage. The sudden usage increase caused a system failure and prevented failover to like-sized standby services. **What we are Doing to Prevent This** * Increased sizing of our Redis cluster and added additional alerting to allow us to more quickly detect usage spikes * Fixed the library that wasn’t properly interacting with Redis

    Resolved

Sep 28, 2018

Sep 21, 2018

Sep 17, 2018

  1. CourseraSome forums not loading

    at 3:06pm EST this issue has been resolved. Some learners may have experienced issues when searching the catalog in addition to forums not loading periodically.

    Resolved

Aug 23, 2018

  1. StreamPartial API outage

    Between 03:59PM and 04:32PM UTC API traffic resulted in HTTP errors or timeouts. Only a part of Stream applications hosted on US were affected by this problem.

    Resolved

Aug 8, 2018

  1. StreamRealtime Redis Failover

    Our distributed realtime cluster uses Redis (on Elasticache) for state management. A failover of the Elasticache cluster caused realtime to be unavailable for 7 minutes. This issue has been resolved and we're investigating why the failover took 7 minutes. This impacted customers using Stream's websocket, SQS or Webhook firehose systems.

    Resolved

Jul 30, 2018

  1. StreamHigh latency spike and increased error rate

    ### January 23 and 24 outage postmortem report Stream suffered two incidents of degraded performance in the past 24 hours. We take uptime very seriously, and would like to be transparent with our operations and to our customers. The spikes occurred on Jan 23 at 3:50PM UTC and on Jan 24 at 11:45AM UTC. Both spikes were caused by a sudden increase of pressure to one of our PostgreSQL databases. Because Postgresql was slow at serving queries, HTTP requests started to pile up and eventually saturated the API workers' connection backlogs. API clients using a very low timeout will have encountered timeout exceptions. Other users of Stream would see 5xx responses on part of their API calls. I am going to add a little bit of background so that it is easier to elaborate on what went wrong. Some of our internal operations rely on moving data from one PostgreSQL database to another. Thanks to `psql` such operation is routinely performed by pipe-ing `COPY TO STDOUT` and `COPY FROM STDOUT` together. In order to not pressure the destination database with writes: we also use `pv` so that we are sure we never end up consuming all our IOPS capacity. The command looks more or less like this: ```psql src_db -c '\copy (...) to stdout' | pv -p --rate-limit 5242880 | psql dst_db -c '\copy (...) from stdout'``` By terminating the same copy command running on the source database we were able to remove **write** pressure on the disk. After that the high latency problem affecting the API service was automatically resolved. After researching on other possible causes, we concluded that the pressure created by the copy command combined with increased traffic was behind this outage and picked a different time to do it again. The same operation was then restarted during low traffic hours. To our surprise write pressure increased again after a couple of hours on the source database and caused another, albeit shorter, outage. After more digging we realized that in both occasions the command was running in the background and could not write to stdout, forcing the source database to store the query results on disk and then causing very high I/O on disk and slow response times to regular traffic. The remediation for this is very straight forward and it simply requires to never block stdout on the source database.

    Resolved

Jul 6, 2018

  1. Engine YardStack Configuration Outage

    All current generation stacks are fully recovered at this time (Stable-v4, Stable-v5). Work continues on a handful of End of Life stacks at this time, but since the Supported stacks are complete we are marking this incident Resolved at this time. If you are experiencing any trouble with your environment please open a ticket for our Support team.

    Resolved

Jun 25, 2018

Jun 21, 2018

Jun 13, 2018

Older incidents