HoneybadgerSourcemap upload issues
We have determined that some of the sourcemap traffic was being incorrectly routed to a staging server, which was encountering errors. This routing problem has been fixed.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
We have determined that some of the sourcemap traffic was being incorrectly routed to a staging server, which was encountering errors. This routing problem has been fixed.
Resolved
Charts should be caught back up - all systems back to normal.
Resolved
Metric ingestion for all customers is now caught up and operating normally.
Resolved
We're back to normal. No data was lost during the ingestion delay.
Resolved
Metric ingestion has caught back up.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We've resolved the issue. Periodic automations should start executing now and catch up shortly. Sorry for the trouble.
Resolved
This issue has been identified and resolved and all platform tasks are running smoothly again.
Resolved
We were experiencing an elevated level of API errors on our us-east region. This incident lasted from 2:11PM UTC to 2:18PM.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We just identified and resolved an issue where displaying some errors in the web UI was failing due to a problem with a 3rd party service. No other systems were impacted.
Resolved
New CDN is accessible worldwide and site should be fully operational for most users until propagation completes.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Zoom has identified and resolved the issue impacting joining and starting meetings. The Zoom integration should be back to full functionality.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved. All gems should be included in the versions API again.
Resolved
This incident has been resolved. We have monitored traffic and all communication seems to be running stable for the last 4 hours.
Resolved
This incident has been resolved.
Resolved
Due to an operation mistake, API service between 12:56PM UTC and 12:58PM UTC API had very high error rate on the Europe West region. The problem is mitigated and resolved. Detailed API error rate over time: 12:56PM 78% 12:57PM 93% 12:58PM 4%
Resolved
On November 13th between 13:38 and 14:14 PT, UserVoice experienced a networking infrastructure issue that caused a sitewide outage and system unavailability. **Business Impact** * During the outage end users and admins would have been unable to load or interact with UserVoice sites, widgets or the API. * Email would have been delayed, but no emails were lost. **Root Cause** In the process of cleaning up unused resources in the UserVoice infrastructure an old kubernetes cluster was removed from production. The automated cleanup of this cluster unintentionally removed a networking firewall rule that allowed our active application cluster to communicate with our backend infrastructure. Initial debugging was incorrectly focused around in-cluster symptoms and we did not immediately determine proper cause of the issue. Manual restoration of a proper firewall rule allowed the service to be fully restored. **What we are Doing to Prevent This** * Proper failover firewall rules are now being controlled via our infrastructure-as-code system preventing automated cleanup of old rules. * Infrastructure cleanup tasks will be scheduled during maintenance windows going forward. We didn’t meet our own or your expectations for using UserVoice with this outage. We do apologize for the pain points this caused for you and your team. If you have any questions or concerns, please reach out and let me know. Claire Talbott Support Manager claire.talbott@uservoice.com
Resolved
This incident has been resolved.
Resolved
On Friday 11/2/18 from 5AM to 5:13AM PDT, UserVoice experienced downtime. ## Business Impact During the time of the incidents end users and admins would have seen 500 errors. They wouldn’t have been able to load the admin console, use the API, interact with ideas on the front end or use the widget or Contributor Sidebar. Email would have been delayed, but no emails were lost. ## Root Cause We saw an issue similar to last Friday’s incident where one of the servers in Uservoice's database cluster experienced an application stall event. This caused a pause in database writes. Our engineering team manually removed the affected node to allow the cluster to resume operation. ## What we are Doing to Prevent This Our team has been focused, since last week, on finding the root issue that is caused one of our database clusters to stall. This work is still in progress. Once the root issue is identified, we will be implementing a fix and updating this report with the information discovered. In the meantime, we have put increased alerting in place so that should the issue repeat, we will identify it immediately. We understand UserVoice being down is an interruption to you and your team, and impacts your workflows! We take this downtime seriously, and are all hands on deck to get this issue fully addressed, so we can prevent it happening again. If you have any questions or feedback for us about the incident please don’t hesitate to contact me at [claire.talbott@uservoice.com](mailto:claire.talbott@uservoice.com). Claire Talbott Support Manager
Resolved
On October 26th between 10:20 and 10:35 PDT UserVoice experienced an infrastructure issue that caused intermittent system unavailability. ## Business Impact During the outage end-users and admins may have been unable to load or interact with UserVoice sites or widgets. ## Root Cause One of the servers in Uservoice's database cluster experienced an application stall event. Due to misconfiguration of our cluster, this caused a pause in database writes. Our engineers were required to manually remove the affected node to allow the cluster to resume writing. ## What we are Doing to Prevent This We have updated our database cluster configuration to more aggressively monitor and remove cluster members that are non-performant. This caused an interruption for your team in UserVoice and your end users trying to view and submit feedback. We are sorry for the pain point this caused. We have already put improvements in place to prevent this type of issue from happening again. If you have any questions, don’t hesitate to reach out at [claire.talbott@uservoice.com](mailto:claire.talbott@uservoice.com). Claire Talbott Support Manager
Resolved
On October 19th between 10:00 and 11:30 PDT UserVoice experienced two approximately 10 minute infrastructure outages that caused site-wide outages and system unavailability. **Business Impact** During the outage end users and admins would have been unable to load or interact with UserVoice sites or widgets. Email would have been delayed, but no emails were lost. **Root Cause** UserVoice uses an in-memory data-store cluster \(Redis\) to handle asynchronous job management and transient data storage. A recent change to one of the libraries that use this service caused a very sudden increase in its usage. The sudden usage increase caused a system failure and prevented failover to like-sized standby services. **What we are Doing to Prevent This** * Increased sizing of our Redis cluster and added additional alerting to allow us to more quickly detect usage spikes * Fixed the library that wasn’t properly interacting with Redis
Resolved
This incident has been resolved.
Resolved
This incident has been resolved
Resolved
at 3:06pm EST this issue has been resolved. Some learners may have experienced issues when searching the catalog in addition to forums not loading periodically.
Resolved
Between 03:59PM and 04:32PM UTC API traffic resulted in HTTP errors or timeouts. Only a part of Stream applications hosted on US were affected by this problem.
Resolved
Our distributed realtime cluster uses Redis (on Elasticache) for state management. A failover of the Elasticache cluster caused realtime to be unavailable for 7 minutes. This issue has been resolved and we're investigating why the failover took 7 minutes. This impacted customers using Stream's websocket, SQS or Webhook firehose systems.
Resolved
### January 23 and 24 outage postmortem report Stream suffered two incidents of degraded performance in the past 24 hours. We take uptime very seriously, and would like to be transparent with our operations and to our customers. The spikes occurred on Jan 23 at 3:50PM UTC and on Jan 24 at 11:45AM UTC. Both spikes were caused by a sudden increase of pressure to one of our PostgreSQL databases. Because Postgresql was slow at serving queries, HTTP requests started to pile up and eventually saturated the API workers' connection backlogs. API clients using a very low timeout will have encountered timeout exceptions. Other users of Stream would see 5xx responses on part of their API calls. I am going to add a little bit of background so that it is easier to elaborate on what went wrong. Some of our internal operations rely on moving data from one PostgreSQL database to another. Thanks to `psql` such operation is routinely performed by pipe-ing `COPY TO STDOUT` and `COPY FROM STDOUT` together. In order to not pressure the destination database with writes: we also use `pv` so that we are sure we never end up consuming all our IOPS capacity. The command looks more or less like this: ```psql src_db -c '\copy (...) to stdout' | pv -p --rate-limit 5242880 | psql dst_db -c '\copy (...) from stdout'``` By terminating the same copy command running on the source database we were able to remove **write** pressure on the disk. After that the high latency problem affecting the API service was automatically resolved. After researching on other possible causes, we concluded that the pressure created by the copy command combined with increased traffic was behind this outage and picked a different time to do it again. The same operation was then restarted during low traffic hours. To our surprise write pressure increased again after a couple of hours on the source database and caused another, albeit shorter, outage. After more digging we realized that in both occasions the command was running in the background and could not write to stdout, forcing the source database to store the query results on disk and then causing very high I/O on disk and slow response times to regular traffic. The remediation for this is very straight forward and it simply requires to never block stdout on the source database.
Resolved
All current generation stacks are fully recovered at this time (Stable-v4, Stable-v5). Work continues on a handful of End of Life stacks at this time, but since the Supported stacks are complete we are marking this incident Resolved at this time. If you are experiencing any trouble with your environment please open a ticket for our Support team.
Resolved
Update complete
Resolved
Connections are back, currently investigating the root cause.
Resolved
From 8:50am to 9:04am PST there were intermittent issues loading courses and assignments. This has now been resolved.
Resolved