Travis CIIssue with IBM Z builds on Travis CI
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
An update of the Plugin Portal needed to be reverted. This incident has been resolved.
Resolved
Replaying data is complete.
Resolved
Netlify fixed it :)
Resolved
Metric ingestion was paused at 20:13, restarted at 22:30 UTC and all apps metrics are caught up and stable as of 2019-09-18 0:00UTC. Operations are back to normal.
Resolved
We've completed all maintenance and all services should be operational.
Resolved
**Incident Timeline:** * 13:20 UTC: Our Virtual NOC alerts us to the failure of the monitoring URLs for the Engine Yard Cloud Dashboard. At this point EY Cloud is returning errors and so is inaccessible for customers. * 13:30 UTC: Investigation of the issue identifies the source as being a failed AWS instance, this instance is restarted at AWS in order to restore the instance. The restart hangs in the “stopping” state at AWS. * 13:35 UTC: Amazon publish a Service Status message to inform customers that a connectivity issue is affecting some instances in a single Availability Zone in the US-East-1 Region. * 13:35 - 14:45 UTC: Further attempts to restore the failed instance are unsuccessful due to the AWS issues, as are attempts to snapshot or reassign the instance’s EBS volume. * 14:45 UTC: Another platform instance running in the Cloud Dashboard environment is repurposed in order to take the place of the failed instance. * 15:00 UTC: Reconfiguration of the Cloud Dashboard application completes successfully and service is restored. * 15:00 UTC and onwards: With the Cloud Dashboard restored Engine Yard Support engineers work with customers to replace affected instances with alternative instances in unaffected AWS AZs. * 20:35 UTC: Amazon announce the issue is resolved and service is restored. Certain EC2 instances, RDS instances and EBS volumes remain in a failed state and cannot be restored by Amazon automatically, so EY Support continue to work with customers to restore these. **Incident Root Causes:** * **From AWS**: At 4:33 AM PDT one of ten data centers in one of the six Availability Zones in the US-EAST-1 Region saw a failure of utility power. Our backup generators came online immediately but began failing at around 6:00 AM PDT. This impacted 7.5% of EC2 instances and EBS volumes in the Availability Zone. Power was fully restored to the impacted data center at 7:45 AM PDT. By 10:45 AM PDT, all but 1% of instances had been recovered, and by 12:30 PM PDT only 0.5% of instances remained impaired. Since the beginning of the impact, we have been working to recover the remaining instances and volumes. A small number of remaining instances and volumes are hosted on hardware which was adversely affected by the loss of power. We continue to work to recover all affected instances and volumes and will be communicating to the remaining impacted customers via the Personal Health Dashboard. For immediate recovery, we recommend replacing any remaining affected instances or volumes if possible. * **From Engine Yard:** The root cause of the issue was quickly identified to be the failure of an AWS instance, responsible for the running of a core component of the EY Cloud Platform . The inability for any other platform instances to communicate with this instance resulted in errors in the platform. It was initially desired to restore the existing instance, or failing that, utilise the snapshot to retain data, but the AWS issues prevented such actions, leaving the only option to be to repurpose another existing instance. Once this action was completed, service was restored. **Incident Impact:** For the period that the Engine Yard Cloud Dashboard was offline no customers were able to view or manage their environments through either the Dashboard or the API, so were unable to make environment or application changes. Running instances outside of the subset of failed instances in the single AZ of US-East-1 were unaffected, so the majority of customer applications were not impacted. For those with instances in the affected AZ, application impact was dependent on the role of the impacted instances, with database and application master instances resulting in application downtime, whilst slaves instances most likely not. EY Support staff worked with customers to restore failed instances where practically possible within the limitations of the AWS issues. **Incident Corrective Actions:** Engine Yard will be working to strengthen the platform environments in order to ensure the highest resilience across all components in order to minimise the disruption from any future infrastructure failures.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This report details the impacts of our outage of August 26th, the cause for the outage, and steps we have and will be taking to prevent a similar kind of outage in the future. Before getting into the details, I want to apologize to everyone who was impacted by this outage – we have worked hard to build a resilient system, and it’s really disappointing when we let you down. On to the deets… ### What happened? A little after 7PM Pacific Time I received an alert from PagerDuty letting me know that one of our ingestion API endpoints was not responding to external monitoring. Reviewing our dashboards revealed that our primary Redis cluster was about to run out of available memory. This cluster stores the Sidekiq queue that we use for processing the payloads of the errors being reported to our API, and it typically rests at 3% memory utilization. Our internal dashboard did show some outliers among our customers for inbound traffic, but a few minutes of research down that path did not lead to a cause for the memory consumption. Running down our list of things to check in case of emergency led me to find that our database server was overloaded with slow-running queries. This caused our Sidekiq jobs to take much longer than usual \(30-50x as long\), which caused a backlog large enough to consume all the memory. Having our main Redis cluster effectively unusable resulted in the following problems: * Our API endpoints were unable to receive error reports, source map uploads, deployment notifications, and check-in reports * Uptime checks were delayed * Some check-ins were switched to the down state because the check-in reports couldn't be received * Some error notifications were lost as we had to juggle Redis instances \(more on that below\) ### When was it fixed? Not long after 9PM our API was responsive again. By 10PM our backlog of error payloads was fully processed, and we were back to normal. We would have been back in business sooner, but a few things tripped us up: * As soon as it was clear we were going to run out of memory on our Redis cluster \(which is hosted by AWS ElastiCache\), and that I wouldn't be able to quickly free up some memory, I started a resize of the existing cluster. When it became clear that would not be quick \(it ended up taking approximately two hours\), I spun up a new, larger ElastiCache cluster. When it become clear _that_ would not be quick, I spun up an EC2 instance in our VPS to host Redis temporarily. * Unfortunately, though we use Ansible for automating all our EC2 provisioning, we did not have a playbook for quickly spinning up a Redis server. When I set up the first server manually, I didn't provision enough disk space on the instance to store the Redis snapshot as the backlog grew \(I didn't have the fix in place for the slow queries yet\). * When I spotted that problem with the Redis instance, I spun up another one with a large-enough disk. * We also didn't have an automated way to update the 4 locations where our app, api, and worker instances were configured with the location of the Redis server, so with each of the two changes to the Redis server location I had to run some Ansible commands to update configurations and bounce services Once that was all settled, though, and traffic was once again flowing in to our new self-hosted Redis instance, I was able to turn my attention to the cause of the problem – the slow queries. It turns out that one query was the cause of the slowness. This query loaded previously-uploaded sourcemaps to be applied to Javascript errors as they were being processed. Since the problem was so localized, I was able to get the database back to a good place by temporarily suspending the sourcemap processing. ### What's the remediation plan? As you can imagine, there are a number of things we can do to help avoid or minimize this kind of situation in the future: 1. Review that database table and/or the query to see how we can get past the tipping-point we encountered that turned a 4-10ms query into a 400ms query for certain customers \(in process\) 2. Persist payloads to S3 sooner in our Sidekiq jobs so we can minimize memory pressure on Redis 3. Increase the size of our Redis cluster \(already done\) 4. Create an Ansible playbook to quickly provision a new Redis instance in case of emergency \(done\) 5. Centralize the 4 app configurations for the URL of the Redis cluster and create an Ansible playbook that can quickly update those configurations \(in process\) We'll continue to improve our systems and processes to deliver the most reliable service we can. We truly appreciate that you've chosen us for your monitoring needs, and we are always eager to show our appreciation by working hard for you. As always, if you have any questions or comments, please reach out to us at [support@honeybadger.io](mailto:support@honeybadger.io).
Resolved
All buffered checkin data has been ingested, and all components are back online.
Resolved
All buffered checkin data has been ingested, and all components are back online.
Resolved
Issue has been fixed for Google. This was a global outage affecting all sites Google was installed on.
Resolved
We've killed a rogue process that was tying up our database, and all pages are responding.
Resolved
On August 15, 2019 at 20:45:43 GMT, our monitoring system alerted that a core component of the platform was not responding. This component is responsible of providing information on stack versions and access to Chef recipes used to configure instances. At 20:55:49 GMT our Support Team begun investigating the issue. At 21:00:01 GMT it was identified that the application running this component couldn't connect to the database. At 21:01:47 GMT it was detected that the db instance had restarted itself and was entering recovery mode. At 21:02:48 GMT the component resumed services. At 21:14:42 GMT, after both automatic and manual testing reported no failures, the issue was deemed as solved.
Resolved
The issue has been resolved. Thank you for your patience and understanding.
Resolved
This incident has been resolved.
Resolved
The backlog of ingestion data has cleared, and everything is up and running.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
On August 5, 2019, from 5:49PM to 6:23PM EDT, the admin console and UserVoice API were down. **Business Impact** * Admins saw 503 errors in the admin console, or noticed excessive slowness when loading parts of the application. * The UserVoice API returned errors. * Users on the web portal or with the widgets may have briefly seen errors or slowness at the start of the incident, but only for a few moments. **Root Cause** * A configuration change during a routine Kubernetes Cluster upgrade caused an unexpected restart for key operational services. One of these services had trouble recovering, which caused the downtime. **What we are Doing to Prevent This** * Our team identified and fixed the issue that caused the trouble during restart. * Our team also identified the setting that triggered the unplanned update, and how best to ensure that does not happen again. We do apologize for the downtime. This interrupted your usage of UserVoice, and for that we truly are sorry. If you have any questions or concerns, please, don’t hesitate to reach out! Claire Talbott Support Manager claire.talbott@uservoice.com
Resolved
We should be back online now. Sorry for inconvenience, this was a misconfiguration issue on a recent deployment.
Resolved
This incident has been resolved.
Resolved
Backlogs have cleared on both sites hence we are resolving this incident.
Resolved
All backlogs have cleared now. Sorry again for the inconvenience and happy building!
Resolved
**Incident Timeline:** * 05:45 UTC: The first customer reports that running Chef on their environment failed due to an ongoing run - this was fixed in the standard fashion and the customer informed. * 06:30 - 07:00 UTC: Three further customers reported the same issue. No further reports were received after this time. * 09:00 UTC: Support Engineers working the customer tickets found that the standard fix did not resolve the issue and that Chef runs instigated via the ey-core gem were failing to initiate correctly in our platform backend. This was found to apply to all requests resulting in platform actions \(Apply, Deploy, etc\) made via the API, either being generated by the use of the ey-core gem or via direct curl requests to the API. Dashboard actions and calls using the older ey gem CLI remained functional. * 09:45 UTC: The status page and dashboard were updated to acknowledge the incident and request customers avoid performing tasks via the API and ey-core gem. * 09:45 UTC - 11:30 UTC: Engineers continued to investigate various platform components in order to identify the issue. * 11:30 UTC: The database instance in the environment of a platform component used for log streaming for API calls was identified as failed, due to failing hardware at AWS. The instance was restarted to move it to a new host, and the application reconfigured and restarted to bring it back online. * 12:25 UTC: After full testing the incident was declared resolved and the impacted customers informed. **Incident Root Cause:** The root cause of the incident was confirmed to be the failure \(at the host hardware level\) of the database instance in an environment not directly used, but depended upon for logging, by the API and thus the ey-core gem that calls to the API. **Incident Impact:** All customers on the EY Platform were unable to perform actions on their environments through API calls, most commonly enacted through the use of the ey-core gem. As the EY Cloud Dashboard was still fully functional customers were able to perform all required actions via the UI, with the sole exception being the uploading of custom Chef recipes, which is an API only process, though this action remained performable throughout due to it not utilising the impacted log streaming component. Also due to the time of day the number of customers impacted was minimal. Therefore the impact can be classified as Minor. **Incident Corrective Actions:** Initial corrective actions will entail ensuring monitoring and alerting is in place on the log streaming environment in order to make any future failures more immediately apparent to frontline staff. Future corrective actions will be to investigate the APIs reliance on the log streaming environment and investigate ensuring failure of that environment is a non-breaking event.
Resolved
This incident has been resolved.
Resolved
Builds are running smoothly and we have significant remaining headroom. Please send your commits our way! We are closing this incident for now and are thanking you again for you enduring patience.
Resolved
On July 3, 2019 at 17:12:45 GMT, our monitoring system alerted that a core component of the platform was not responding. This component is responsible of providing information on stack versions and access to Chef recipes used to configure instances. At 17:20:05 GMT it was identified that the application running this component couldn't connect to the database. At 17:21:15 GMT it was detected that the db instance had restarted itself and was entering recovery mode. At 17:30:54 GMT the component resumed services. At 17:54:33 GMT, after both automatic and manual testing reported no failures, the issue was deemed as solved.
Resolved
Backlog on travis-ci.org has cleared has of 2019-07-03 2:14 UTC. Thanks again for your continued patience during this incident.
Resolved
The networking issues have now been resolved and the Support Portal is now fully back online.
Resolved
All services have been confirmed to be performing as expected.
Resolved
The regression introduced for PHP applications in stack v5-3.0.59 has now been identified and fixed in a new stack update, v5-3.0.60. PHP environments already on v5-3.0.59 should upgrade to v5-3.0.60 and those not yet upgraded should skip the .59 release. Contact EY Support for more information.
Resolved
All chart metrics are now completely caught up. The root cause of the incident was due to attempted table partitioning during a database vacuum, which caused a lock on a critical table and cascaded to impact the rest of the application. We'll be adjusting our vacuum and partitioning schedules to avoid this lock again.
Resolved
This incident has been resolved.
Resolved
This issue is resolved. We have implemented filters to catch the spam, and are seeing it successfully catch these emails. We have also cleaned up the emails that made it past our filters earlier today. We are also investigating ways to continue to update and improve our spam filters so we can address these types of spammers quicker and more efficiently. We do apologize for the pain point this caused for you and your team, and if you have any questions, please, don't hesitate to reach out!
Resolved
Server responses should be back to normal now. Please email support@workramp.com if you have any other issues.
Resolved
This incident has been resolved.
Resolved
This issue has been resolved.
Resolved
This incident has been resolved.
Resolved