Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Sep 1, 2023

Aug 30, 2023

  1. KeeperGovCloud Admin Console login with ARAM permissions

    The GovCloud DevOps team uses automated infrastructure deployment systems via Terraform to manage all environments. A change was made to subnet VPC routing which inadvertently caused connectivity issues between certain backend services and a specific Redshift cluster. This issue affected admins on the Keeper Admin Console who have ARAM permissions applied to their role policies. Delegated admins without reporting permissions were not affected. Vault login and other client-side vault applications were not affected. Recent improvements to the Vault and Admin Console backend APIs masked the issue for the automated detection systems which would have immediately flagged the error and alerted the DevOps team. We have resolved the infrastructure route and updated the monitoring systems to take this into account and will continue to monitor for any future issues.

    Resolved

Aug 29, 2023

  1. GOV.UKGOV.UK assets unavailable

    Assets on GOV.UK (including images and PDFs) were unavailable for around 12 minutes while testing a CDN configuration change.

    Resolved

Aug 28, 2023

  1. OpsgenieWe observe degraded performance on incident timeline functionality

    The issue has been resolved completely and all the functionality is fully working now. The customers experienced some latencies for the entries added to the incident timeline during the incident. However, there is not data loss and all the messages processed successfully.

    Resolved

  2. MailChannelsReseller API Degradation

    We have now identified and resolved the performance issues with the Reseller API. If you have any questions or concerns, please contact MailChannels Support (support@mailchannels.com)

    Resolved

Aug 25, 2023

  1. NeonSome computes stuck during the suspend in us-east-2

    From **Aug 25th, 2023 04:20 PM UTC** till **Aug 25th, 2023 04:36 PM UTC**, a few compute endpoints where stuck during a suspend operation, leading to retries and increased suspend time. During this timeframe, the impacted compute endpoints weren't reachable.

    Resolved

  2. MailChannelsReseller API Degradation

    We have now identified and resolved the performance issues with the Reseller API. If you have any questions or concerns, please contact MailChannels Support (support@mailchannels.com)

    Resolved

  3. MailChannelsReseller API Degradation

    We have now identified and resolved the performance issues with the Reseller API. If you have any questions or concerns, please contact MailChannels Support (support@mailchannels.com)

    Resolved

Aug 24, 2023

  1. SvixElevated response times in the US region for a small percentage of the requests

    There were elevated response times in the US region for a small percentage of the requests. The issue was due to a memory leak in a new third party library we integrated with, which caused some nodes to operate with constrained resources (heavy CPU/MEM usage). This fixed itself automatically by killing the affected nodes (each one as it exhausted its resources over the span of a few minutes) and starting new ones instead. The team has also deployed a fix to revert the inclusion of the new library pending further investigation on why this was happening, and why it wasn't detected in the staging environment.

    Resolved

  2. KeeperFirefox extension update 16.6.2 - Resolved

    Version 16.6.2 has been published by Mozilla and it is available for update from the Firefox add-on store https://addons.mozilla.org/en-US/firefox/addon/keeper-password-manager/?src=search

    Resolved

  3. NeonSome computes stuck during the suspend in us-east-2

    This incident has been resolved. An EC2 node within our Kubernetes cluster experienced an issue where it failed to delete pods. This resulted in the project being stuck in the "suspend" operation. We have since cordoned off that particular EC2 node and redistributed the load. Our team is actively working on implementing a fix to prevent this issue from occurring in the future.

    Resolved

Aug 23, 2023

  1. KeeperVault login issues in US resolved

    A redshift cluster in the AWS region was unhealthy due to high CPU load from a recent backend API change. DevOps was alerted to the issue and worked with the engineering team to revert the change. Engineering will redeploy the update once the software changes have been implemented, and in addition a change is being made to prevent any redshift related issues to affect vault logins in the future \(KA-5630\). Additional scaling is planned on redshift \(SYS-2595\).

    Resolved

  2. MailChannelsReseller API Degradation

    We have now identified and resolved the performance issues with the Reseller API. If you have any questions or concerns, please contact MailChannels Support (support@mailchannels.com)

    Resolved

Aug 21, 2023

Aug 19, 2023

  1. NeonSome computes stuck during the suspend in us-east-2

    The incident has now been resolved. An EC2 node within our Kubernetes cluster experienced an issue where it failed to delete pods. This resulted in the project being stuck on "suspend" operation. We have since cordoned off that particular EC2 node and redistributed the load.

    Resolved

Aug 18, 2023

Aug 17, 2023

Aug 16, 2023

Aug 15, 2023

Aug 12, 2023

  1. KeeperEmail validation error on web vault login screen

    The login issue has been resolved. Please force-reload the web vault if you are still experiencing any issue. Download Keeper Desktop version 16.10.5 from our website or update from the app store when this version has been published in the next 24 hours.

    Resolved

Aug 11, 2023

Aug 10, 2023

Aug 9, 2023

Aug 8, 2023

Aug 7, 2023

Older incidents