Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Mar 25, 2025

  1. AsanaApp and API errors

    This incident has been resolved. We are doing root cause analysis and will post an update within 72 hours.

    Resolved

Mar 24, 2025

  1. AppsFlyerIssue with install attribution

    Following our analysis, starting from approximately 20:15 UTC, installs have been attributed correctly to new impressions recorded after that time. Installs will not be attributed to impressions received before 20:15 UTC. Additionally, we observed a lower rate of re-engagement attributions during the incident window (approximately 19:35–20:10 UTC). The issue has been fully resolved. If you experience any further issues, please contact our support team.

    Resolved

Mar 23, 2025

Mar 22, 2025

Mar 21, 2025

Mar 20, 2025

  1. BitbucketBitbucket pipelines showing error when running a build.

    The issue where users running builds would receive an error message should now be resolved. This issue was impacting only those using self-hosted runners. It is expected that builds should have still run as expected despite this error message in the meantime.

    Resolved

Mar 19, 2025

  1. LeverData API Unavailable

    On the early morning of Feb 18, 2025 \(Pacific Time\), Lever product engineers were alerted by internal monitoring tools for a database instance being unavailable. Around 50% of customers may have initially experienced higher latency on Lever Hire and the Lever API. A few hours later, compounding issues in the database replicas caused Lever Hire to be inaccessible for those ~50% of customers, for a few hours. The impacted customer accounts were unable to access Lever Hire and candidate-related Data API endpoints at all on Feb 18 from 5:55-10:05 PST \(the longest outage, ~4 hours\), 14:08-14:18, 14:48-14:52, 21:52-22:09; and Feb 19 from 00:06-00:14. Lever-hosted job sites continued to work for all customers. Lever product engineers were engaged for investigation and troubleshooting pointed to database issues caused by unusual external Lever API load.  The issue was resolved by: * Rebuilding the affected database replicas * Spreading Lever API load across additional database replicas As part of this database recovery and mitigation measures, the Lever API also became inaccessible for all customers for a few hours. The database rebuild also caused some Lever Hire pipeline numbers and search results to be temporarily out of sync. To mitigate this situation from occurring in the future and to reduce the risk of a future impact the following measures have been put into place: * Per above, spreading Lever API load across additional database replicas * Limiting database time for individual Lever API requests, to prevent a few individual requests from having a wider impact * Optimizing database query performance

    Resolved

  2. LeverIntermittent issues impacting Lever Hire

    On the early morning of Feb 18, 2025 \(Pacific Time\), Lever product engineers were alerted by internal monitoring tools for a database instance being unavailable. Around 50% of customers may have initially experienced higher latency on Lever Hire and the Lever API. A few hours later, compounding issues in the database replicas caused Lever Hire to be inaccessible for those ~50% of customers, for a few hours. The impacted customer accounts were unable to access Lever Hire and candidate-related Data API endpoints at all on Feb 18 from 5:55-10:05 PST \(the longest outage, ~4 hours\), 14:08-14:18, 14:48-14:52, 21:52-22:09; and Feb 19 from 00:06-00:14. Lever-hosted job sites continued to work for all customers. Lever product engineers were engaged for investigation and troubleshooting pointed to database issues caused by unusual external Lever API load. The issue was resolved by: Rebuilding the affected database replicas Spreading Lever API load across additional database replicas As part of this database recovery and mitigation measures, the Lever API also became inaccessible for all customers for a few hours. The database rebuild also caused some Lever Hire pipeline numbers and search results to be temporarily out of sync. To mitigate this situation from occurring in the future and to reduce the risk of a future impact the following measures have been put into place: Per above, spreading Lever API load across additional database replicas Limiting database time for individual Lever API requests, to prevent a few individual requests from having a wider impact Optimizing database query performance

    Resolved

  3. Lever500 Error when accessing Lever platform

    On the early morning of Feb 18, 2025 \(Pacific Time\), Lever product engineers were alerted by internal monitoring tools for a database instance being unavailable. Around 50% of customers may have initially experienced higher latency on Lever Hire and the Lever API. A few hours later, compounding issues in the database replicas caused Lever Hire to be inaccessible for those ~50% of customers, for a few hours. The impacted customer accounts were unable to access Lever Hire and candidate-related Data API endpoints at all on Feb 18 from 5:55-10:05 PST \(the longest outage, ~4 hours\), 14:08-14:18, 14:48-14:52, 21:52-22:09; and Feb 19 from 00:06-00:14. Lever-hosted job sites continued to work for all customers. Lever product engineers were engaged for investigation and troubleshooting pointed to database issues caused by unusual external Lever API load.  The issue was resolved by: * Rebuilding the affected database replicas * Spreading Lever API load across additional database replicas As part of this database recovery and mitigation measures, the Lever API also became inaccessible for all customers for a few hours. The database rebuild also caused some Lever Hire pipeline numbers and search results to be temporarily out of sync. To mitigate this situation from occurring in the future and to reduce the risk of a future impact the following measures have been put into place: * Per above, spreading Lever API load across additional database replicas * Limiting database time for individual Lever API requests, to prevent a few individual requests from having a wider impact * Optimizing database query performance

    Resolved

Mar 18, 2025

  1. New RelicData Irregularities in the EU Region

    Between 18:28 UTC and 18:58 UTC Tuesday March 18, 2025, some customers in the EU region may have experienced errors when sending data to New Relic, issues in viewing data in real time, and delayed or missing alert notifications related to real time data. We have investigated and resolved this service interruption and impacted services have returned to normal operations.

    Resolved

Mar 17, 2025

  1. HotjarHotjar.com down for some users

    We've resolved an issue where hotjar.com was down for some users. This did not affect Insights nor data collection. hotjar.com is back up and running. Thank you for your patience! If you’re still seeing problems on your end please raise a ticket through our Help Center: https://help.hotjar.com/hc/en-us/requests/new. Thanks!

    Resolved

  2. KeeperVault login errors

    ### **Incident Summary** The Keeper DevOps team was alerted to API errors in the US region. Upon investigation, we identified a high number of database deadlocks affecting performance. To mitigate the immediate impact, we restarted the affected writer instance. ### **Root Cause** Historically, similar issues were linked to specific queries impacting MySQL 8.x updates on the RDS Aurora platform. In this case, we discovered that a utility designed to monitor deadlocks and capture diagnostic data was inadvertently contributing to the problem. A known bug in the MySQL minor version used in production caused our data-gathering query—intended to analyze deadlocks—to trigger additional locking, exacerbating the issue. ### **Resolution & Mitigation** * We promptly disabled the problematic monitoring utility and applied database parameter optimizations in collaboration with the AWS team to enhance performance under peak load. * These changes were fully implemented within 24 hours, and no further excessive row locks have been observed. * Our DBA team is actively working with backend engineers to further optimize write operations for improved performance. * A backend code update with additional optimizations is scheduled for release the week of March 17. We appreciate your patience and will continue to monitor and enhance system performance to prevent similar issues in the future.

    Resolved

Mar 16, 2025

  1. KeeperFile attachment download errors for new uploaded files

    ### **Incident Summary** In the US, EU, and AU regions, users experienced vault errors when attempting to download a file attachment immediately after adding it to a record. This issue affected only newly uploaded attachments in these three data center regions. The Gov, JP, and CA regions were not impacted. ### **Root Cause** During our investigation, we identified a discrepancy in how file attachment triggers were implemented across different regions. A recent Terraform change inadvertently removed the triggers responsible for moving file attachments in the US, EU, and AU regions, preventing newly uploaded files from being immediately available for download. ### **Resolution** We corrected the issue by restoring the missing triggers and applying the necessary changes. Additionally, any affected files were successfully moved to their intended locations. ### **Impact & Mitigation** * No data loss occurred. * All impacted files were successfully resolved by our corrections. * We corrected the issue to prevent similar discrepancies in future infrastructure changes. We apologize for any inconvenience and appreciate your patience while we resolved this issue.

    Resolved

  2. AppsFlyerIssue with Install Attribution to Impressions

    As of approximately 14:05 UTC, installs are being attributed to new impressions recorded after this time. Installs will not be attributed to impressions received before 14:05 UTC. The issue has now been fully resolved. If you experience any further issues, please contact our support team.

    Resolved

Older incidents