Resolved incident
Data API Unavailable
Lever · Major · Started
- Duration
- 74h 29m
- Severity
- Major
- Status
- Postmortem
Summary
Major · Postmortem
On the early morning of Feb 18, 2025 \(Pacific Time\), Lever product engineers were alerted by internal monitoring tools for a database instance being unavailable. Around 50% of customers may have initially experienced higher latency on Lever Hire and the Lever API. A few hours later, compounding issues in the database replicas caused Lever Hire to be inaccessible for those ~50% of customers, for a few hours. The impacted customer accounts were unable to access Lever Hire and candidate-related Data API endpoints at all on Feb 18 from 5:55-10:05 PST \(the longest outage, ~4 hours\), 14:08-14:18, 14:48-14:52, 21:52-22:09; and Feb 19 from 00:06-00:14. Lever-hosted job sites continued to work for all customers. Lever product engineers were engaged for investigation and troubleshooting pointed to database issues caused by unusual external Lever API load. The issue was resolved by: * Rebuilding the affected database replicas * Spreading Lever API load across additional database replicas As part of this database recovery and mitigation measures, the Lever API also became inaccessible for all customers for a few hours. The database rebuild also caused some Lever Hire pipeline numbers and search results to be temporarily out of sync. To mitigate this situation from occurring in the future and to reduce the risk of a future impact the following measures have been put into place: * Per above, spreading Lever API load across additional database replicas * Limiting database time for individual Lever API requests, to prevent a few individual requests from having a wider impact * Optimizing database query performance
Event timeline
How this incident unfolded, from the official status page.
Identified
Lever
We’ve identified an issue with the Lever Data API, and we’re working to implement a fix. The data API will be unavailable as we work to resolve the behaviors. Additional updates will be provided as soon as a resolution is in place.
Identified
Lever
We are continuing to work on a fix for this issue.
Monitoring
Lever
A short term fix has been implemented for the issue that was identified while our team works to implement a long term fix. Customers should now be able to leverage the data API while our team continues their investigation. We’re currently monitoring to ensure that there are no further issues, and we’ll send a final update to confirm that there have been no recurrences.
Identified
Lever
Our team has identified an issue with the Lever Data API. Our team has a short term fix in place while we work to implement a full resolution. Customers may experience intermittent issues while we work towards a full resolution. Our engineering teams are working to resolve this issue as soon as possible.
Monitoring
Lever
Customers should now be able to utilize Lever Data APIs with no issues. We’re currently monitoring to ensure that there are no further issues, and we’ll send a final update to confirm that there have been no recurrences.
Resolved
Lever
The issue where the Lever Data API being unavailable has been resolved. There should be no further impact at this time, but please reach out to us at Support if any additional assistance is needed: https://help.lever.co/hc/en-us/requests/new
Postmortem
Lever
On the early morning of Feb 18, 2025 \(Pacific Time\), Lever product engineers were alerted by internal monitoring tools for a database instance being unavailable. Around 50% of customers may have initially experienced higher latency on Lever Hire and the Lever API. A few hours later, compounding issues in the database replicas caused Lever Hire to be inaccessible for those ~50% of customers, for a few hours. The impacted customer accounts were unable to access Lever Hire and candidate-related Data API endpoints at all on Feb 18 from 5:55-10:05 PST \(the longest outage, ~4 hours\), 14:08-14:18, 14:48-14:52, 21:52-22:09; and Feb 19 from 00:06-00:14. Lever-hosted job sites continued to work for all customers. Lever product engineers were engaged for investigation and troubleshooting pointed to database issues caused by unusual external Lever API load. The issue was resolved by: * Rebuilding the affected database replicas * Spreading Lever API load across additional database replicas As part of this database recovery and mitigation measures, the Lever API also became inaccessible for all customers for a few hours. The database rebuild also caused some Lever Hire pipeline numbers and search results to be temporarily out of sync. To mitigate this situation from occurring in the future and to reduce the risk of a future impact the following measures have been put into place: * Per above, spreading Lever API load across additional database replicas * Limiting database time for individual Lever API requests, to prevent a few individual requests from having a wider impact * Optimizing database query performance