Updates
Resolved
The incident has been successfully resolved, and normal operations have been restored to all affected systems and services. We found the root cause in doing too many processing operations for some of the API endpoints we currently use to provision new database connections. We find that this amount of processing operations can and should be reduced and this will stop these kind of errors from appearing in the future, and we will work on that.
Investigating
Investigating: (13:13) Update: 🛠️ **Affected Component**: Console and API requests 🚀 **Impact**: Partial outage We are currently experiencing technical issues that may affect our services. Our team is diligently working to resolve these issues and restore normal operations. We understand the inconvenience this may cause and appreciate your patience. Stay tuned for further updates. Identified: (13:21) The console release led to huge amount of extra connections to console database, producing extra load. We're working on the solution. Identified: (13:37) We have stabilized the API by providing rate limiting on certain endpoints, and currently looking for the root cause of the problem. Some rare database connectivity problems are still possible, while the system is recovering, we expect the system to become fully operational in minutes. Monitoring: (14:01) The incident is currently in the "Monitoring" state, and our team is actively monitoring the situation to ensure stability and prevent any recurrence. We are closely observing key metrics and trends to make informed decisions about the incident's progress. If you have any questions or need further information about this incident, please reach out to our support team at support@neon.tech