Degraded Ingest due to database connectivity issues
Resolved
Minor impact
Updates
Resolved
This incident is now resolved. We are also noting that some other parts of honeycomb (the query UI) may have seen spurious failures as well during the impacted period (13:22—13:44 UTC).
Monitoring
We have stabilized the cluster and are monitoring the state while investigating to understand what triggered the degradation. Overall, 0.04% traffic was rejected with an error due to this degradation during the 20 minute window we were impaired.
Investigating
We are currently seeing elevated errors in connecting to some of our databases, which results in low amounts of Ingest traffic being rejected. We are looking into correcting the situation.