Resolved incident

Database Connection Issues

Scout APM · Major · Started

Duration
49m
Severity
Major
Status
Resolved

Summary

Major · Resolved

All chart metrics are now completely caught up. The root cause of the incident was due to attempted table partitioning during a database vacuum, which caused a lock on a critical table and cascaded to impact the rest of the application. We'll be adjusting our vacuum and partitioning schedules to avoid this lock again.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    Scout APM

    We appear to be using more than the expected number of database connections, causing failures on our Web UI. Ingestion is backed up, but the incoming data is safe and collected.

  2. Investigating

    Scout APM

    We've identified and fixed the database connection issue. We are currently loading the backlog of data that was held during the incident. Data will be appearing in the UI shortly.

  3. Resolved

    Scout APM

    All chart metrics are now completely caught up. The root cause of the incident was due to attempted table partitioning during a database vacuum, which caused a lock on a critical table and cascaded to impact the rest of the application. We'll be adjusting our vacuum and partitioning schedules to avoid this lock again.