Resolved incident
RESOLVED: Issue affecting multiple services
Wix · Minor · Started
- Duration
- 16m
- Severity
- Minor
- Status
- Postmortem
Summary
Minor · Postmortem
On July 7th at 7:50 AM UTC, some users experienced issues loading the Wix Dashboard and Wix Editors. All functionality was fully restored within approximately 20 minutes. The root cause was an interaction between two events: a new automated database infrastructure optimization feature introduced on July 5th, and a routine planned traffic shift between data center regions carried out on the morning of July 7th. The new feature is designed to improve infrastructure efficiency by continuously monitoring database throughput usage and automatically adjusting capacity limits to reflect actual demand. On July 7th, before the day's planned traffic operations had taken place, the feature applied reduced capacity limits to several database clusters in one of our regions, based on the lower load observed earlier that morning. Shortly after, a routine planned traffic shift moved additional user traffic to that same region. The database clusters, now operating under reduced limits, received more load than those limits could accommodate, causing database response times to spike and errors to cascade across our dashboard and editors. Our team identified the root cause and resolved the incident by redirecting traffic to an unaffected region. We also provisioned new database nodes with higher capacity limits to restore full throughput in the impacted region. Following this incident, we have paused the automated optimization feature while we put additional safeguards in place. We have audited all database clusters where capacity adjustments were applied to confirm they are operating correctly. We are also implementing coordination checks to ensure that automated capacity adjustments and planned traffic operations are always validated against one another. We have additionally improved monitoring and alerting for automated infrastructure changes to enable faster detection and response if a similar situation were to arise in the future.
Event timeline
How this incident unfolded, from the official status page.
Investigating
Wix
We are investigating issues with multiple services.
Monitoring
Wix
A fix has been implemented and we are monitoring the results.
Resolved
Wix
This incident has been resolved.
Postmortem
Wix
On July 7th at 7:50 AM UTC, some users experienced issues loading the Wix Dashboard and Wix Editors. All functionality was fully restored within approximately 20 minutes. The root cause was an interaction between two events: a new automated database infrastructure optimization feature introduced on July 5th, and a routine planned traffic shift between data center regions carried out on the morning of July 7th. The new feature is designed to improve infrastructure efficiency by continuously monitoring database throughput usage and automatically adjusting capacity limits to reflect actual demand. On July 7th, before the day's planned traffic operations had taken place, the feature applied reduced capacity limits to several database clusters in one of our regions, based on the lower load observed earlier that morning. Shortly after, a routine planned traffic shift moved additional user traffic to that same region. The database clusters, now operating under reduced limits, received more load than those limits could accommodate, causing database response times to spike and errors to cascade across our dashboard and editors. Our team identified the root cause and resolved the incident by redirecting traffic to an unaffected region. We also provisioned new database nodes with higher capacity limits to restore full throughput in the impacted region. Following this incident, we have paused the automated optimization feature while we put additional safeguards in place. We have audited all database clusters where capacity adjustments were applied to confirm they are operating correctly. We are also implementing coordination checks to ensure that automated capacity adjustments and planned traffic operations are always validated against one another. We have additionally improved monitoring and alerting for automated infrastructure changes to enable faster detection and response if a similar situation were to arise in the future.