Resolved incident
Some HTTP Actions returning 404
Convex · Minor · Started
- Duration
- 20m
- Severity
- Minor
- Status
- Resolved
Summary
Minor · Resolved
This incident is resolved. Between 00:20 and 04:15 UTC on 12 August, some HTTP Actions returned a 404 response. This affected requests that reached Convex through an intermediate proxy or CDN, because those requests carry an `X-Forwarded-Host` header. A change to our edge configuration made it read that header instead of the request's own hostname, so it looked for the wrong deployment. We reverted the change in all regions and error rates are back to their normal level. No data was lost, and requests that did not carry that header were unaffected. We are sorry for the disruption, and are improving our testing and monitoring of edge configuration changes to catch issues like this before they reach production.
Event timeline
How this incident unfolded, from the official status page.
Identified
Convex
We're investigating reports that HTTP Actions are returning 404 responses when an `X-Forwarded-Host` header is set on the incoming request. We've identified a potential infrastructure change and are reverting it now.
Resolved
Convex
This incident is resolved. Between 00:20 and 04:15 UTC on 12 August, some HTTP Actions returned a 404 response. This affected requests that reached Convex through an intermediate proxy or CDN, because those requests carry an `X-Forwarded-Host` header. A change to our edge configuration made it read that header instead of the request's own hostname, so it looked for the wrong deployment. We reverted the change in all regions and error rates are back to their normal level. No data was lost, and requests that did not carry that header were unaffected. We are sorry for the disruption, and are improving our testing and monitoring of edge configuration changes to catch issues like this before they reach production.