Resolved incident

Dashboard and application portal were temporarily inaccessible for some users

Svix · Major · Started

Duration
< 1m
Severity
Major
Status
Postmortem

Summary

Major · Postmortem

## Chain of events 1. At 11:51pm UTC a new deployment changed the certificate to include the wrong CNAMEs. 2. The team discovered it at 11:52pm UTC. 3. Found the cause and prepared a fix at 11:57pm UTC 4. Issue resolved \(TLS certificate propagated\) at 12:07am UTC. ## What went wrong 1. A change for an upcoming big feature inadvertently changed the certificates used by the servers serving our static content \(dashboard and app portal\). 2. Since AWS may cache old certificates for short period of time both our automatic and manual tests were served the good \(old\) certificates and passed. This happened both in staging and in production. 3. We have ongoing monitoring on the API endpoints in both staging and production - but for the dashboard and frontend only in production. ## Aftermath We have yet to conclude our full analysis though will draw our conclusions and put additional safety mechanisms in place to ensure this \(and similar incidents\) never happen again.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Resolved

    Svix

    The dashboard and application portal were temporarily inaccessible for ~15 minutes for some users. There was a configuration issue that was causing wrong TLS certificates to be used.

  2. Postmortem

    Svix

    ## Chain of events 1. At 11:51pm UTC a new deployment changed the certificate to include the wrong CNAMEs. 2. The team discovered it at 11:52pm UTC. 3. Found the cause and prepared a fix at 11:57pm UTC 4. Issue resolved \(TLS certificate propagated\) at 12:07am UTC. ## What went wrong 1. A change for an upcoming big feature inadvertently changed the certificates used by the servers serving our static content \(dashboard and app portal\). 2. Since AWS may cache old certificates for short period of time both our automatic and manual tests were served the good \(old\) certificates and passed. This happened both in staging and in production. 3. We have ongoing monitoring on the API endpoints in both staging and production - but for the dashboard and frontend only in production. ## Aftermath We have yet to conclude our full analysis though will draw our conclusions and put additional safety mechanisms in place to ensure this \(and similar incidents\) never happen again.