Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Sep 1, 2021

  1. StreamElevated API error rate in Dublin

    Traffic to our Dublin infrastructure experienced elevated error rate due to a AWS outage. The incident started at 11:20PM, error rate decreased at 11:38PM and the incident was resolved by 11:59PM We are still performing impact and root-cause analysis, a postmortem with more information will be posted here.

    Resolved

Aug 31, 2021

Aug 24, 2021

Aug 23, 2021

Aug 17, 2021

  1. SvixMessage sending delays

    We've experienced delays in message delivery, the issue has now been resolved.

    Resolved

Aug 11, 2021

Aug 5, 2021

  1. SvixDashboard and application portal were temporarily inaccessible for some users

    ## Chain of events 1. At 11:51pm UTC a new deployment changed the certificate to include the wrong CNAMEs. 2. The team discovered it at 11:52pm UTC. 3. Found the cause and prepared a fix at 11:57pm UTC 4. Issue resolved \(TLS certificate propagated\) at 12:07am UTC. ## What went wrong 1. A change for an upcoming big feature inadvertently changed the certificates used by the servers serving our static content \(dashboard and app portal\). 2. Since AWS may cache old certificates for short period of time both our automatic and manual tests were served the good \(old\) certificates and passed. This happened both in staging and in production. 3. We have ongoing monitoring on the API endpoints in both staging and production - but for the dashboard and frontend only in production. ## Aftermath We have yet to conclude our full analysis though will draw our conclusions and put additional safety mechanisms in place to ensure this \(and similar incidents\) never happen again.

    Resolved

Aug 2, 2021

Jul 28, 2021

Jul 26, 2021

Jul 19, 2021

  1. LogRocketSDK breaking lodash on some sites

    A bug in our build pipeline caused our SDK to expose an internal library globally. As a result, our subset of the lodash library was bound to `window._`, overwriting the existing `_` global for some customer sites. This meant that some expected library functions were not defined after our script loaded. #### Technical Details: * A code change imported all of lodash and accessed a method on the import instead of importing just the method. * Webpack bundled lodash, including its `window._ `assignment despite the library being used in a module import context. #### Remediations: * We’ve added additional tests to catch unwanted pollution of the window object by our SDK * We’ve added an internal linting rule to avoid full lodash imports

    Resolved

Jul 13, 2021

Jul 6, 2021

Jun 25, 2021

Jun 24, 2021

Jun 22, 2021

Jun 11, 2021

Jun 10, 2021

  1. ApigeeDatastore Errors in AWS EU-Central-1

    AWS Availability Zone euc1-az1 experienced an outage today at roughly 1:25PM Pacific. In-flight requests to our database servers in that zone were rejected as the machines were unavailable, but only in-flight requests at the instance of the outage were affected. There is no ongoing impact, as our High Availability immediately routed all incoming requests to available nodes. We will continue to monitor the situation but expect no other issues to arise.

    Resolved

  2. Webhooks.ioAPI and Gateway Outage

    Customers seeing error messages connection to API or sending messages into the gateway. We started receiving connection errors at 11:10 AM U.S. Central Time (CT) this morning from our API and gateway that connections were refused. We immediately identified the cause of the issue and found a resolution to the problem. This issue was fix and traffic returned back to normal at around 1:30PM U.S. Central Time (CT). We are continuing to monitor the situation but believe the issue has been fully repaired. We will keep you updated here with the latest updates if anything changes.

    Resolved

Jun 9, 2021

Jun 8, 2021

Jun 4, 2021

  1. LogRocketSession Ingestion Delays

    Systems have remained stable while continuing to process the backlog of events. We estimate secondary processing will be live in the next 6 hours.

    Resolved

Jun 3, 2021

  1. Scout APMLoad balancer change caused metric gap

    A change to our elastic load balancer which accepts the metrics payloads from agents was temporarily in a nonfunctional state from 3:30PM MT to 3:35PM MT on 6-3-2021. The issue has been resolved. You may have missing metrics in charts for this time period.

    Resolved

Jun 1, 2021

May 29, 2021

  1. HoneybadgerBug in error grouping

    We accidentally broke grouping for some errors for a period of about 8 hours today, ending right before this update was published. No errors were lost, but you may have seen what looked like duplicate errors, as we failed to group some errors together that should have been. Sorry for the inconvenience!

    Resolved

May 21, 2021

May 17, 2021

May 6, 2021

Apr 28, 2021

Apr 27, 2021

Apr 20, 2021

  1. LogRocketAuthentication Service Outage

    Dashboard users appear to be able to log in successfully again. We will continue monitoring the incident until our authentication provider reports that all functionality has been restored and provide any relevant updates.

    Resolved

Apr 15, 2021

Apr 14, 2021

  1. StreamIncreased error rate on Chat API

    We experienced higher than normal error rates during a database maintenance on Chat API. The error increased started at 5:24AM and resolved at 5:42AM UTC.

    Resolved

  2. DopplerCLI Tokens Authentication Issues

    The issue causing CLI authentication to fail has been resolved. CLI tokens should now be functioning as expected. If your CLI token continues not to work, please reach out to support.

    Resolved

Older incidents