LogRocketPerformance issues with primary search database
We've seen no additional performance degradation after our mitigation work.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
We've seen no additional performance degradation after our mitigation work.
Resolved
# Summary From 2023-01-16 19:11 UTC to 2023-01-16 20:04 UTC, Doppler experienced a partial outage which prevented sync integrations, webhooks, and activity log notifications from executing. The outage also prevented an internal job from firing which recomputes the version hashes for Doppler configs. This resulted in API clients \(e.g. the Doppler CLI and Kubernetes Operator\) failing to receive secrets updates which were made during this window. A recovery migration was run at 2023-01-17 00:37 UTC, re-triggering all syncs, webhooks, and activity log notifications — as well as recomputing config version hashes to restore the functionality for all clients to fetch secret updates. # Incident Details Doppler uses RabbitMQ to queue jobs which need to be executed as a result of secret updates. On 2023-01-13, Doppler’s security team rotated a RabbitMQ password, mistakenly identifying the credential as unused in production. It took several days for the RabbitMQ sessions in Doppler’s production services to expire and once they did, queue jobs could no longer be published. Once the incident was identified, Doppler’s security team created new RabbitMQ users to be used by our production services. The change was deployed and the incident was resolved at 2023-01-16 20:04 UTC. At 2023-01-17 00:37 UTC, Doppler ran a recovery migration to re-fire queue events for sync integrations, webhooks, activity log notifications, and secret version hash recomputations that were meant to fire during the incident window. # Next Steps Doppler has switched from using a single RabbitMQ credential to using one user per service. RabbitMQ users are now clearly named to mitigate the risk of accidental rotation in the future. We’ve also identified that the ability for API clients to fetch secrets should not be dependent on our application’s ability to connect to RabbitMQ. Our engineering team will move the config version hash computation to our atomic secrets write operation to ensure that the latest secrets are always fetched by clients. Lastly, our engineering team is reconfiguring the way we queue asynchronous jobs to ensure that if secrets are modified during a partial infrastructure failure, all post-update jobs will eventually be executed — without the need for manual recovery migrations.
Resolved
Coursera.org is currently unavailable on some browsers. We are investigating the issues and working on a resolution.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The event was momentary and lasted only for a small number of requests in a duration less than 30 seconds. It's unlikely that any customer has noticed a degradation in TaT for any service. Streaming API connections may have been terminated by the server or partial/final hypothesis delayed by up to a minute during the event. If users are following our guidance and buffering audio locally and attempting to reconnect to the server when a server side termination is initiated, impact to end users would have been minimal. We are continuing to investigate the underlying cause to prevent any future disruptions such as these.
Resolved
Post mortem coming soon.
Resolved
**Root Cause** Doppler syncs secrets to Heroku via a Heroku OAuth application. This application is created in the Heroku dashboard and must be owned by a single Heroku user account. Doppler’s previous Heroku OAuth application was owned by a specific Doppler employee’s Heroku account without access to any additional resources. During a routine external account audit, this account was mistakenly identified as unused and manually deleted by our security team. This irrecoverably deleted Doppler’s existing Heroku OAuth application, thereby breaking any existing syncs and requiring the creation of a new OAuth application in a new account. **Resolution** Because users had authorized our previous Heroku OAuth application to their Heroku account\(s\), users need to authorize the new Heroku OAuth application. This involves reconnecting the integration from the [Doppler dashboard](https://dashboard.doppler.com/workplace/settings). Once the integration is reconnected, Doppler will re-enable all associated syncs that have been disabled and perform a fresh sync. Note that the previous OAuth application was deleted and therefore no action is required to remove its access to your Heroku account. **Next Steps** Internally, we’re reorganizing how shared accounts used for critical functionality are stored in 1Password. This new 1Password organization should help prevent this kind of accidental deletion in the future. We avoid shared accounts whenever possible, but this isn’t always feasible given third party implementations. We'll also be adding our individual integrations to our status page. This will allow customers to more easily see which integrations, if any, are currently experiencing issues.
Resolved
This issue is resolved. The aggregated reports are up to date. Thank you for your patience.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved. You may need to reinstall Travis CI GitHub App if you added a new repository or created a new account between 19 December 2022 12:06:34 and 20 December 2022 GMT 13:00:00.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved as of 10:44 AM PT.
Resolved
We have resolved the problem. The portal can be used to create and update cases.
Resolved
All systems are functioning normally again.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Opsgenie analytics dashboards were not available between 2022-11-29 18:15 (UTC) and 2022-11-29 19:30 (UTC). We observed a spike in traffic patterns which caused degradation in one of our services. After detecting the problem, our engineering team worked towards getting the service back up as quickly as possible. The analytics dashboards started working partially from 2022-11-29 19:30 (UTC) and were fully functional by 2022-11-29 20:10 (UTC).
Resolved
Platform loading issues, which may cause a browser timeout error. This was roughly a 10 minute outage.
Resolved
JCenter's certificate has been renewed.
Resolved
This incident has been resolved.
Resolved
Our API service experienced an abnormal number of timeouts affecting 0.6% of total traffic due to DNS errors between 12:30-1:10 pm PST.
Resolved
The issue with logging into our platform has been resolved. We apologize for the inconvenience.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This issue is resolved. Data related to installs and inapp events in our dashboards and aggregated reports is up to date.
Resolved
Fix is confirmed and stable.
Resolved
We believe the provider issues have been resolved and will be working with them to ensure this does not happen again.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Between 2022-11-06 and 2022-11-11, 18:45 EST, we experienced an issue where Android users of Jira, Confluence, and Opsgenie apps with the Compromised device check feature turned on is getting locked out of their apps for Confluence, Jira Software, and Opsgenie. The issue has been resolved and the service is operating normally. If a customer is locked out post-login and there is no retry option, we request that a user either clears the app data or reinstall the app.
Resolved
This incident has been resolved. All notifications have been processed. No notifications were lost.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The incident has been resolved. Logs are available without any delay.
Resolved
The incident has been resolved. It was due to a programming error that made a query less efficient for a particular set of endpoints.
Resolved