TwitchIdentified Purchases Issues
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
🛠 We have resolved all known issues causing people to be unable to make purchases.
Resolved
This incident has been resolved.
Resolved
🛠 We have resolved all known login issues.
Resolved
🛠 We have resolved all known issues causing some users to be unable to checkout.
Resolved
This incident has been resolved.
Resolved
🛠 We have resolved all known issues causing people to be unable to make purchases.
Resolved
🛠 We have resolved all known issues causing the Twitch website to fail to load for some users.
Resolved
🛠 We've resolved all known issues of broadcasters not being able to stream on Twitch.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We have received reports of impacted services. This could manifest in chat not working, login difficulties, search impaired, etc. We're looking into the issue. 🛠 We have resolved all known negative service impacts.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Our primary redis instance was having a bad time. This has been fixed, and now we're back to normal.
Resolved
This incident has been resolved.
Resolved
New deployments were blocked, and users deploying their apps received 500 response codes. No active deployments were affected. This is fully resolved now.
Resolved
We're all set! Users can now create documents via API. If you continue to experience any issues with this, please reach out to us at support@pandadoc.com.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This issue is now resolved. If you are using Chrome 110 and the date time picker is not working correctly on a loaded doc, it should work again after refreshing the doc.
Resolved
The fix was completely rolled out in production and this incident is now fully resolved.
Resolved
The manage portal was out of service between 11:21-11:25 AM PST. The problem has resolved itself and we have returned to normal operations.
Resolved
We're all set! If you continue to experience any issues with the platform, please reach out to us at support@pandadoc.com.
Resolved
A fix has been made, and is being deployed. Update: fix is confirmed to be working.
Resolved
**Dates:** Start Time: Monday, February 6, 2023, at 20:05 UTC End Time: Tuesday, February 7, 2023, at 00:30 UTC Duration: 4 hours and 25 minutes **What happened:** Searches returned results slowly or not at all. Our Web UI was intermittently unresponsive, particularly for pages like Live Tail, Graphing, and Timelines. No data was lost and ingestion was not halted. **Why it happened:** We initiated an upgrade of all nodes in our service, including the nodes that store logs. Pods were gradually moved to other nodes and restarted, so as to prevent any interruption in service. A single pod that stores logs did not restart normally. Upon investigation, we found that it had not shut down cleanly and some files essential to a normal startup had not been written to disk. More significantly, we discovered that all nodes that store logs were using a podManagementPolicy of “orderedReady” \(the default setting\). This forced pods to restart in an ordered sequence. The single pod that would not restart was in the middle of the sequence; all the pods later in the sequence followed the policy and did not start either. In effect, about 25% of the pods within one zone \(out of the three zones devoted to storing logs\) were unable to start. The remaining pods in the zone were forced to take on extra work, such as accepting new logs, compacting data, and answering queries from our internal APIs. This led to slow searches and slow load times for any part of the Web UI that displays data about logs. **How we fixed it:** We temporarily added more pods to run API calls to increase the odds of them succeeding. We changed the podManagementPolicy to “Parallel” to allow all pods to restart, regardless of their position in the ordered sequence for starting up. We made manual edits to the pod that had not restarted cleanly so it could start again. These steps brought search latency back to normal speeds and made API calls work again. We cordoned off two pods that had fallen far behind in processing to allow them to recover without taking on new tasks. This temporarily removed ~2% of logs from all search results. When these pods were caught up with all pending tasks, we made them available again for search queries. **What we are doing to prevent it from happening again:** We have changed the podManagementPolicy to “Parallel” for all nodes that store logs. We will review the podManagementPolicy of all other areas of our service and make changes where appropriate. We will add alerting and monitoring to detect high latency in search speeds and the average time to compact newly inserted logs. We’ll explore options for adding more resources to each zone of pods, so they are less likely to fall behind on processing tasks when some pods are unavailable. We’ll explore ways to prevent unclean shutdowns of pods when nodes are upgraded.
Resolved
Game Server Services have closed for system upgrades.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
A small number of customers using SAML for authentication may have experienced errors when logging in. This incident is now resolved.
Resolved