RedditDegraded performance for reddit.com
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Between 08:00 UTC to 11:54 UTC, we experienced problems with the invitation of new users for Cloud customers on admin.atlassian.com. The issue has been resolved and the service is operating normally.
Resolved
We're happy to report that the in-app events processing is back to normal. Thank you for your patience. If you're still experiencing any related issues, please reach out to customer support.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Between 00:36 UTC and 02:35 UTC on November 1st, 2024, some customers in the EU region may have experienced data visibility irregularities and delayed alert notifications for affected data. The interruption with our cloud provider has been resolved and affected services have returned to normal operations.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
**Dates:** Start Time: Saturday, October 26, 2024 at 01:00 UTC End Time: Saturday, October 26, 2024 at 20:23 UTC Duration: 19 hours and 23 minutes \(Note that customer impact was limited to the first four hours of the incident.\) **What happened:** For approximately two hours, pages on the Web UI for Pipeline did not load. Afterwards, pages were able to load, but often only after delays of 15 seconds or longer. After another two hours, the Web UI returned to normal usage. The incident was kept open until all remediation was completed. The Web UI for Log Analysis and the ingress and egress of Pipeline data were unaffected. **Why it happened:** On Wednesday, October 23, 2024, we deployed a change to our Pipeline service, by which metrics on pipeline usage began to be saved to a postgres database. The same database also stores account configuration information; this information must be accessed to display pages in the Web UI. The deployment on Wednesday changed the performance profile of the database, most notably in the number and frequency of writes. There was no immediate customer impact, but we noted that backups of the database were unable to run successfully because of the increased load. Customer impact only began on Saturday, October 26th when an unrelated user action – running a Profiler from the Web UI – placed even more demands on the postgres database. It was unable to process queries and, according to its design, moved into “read-only” mode to prevent any loss of data. At times the database was entirely inaccessible or very slow to respond. This made our Web UI unable to display pages, as it relies on configuration information stored in the database. **How we fixed it:** We first enabled two replicas of the database, which allowed the Web UI to load pages again, albeit slowly. We then deployed a new code change that removed all superfluous writes about metric usage to the postgres database. This reduced the number of queries and Web UI usage returned to normal. We kept the incident open while working on further remediation. Finally we took steps to bring the postgres database back to a normal state. This remediation phase was complicated by the fact that a full backup had not completed successfully in the last two days. By the end of the incident, the database was operating normally and a full backup had completed. **What we are doing to prevent it from happening again:** The problems with the deployment on Wednesday, October 23, 2024 only revealed themselves under the load of our production environment. To better simulate production, we will update our testing environment and processes to use higher data loads. We will closely evaluate the current workload of the postgres database and consider making a separate database to store just metrics, rather than combining metrics and configurations in one location. Separate databases would have prevented Web UI pages from not loading, thus avoiding any customer impact. We will re-evaluate our backup strategy for the postgres database, since this slowed down the remediation phase.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Between 07:03 UTC and 07:15 UTC on October 31st, some customers in the US region may have experienced delayed or missing alert notifications. This issue has been resolved and impacted services have returned to normal operations.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The issue has been resolved. Our team will monitor for further concerns.
Resolved
A small percentage of sessions between 14:00 UTC to 14:03 UTC were dropped from our EU/US RT cloud due to an issue triggered during infrastructure maintenance. We have mitigated the issue.
Resolved
This incident has been resolved.
Resolved
We're happy to report that the latency issues have been resolved. Thank you for your patience. If you're still experiencing any related issues, please reach out to customer support.
Resolved
Issues affecting the NGC Core Services have been resolved and the service is operating normally.
Resolved
A large backlog of reports caused the system to lag behind processing. This has been remedied and the system is now fully operational.
Resolved
We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.
Resolved
We have identified a brief traffic interruption that affected the loading of typeforms in the renderer from 00:30 to 01:30 CET. During this period, some users may have intermittently encountered a "no healthy upstream" error while attempting to load their forms. We are glad to inform you that a fix was implemented and the incident was resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused. REF: ZD 808202 | INC-68
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The upstream incident has been resolved, and we have confirmed that cluster creation is working normally.
Resolved
This incident has been resolved.
Resolved
From 7 AM - 3 PM UTC time, there was a problem with signing up to Asana. This issue has now been resolved, and users should be able to sign up again.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved. We apologize for the disruption this may have caused you. Please reach out to Airtable support or your account team if you have any questions about this incident.
Resolved
We’re back! Dropbox should be up and running. Thanks for bearing with us.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The backlog has been processed and all systems are running normally.
Resolved
Users were encountering error messages on GOV.UK pages, such as Report Child Abuse and Find a Local Council, when using the postcode lookup feature. Some received a 403 error stating, "You are not permitted to see this page," while others encountered a 503 error saying, "Sorry, we're experiencing technical difficulties."
Resolved
This incident has been resolved.
Resolved
We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.
Resolved