Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Nov 6, 2024

Nov 5, 2024

Nov 4, 2024

Nov 3, 2024

  1. AppsFlyerDelay in installs attribution

    We're happy to report that the in-app events processing is back to normal. Thank you for your patience. If you're still experiencing any related issues, please reach out to customer support.

    Resolved

Nov 1, 2024

  1. New RelicData Processing and Alerts Delays in EU Region

    Between 00:36 UTC and 02:35 UTC on November 1st, 2024, some customers in the EU region may have experienced data visibility irregularities and delayed alert notifications for affected data. The interruption with our cloud provider has been resolved and affected services have returned to normal operations.

    Resolved

Oct 31, 2024

  1. MezmoPipeline Web UI is Unresponsive

    **Dates:** Start Time: Saturday, October 26, 2024 at 01:00 UTC End Time: Saturday, October 26, 2024 at 20:23 UTC Duration: 19 hours and 23 minutes \(Note that customer impact was limited to the first four hours of the incident.\) ‌ **What happened:** For approximately two hours, pages on the Web UI for Pipeline did not load. Afterwards, pages were able to load, but often only after delays of 15 seconds or longer. After another two hours, the Web UI returned to normal usage. The incident was kept open until all remediation was completed. The Web UI for Log Analysis and the ingress and egress of Pipeline data were unaffected. ‌ **Why it happened:** On Wednesday, October 23, 2024, we deployed a change to our Pipeline service, by which metrics on pipeline usage began to be saved to a postgres database. The same database also stores account configuration information; this information must be accessed to display pages in the Web UI. The deployment on Wednesday changed the performance profile of the database, most notably in the number and frequency of writes. There was no immediate customer impact, but we noted that backups of the database were unable to run successfully because of the increased load. Customer impact only began on Saturday, October 26th when an unrelated user action – running a Profiler from the Web UI – placed even more demands on the postgres database. It was unable to process queries and, according to its design, moved into “read-only” mode to prevent any loss of data. At times the database was entirely inaccessible or very slow to respond. This made our Web UI unable to display pages, as it relies on configuration information stored in the database. ‌ **How we fixed it:** We first enabled two replicas of the database, which allowed the Web UI to load pages again, albeit slowly. We then deployed a new code change that removed all superfluous writes about metric usage to the postgres database. This reduced the number of queries and Web UI usage returned to normal. We kept the incident open while working on further remediation. Finally we took steps to bring the postgres database back to a normal state. This remediation phase was complicated by the fact that a full backup had not completed successfully in the last two days. By the end of the incident, the database was operating normally and a full backup had completed. ‌ **What we are doing to prevent it from happening again:** The problems with the deployment on Wednesday, October 23, 2024 only revealed themselves under the load of our production environment. To better simulate production, we will update our testing environment and processes to use higher data loads. We will closely evaluate the current workload of the postgres database and consider making a separate database to store just metrics, rather than combining metrics and configurations in one location. Separate databases would have prevented Web UI pages from not loading, thus avoiding any customer impact. We will re-evaluate our backup strategy for the postgres database, since this slowed down the remediation phase.

    Resolved

  2. New RelicDelayed alert notifications for US Region

    Between 07:03 UTC and 07:15 UTC on October 31st, some customers in the US region may have experienced delayed or missing alert notifications. This issue has been resolved and impacted services have returned to normal operations.

    Resolved

Oct 30, 2024

Oct 29, 2024

  1. SpeechmaticsRT SaaS EU/US - Partial Outage

    A small percentage of sessions between 14:00 UTC to 14:03 UTC were dropped from our EU/US RT cloud due to an issue triggered during infrastructure maintenance. We have mitigated the issue.

    Resolved

  2. AppsFlyerDashboards latency issue

    We're happy to report that the latency issues have been resolved. Thank you for your patience. If you're still experiencing any related issues, please reach out to customer support.

    Resolved

  3. CodecovDelays with Report Processing

    A large backlog of reports caused the system to lag behind processing. This has been remedied and the system is now fully operational.

    Resolved

Oct 28, 2024

  1. TypeformTypeform Renderer - Forms intermittently unavailable

    We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

  2. TypeformTypeform Renderer - Forms intermittently unavailable

    We have identified a brief traffic interruption that affected the loading of typeforms in the renderer from 00:30 to 01:30 CET. During this period, some users may have intermittently encountered a "no healthy upstream" error while attempting to load their forms. We are glad to inform you that a fix was implemented and the incident was resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused. REF: ZD 808202 | INC-68

    Resolved

Oct 26, 2024

Oct 25, 2024

Oct 24, 2024

  1. AsanaIssues with signing up

    From 7 AM - 3 PM UTC time, there was a problem with signing up to Asana. This issue has now been resolved, and users should be able to sign up again.

    Resolved

Oct 23, 2024

  1. AirtableElevated rate of errors for API requests

    This incident has been resolved. We apologize for the disruption this may have caused you. Please reach out to Airtable support or your account team if you have any questions about this incident.

    Resolved

Oct 22, 2024

  1. GOV.UKGOV.UK: Local Links Manager broken

    Users were encountering error messages on GOV.UK pages, such as Report Child Abuse and Find a Local Council, when using the postcode lookup feature. Some received a 403 error stating, "You are not permitted to see this page," while others encountered a 503 error saying, "Sorry, we're experiencing technical difficulties."

    Resolved

Oct 21, 2024

  1. TypeformTypeform Results/Summary Report - Unavailable

    We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

Older incidents