Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Mar 27, 2023

Mar 25, 2023

  1. HoneybadgerUnhealthy API server

    Starting around 3:30pm PST, we had an incident where an unhealthy API server was in rotation in our load balancer. During that time requests routed to that server responded with a 502 which could have resulted in the clients dropping notices. At around 5:30pm PST we removed the unhealthy server from rotation. 7:22pm PST edit: Reworded message to describe that notices are not queued on the client, but dropped when the server responds in an error state

    Resolved

Mar 24, 2023

  1. Travis CIBuild job errors when acquiring repository from GitHub (GitHub SSH key update)

    The issue seems resolved. If you encounter further roadbloacks in repositories using Custom SSH Key in Travis CI, specifically on git repository cloning stage, please validate if a matching key pair is still available at GitHub. If not, removing old Custom SSH Key in Travis CI repository settings and regenerating the Custom SSH Key in Travis CI may help unblock your build jobs.

    Resolved

  2. HoneybadgerIssues processing backlog

    We identified that our workers had an issue deploying. We deployed a fix and upped our capacity to deal with the current backlog. Everything is back to normal now. Turns out this issue was related GitHub rotating their public RSA key: https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/

    Resolved

Mar 23, 2023

  1. Atlassian StatuspageStatuspage was unable to accept new signups

    Between 9:09 PM PST on March 22nd and 9:46 PM PST on March 23rd, there was an issue preventing new signups for Statuspage. Customers attempting to sign up during that time may have encountered difficulties. We have since identified and resolved the issue, which only affected new signups and did not impact any existing Statuspages or their components. We appreciate your patience and understanding while we addressed this matter.

    Resolved

Mar 22, 2023

  1. DopplerDoppler CLI Failing to Install From OS Package Registries

    An incident was identified at our provider for OS package registry downloads. They've fixed the issue and installs from OS package registries are available again. Impact: The incident affected all Doppler users attempting to install the Doppler CLI via their OS package registeries. This resulted in disruptions to their workflows relying on the Doppler CLI. Resolution: Cloudsmith resolved the incident after around an hour, and users were once again able to install the Doppler CLI from their OS package registries.

    Resolved

  2. UserVoiceEmails Not Being Received

    The cause was identified and the issue has been resolved. Issue: The email template configured for these types of notification emails had an error during editing. Fix: The template has been reverted back to a working version.

    Resolved

Mar 21, 2023

Mar 20, 2023

Mar 18, 2023

  1. PyPIPyPI Emails Not Sending

    ## Summary At 20:36 UTC on 2023-03-17, PyPI’s email provider enforced a suspension against our account. After our remediation efforts and response, our account was restored at 00:36 UTC 2023-03-18. In response, the PyPI Administrators did the following: * Audited our email logs, including delivery and complaint status to determine the underlying factor that had led to our suspension. * Rolled out updates to our signup form to mitigate the behavior that was leading to complaint emails. * Notified our provider of our remediation efforts * Rolled out updates to our retry logic to report exceptions in tasks that were being swallowed by retry logic. The following was determined in the course of our response: * Our monitoring was blind to email failures due to retry logic swallowing the apparently transient errors, so response was delayed by nearly 15 minutes. * An account review was initiated 2021-09-04 by our provider due to a marginal complaint rate. This had been considered at the time and PyPI Administrators somewhat ironically began sending more valid notification emails whenever reasonable to bump our reputation. PyPI’s email sending pattern is consistently low volume aside from routine account signup and password resets. After rolling these changes out, it wasn’t clear if “enough” had been done and it eventually fell out of the collective mind of the PyPI administrators. * Our China accessible \(not reCaptcha as it is blocked there\) bot detection honeypot had been letting just enough bad signups through to maintain a 0.6% complaint rate from victims of whatever malfeasance the bots were enabling. ## Impact Actions that triggered emails during the outage window from 2023-03-17T20:36Z - 2023-03-18T00:36Z were delayed in sending . All emails enqueued during that time were delivered by retry before 2023-03-18T01:00Z \(up to 4.5 hours delayed\). ## Timeline * 2018-03-08: [Issue](https://github.com/pypi/warehouse/issues/3174) filed regarding inability to signup for PyPI from China due to reCaptcha being blocked. * 2018-03-21: Implementation of a relaxed bot detection in place of reCaptcha [merged](https://github.com/pypi/warehouse/pull/3339#event-1534405160) and deployed. * 2021-09-04: Report of account review received from our email provider. * 2023-03-17: Account suspension enforced by our email provider. * 2023-03-17: reCaptcha [re-enabled](https://github.com/pypi/warehouse/pull/13232) for our signup form. * 2023-03-17: Error reporting for exceptions in failed tasks [merged](https://github.com/pypi/warehouse/pull/13236) and deployed. ## Future Work * Investigate bot detection mechanisms that are robust _and_ available to users in China. * Research backup mail providers for failover. * Send more email to retain reputation.

    Resolved

Mar 16, 2023

Mar 15, 2023

Mar 14, 2023

Mar 12, 2023

  1. SvixMost routes 5xx in the US region [resolved]

    We are still investigating, but here is our more complete update regarding this incident: [https://www.svix.com/blog/we-had-a-partial-outage/](https://www.svix.com/blog/we-had-a-partial-outage/)

    Resolved

Mar 10, 2023

Mar 9, 2023

  1. HoneybadgerWeb app unavailable

    We had an incident (from 1:00 to 1:27 PST) where our apps were unresponsive. They have since recovered.

    Resolved

Mar 4, 2023

Mar 2, 2023

  1. DopplerDoppler CLI Failing to Install Due to GitHub Incident

    For several hours on February 27, 2023, users intermittently encountered errors when attempting to install the Doppler CLI. This issue was due to a related incident on GitHub, where Doppler hosts all CLI binaries and signatures. According to GitHub's incident report, GitHub experienced degraded performance and increased error rates for their packages service. Impact: The incident affected all Doppler users attempting to install the Doppler CLI. This resulted in disruptions to their workflows and GitHub actions relying on the Doppler CLI. Resolution: GitHub resolved the incident after a few hours, and users were once again able to install the Doppler CLI.

    Resolved

Feb 28, 2023

Feb 27, 2023

Feb 26, 2023

Feb 25, 2023

  1. SvixElevated API errors (tiny fraction of requests)

    The issue has been fixed, and we finished monitoring. There was an issue with one of our redis shards acting up due to an issue with the code implementing endpoint rate limiting. We are working on a long term fix.

    Resolved

Feb 22, 2023

Feb 21, 2023

  1. MezmoSearches are running slowly

    **Dates:** Start Time: Friday, February 10, 2023 at 16:45 UTC End Time: Friday, February 10, 2023 at 18:14 UTC Duration: 89 minutes **What happened:** Searches returned results slowly or not at all. No data was lost and ingestion was not halted. ‌ **Why it happened:** In a previous incident on February 6, 2023 \(more details at [https://status.mezmo.com/incidents/3yl9x1t7qcw5\),](https://status.mezmo.com/incidents/3yl9x1t7qcw5),) two pods storing logs were temporarily removed from the pool of pods available for receiving and inserting new batches of logs into our data store. The pods continued to return results for previously processed logs. We took this step because the pods had fallen behind on their tasks, which we believe was a consequence of an ungraceful shutdown during the incident. We gave the pods several days to catch up on tasks and then made them available for insertion of new logs into the data store again, a change we expected to have no impact. One of the pods immediately began integrity checks to confirm the same data existed on its local disk and on our S3 storage. As a side effect of the previous incident, the pod incorrectly determined that data was missing from the local disk and began sending http requests to our S3 storage to locate the missing data. In fact, the data in question is designed to only reside on local disk and was not supposed to be stored on S3. The requests failed with 404 errors when the data was not found on S3 \(as expected\). Every new attempt to retrieve search results generated another request. The rate of requests was high enough to slow down all requests related to search results within the pod’s zone \(one out of three total\). This led to search results being returned slowly or not at all. ‌ **How we fixed it:** We removed the pod from the pool available for receiving and inserting new batches of logs into our data store. The pod continued to return results for previously processed logs. ‌ **What we are doing to prevent it from happening again:** We marked this pod to remain unavailable for new logs until all previously processed logs on the pod have passed their retention period, whose maximum is 30 days. At that time, the pod will be rebuilt and begin accepting newly submitted logs again. We’ll fix the logic of our search engine so it doesn’t request data from S3 that is intentionally not stored there. This will prevent the widespread 404 errors that slowed down all searching, should a pod again incorrectly determine it is missing data from its local disk. We have added alerting and monitoring to detect high latency in search speeds and the average time to compact newly inserted logs.

    Resolved

  2. Atlassian StatuspageDelayed delivery of Incident notifications via email

    We experienced a delay in delivery of notifications of incidents for a period of 15 minutes which has been resolved. We have identified the cause of this is an ongoing incident with one of our messaging vendors, and we have migrated all possible notification traffic to our alternate messaging vendors.

    Resolved

Older incidents