Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Sep 2, 2022

Aug 30, 2022

Aug 29, 2022

  1. DopplerSecrets unable to be read

    This issue was caused by a redis server running out of memory. We've increased the amount of memory allotted to this server. We'll also be setting up additional alerting on redis resource usage so that we can catch and prevent these issues in the future.

    Resolved

Aug 28, 2022

Aug 26, 2022

Aug 24, 2022

Aug 23, 2022

  1. PandaDocTrouble Opening Documents

    We're all set! If you continue to experience any issues with this, please reach out to us at support@pandadoc.com.

    Resolved

Aug 22, 2022

  1. PandaDocPandaDoc Performance Degradation

    We're all set! The application is fully operational now. If you continue to experience any issues with this, please reach out to us at support@pandadoc.com.

    Resolved

Aug 18, 2022

Aug 17, 2022

Aug 16, 2022

  1. SvixElevated errors for create message requests with a UID set (US region)

    Due to an internal AWS issue the UID mechanism was failing for some of the requests with a UID set (only US region affected). We had an issue with our fail-safe mechanism which was causing it to return 500 errors even though the messages were correctly processed and sent.

    Resolved

Aug 10, 2022

Aug 8, 2022

Aug 3, 2022

Aug 2, 2022

Aug 1, 2022

  1. DopplerUpdating secrets fails for a limited number of environments

    An error was discovered while running a data backfill job that resulted in users being unable to update secrets for affected environments. The data that was generated triggered an unexpected assertion in the secrets validation code. Users that attempted to save secrets in these environments would have received a 500 response from the server with the message "An error has occurred but don't fret. Our team has been notified." The data has been fixed for all environments and secrets write operations are fully functional again.

    Resolved

Jul 29, 2022

Jul 28, 2022

  1. PyPIPyPI Outage 2022-07-28 18:46-18:53 UTC

    PyPI experienced an outage in its backends from 18:46-18:53 UTC on 2022-07-28. This outage stemmed from a cascading failure in the backend infrastructure that disrupted the management layer that deploys and secures PyPI's application servers. ## Details Beginning at 17:24 UTC multiple cloud instances that are members of the Kubernetes cluster that PyPI runs on were automatically replaced by the provider due to an unknown failure. By 18:20 UTC all instances were replaced and healthy as part of the Kubernetes cluster. This replacement event caused the system to reshuffle _many_ pods which impacted both our Consul and Vault clusters inside Kubernetes. Both services were restarted across new nodes. At 18:23 UTC the first alert was delivered to PSF Infrastructure to notify that at least one Vault container was down and required unsealing to come back online. Upon investigation it was determined that all Vault containers in the cluster were unavailable and that the Consul containers were unable to establish a new leader. The responding Infrastructure Team member worked to bring Consul back online, then unseal the new Vault containers. Complications arose in bringing Consul back into a healthy state as one of the pods was stuck in the “Terminating” state, which caused the discovery mechanisms for Consul servers to get stuck expecting an additional server to participate in elections. Once this pod was forcefully removed and allowed to be re-created, recovery commenced. These backing services being offline cause new containers launched to serve PyPI’s applications to fail. As a result, containers rescheduled due to node replacements were unable to start. This eventually caused PyPI to become unresponsive at 18:46 UTC as more and more containers were rescheduled by Kubernetes in an attempt to meet the specified instance count.

    Resolved

Jul 21, 2022

  1. Atlassian StatuspageErrors accessing manage portal

    A recent deploy was found to contain errors. Our infrastructure has been successfully rolled back to a previous version of the code, and traffic is being served as normal again.

    Resolved

Jul 20, 2022

Jul 18, 2022

Jul 13, 2022

  1. HoneybadgerNetwork disruption

    We had a network disruption that lasted approximately one minute. The cause has been identified and resolved.

    Resolved

Jul 6, 2022

Jul 4, 2022

Jul 1, 2022

  1. MezmoSome customers' logs are not currently being processed

    **Dates:** Start Time: Thursday, June 30, 21:40 UTC End Time: Thursday, June 30, 23:32 UTC Duration: 1 hour and 52 minutes **What happened:** Some log lines for some customers were discarded by our service. The log lines were successfully accepted by our ingestion service, but a downstream service – the parser – removed some of them. All further downstream services, such as Alerting, Live Tail, Searching, and Archiving never received these logs. In some cases, lines were received by Live Tail and were appended with the phrase “\(not retained\)”. The great majority of customers – 94.2% – were unaffected and had no log lines discarded. Approximately 3.5% had a relatively small number of log lines discarded. Approximately 2.3% had most or all of the log lines submitted during the incident discarded. **Why it happened:** We inadvertently released code into production that contained a bug in the parser service. This bug was known to us and in the process of being fixed in our development environment, but was not yet ready for release to production. The parser service is where exclusion rules are applied to recently submitted log lines that have been ingested but not yet passed to downstream services \(e.g. Alerting, Live Tail, Searching, and Archiving\). The bug made the parser exclude log lines that matched rules for inactive exclusion rules. This included exclusion rules made by customers in the past and then disabled. Customers with such rules had some log lines excluded: whichever lines matched the inactive rules. If those rules had the “Preserve these lines for live-tail and alerting” option enabled, then the excluded lines would still be processed for alerts and appear in Live Tail with the phrase “\(not retained\)” appended. This affected 3.5% of our customer accounts. The usage quota feature is implemented as a particular type of exclusion rule even though it is not presented in the UI as an exclusion rule. The bug made the parser exclude all log lines if the usage quota feature was enabled for an account. This affected 2.3% of our customer accounts. Our monitoring did not detect the decrease in lines being passed from the parser to downstream services because the change was within the range of normal fluctuation rates. These rates vary significantly as traffic changes and as customers choose to enable/disable exclusion rules. **How we fixed it:** We reverted the last release of parser code to the previous version. Once the previous version was deployed to all pods running the parser service, log lines stopped being discarded. **What we are doing to prevent it from happening again:** We added a code level test to ensure inactive exclusion rules are never applied by the parser \(such tests are part of our standard operating procedure\). We will review our release process to understand how the code containing the bug was moved into production and improve our processes to prevent a similar event in the future.

    Resolved

Jun 30, 2022

Jun 29, 2022

Jun 28, 2022

  1. PyPIQueue delay affecting CDN purges and emails

    ### Summary PyPI initially experienced delays in features which used a centralized task queue \(such as purging the CDN cache and sending emails\) due to a poorly performing task type. This was followed by an outage due to the combination of the steps taken to mitigate the queue issue and unrelated changes that affected PyPI’s performance, resulting in PyPI becoming unavailable via the web or for installations. ### Details On Thu, 23 Jun 2022 at 04:55 UTC, PyPI admins deployed a change which introduced a new periodic task designed to aggregate statistics about 2FA usage on PyPI. This task was designed to run every 5 minutes and sampled multiple counts of different characteristics of 2FA usage. This task used a database query that attempted to count a large number of rows, and had poor performance, causing high database utilization and a long processing time for the task. However, while long-running, the task was finishing and publishing metrics, and so PyPI admins were unaware of its poor performance. Over time, however, the tasks that workers were actively working on began to include more and more of these poorly-performing tasks, exhausting the number of available workers to process other tasks. On Sun, 26 Jun 2022 at 17:29 UTC, PyPI admins deployed an unrelated change which replaced the JSON serializer used by PyPI from the standard library JSON serializer to a third-party JSON serializer, with benchmarks indicating that this would result in a 6x savings on time to serialize JSON, and thus a significant savings on response time for PyPI’s various JSON-rendering endpoints.  At 21:06 UTC, PyPI admins began to receive reports that newly published versions were not being immediately updated in the web frontend for PyPI, not installable, and that emails were delayed. However, because the queue backlog was sufficiently small at this point, this small number of early reports were resolved within an hour and attributed to transient errors. Over the course of the following 12 hours, which coincided with the US EST evening, night and early morning, PyPI admins received an increasing stream of user reports that emails were delayed and project pages were not updating. On Mon, 27 Jun 2022 at 15:43 UTC, PyPI admins determined that PyPI’s task queue had a significant backlog, with more than 46K pending tasks, with the oldest task being more than 14 hours old, and began attempting to mitigate the backlog. PyPI admins determined that the backlog correlated with an increase in CPU usage by the database, and that there were at least two poorly-performing queries: one which had been introduced in the task for the 2FA metrics, and another which was due to a historical implementation detail of how PyPI projects are modeled in the database, where normalized project names are not stored in the database, but generated upon request based on the un-normalized project name To mitigate the load on the database, PyPI admins immediately began canceling the long-running database queries that resulted from the poorly performing task, in an attempt to remove as many as possible from the queue and free up workers to complete other tasks. Simultaneously, admins prepared two separate changes to mitigate the two poorly-performing queries: one to temporarily disable the poorly performing metrics task, and a second to add a column in the database which stored the normalized project name. At 15:53 UTC, PyPI admins deployed the first change that disabled the 2FA metrics task, which resulted in stopping additional long-running queries from being created. Admins continued to manually reap the poorly performing query as PyPI’s workers began to drain the backlog of other tasks. At this time, the size of the queue began to decrease. At 16:19 UTC, PyPI admins deployed the second change which added a column for the normalized project name. This included a large migration which unexpectedly locked PyPI’s database for a significant amount of time, resulting in PyPI backends being unable to create new database connections, becoming unhealthy, and causing responses to fail. At 16:45 UTC, the migration finished and PyPI backends began to recover, however PyPI still continued to intermittently return errors and did not become fully healthy, which was not explicable based on the previous mitigations. After investigation, PyPI admins determined that workers were crashing due to exceeding their allotted memory usage. Admins determined that the natural increase in requests after having an outage resulted in a large amount of memory being consumed by the new JSON serializer, which had higher memory footprint than the previous JSON serializer. At 17:41 UTC, PyPI admins deployed a change reverting the new JSON serializer back to the original serializer, and backends eventually returned to healthy status. ### Impact As a result of the task queue becoming increasingly large, user-facing tasks such as CDN purges and emails became more and more delayed. Delayed CDN purges meant that the PEP-503 ‘simple’ API was not getting updated and installers were unable to become aware of newly-published packages or new versions of existing packages. Additionally, the web frontend was not displaying new versions or new packages. Users attempting to create accounts and verify email addresses were unable to do so as well. At the peak, the task queue had nearly 50K pending tasks, and the oldest task in the queue was more than 14 hours old, whereas this queue normally has less than 100 pending tasks on average and tasks are completed in less than 1 second. ![](https://dustingram.com/assets/images/pypi-incident-jun-27-2022/1.png) ![](https://dustingram.com/assets/images/pypi-incident-jun-27-2022/2.png) As a result of the locked migration, PyPI backends became unhealthy and served 500-level errors to users, preventing much of the web traffic, API traffic and uploads from succeeding for approximately 40 minutes. ![](https://dustingram.com/assets/images/pypi-incident-jun-27-2022/3.png) ![](https://dustingram.com/assets/images/pypi-incident-jun-27-2022/4.png) As a result of the increased memory usage due to the new JSON serializer, the PyPI backends did not fully recover from the prior outage and continued to return intermittent errors as pods consumed their available memory and were terminated. During the entire process, PyPI deployments were delayed due to their dependence on a successful run of a shared CI pipeline which had a significant backlog at the time changes were attempting to be introduced. This impacted the delivery times for changes to mitigate the issues. ### Future work During the outage, it was identified that the service PyPI uses for alerting and monitoring was insufficiently configured to notify the team of an excessively large task queue, which would have given administrators early warning that the 2FA metrics task was failing to complete and that delays were increasing. Additionally, the monitoring system lacked alerts for high CPU utilization in the database which was indicative of poorly performing queries. Alerts for both of these have already been added. When PyPI admins re-enable the 2FA metrics task, it will be redesigned to speed up queries by denormalizing counts rather than naively counting rows. In order to mitigate delays in PyPI’s CI pipeline, PyPI admins will migrate PyPI’s source repository to a separate organization, in order to ensure that the CI queue remains small and changes can be quickly deployed. In order to prevent long-running migrations from locking the database for extended periods of time, PyPI’s migration runner will be modified to make migrations safer, including adding timeouts after a certain short period of time rather than block for an arbitrarily long time. In order to mitigate complications when attempting to recover from an outage, PyPI admins will implement a policy to ensure that changes which are not directly related to the recovery efforts will not be merged until the outage has been resolved.

    Resolved

Jun 27, 2022

Jun 24, 2022

Older incidents