Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Nov 3, 2022

Nov 1, 2022

Oct 31, 2022

  1. GradleJCenter downtime impact on resolution

    The new settings have resolved most of the issues. Note that you may still experience problems if the package only exists on JCenter and was never mirrored until now.

    Resolved

  2. GradleJCenter issues

    Impacts from the JCenter issue should be resolved. We will keep monitoring the situation.

    Resolved

Oct 28, 2022

Oct 25, 2022

  1. LogRocketSearch Performance Degradation

    We've worked around the issue, and will be performing a larger maintenance overnight but at this time the issue appears to be resolved.

    Resolved

Oct 24, 2022

  1. PyPIPyPI Backends are unavailable.

    ## Summary The cluster that hosts PyPI’s backends as well as multiple ancillary services experienced an outage during maintenance that interrupted access to services over HTTPS. ## Details From 2022-10-24 14:43 UTC until 2022-10-24 15:03 UTC, PyPI’s backends were not accessible over HTTPS. This interfered with our CDNs ability to fetch pages, made uploads to [uploads.pypi.org](http://uploads.pypi.org) impossible, and interrupted other services such as our legacy file redirect service. PyPI services run as deployments in a Kubernetes cluster and are exposed via Ingress with the AWS Elastic Load Balancer \(ELB\) integration. The TLS certificate that this load balancer uses is managed via Amazon’s Certificate Manager \(ACM\). When initially deploying our Kubernetes cluster, the Ingress managed ELB was configured to use an existing ACM TLS certificate. Earlier today, regular maintenance of our Kubernetes cluster required a rolling restart of all nodes in the cluster to distribute upgrades and new configurations to all nodes. During this rolling restart as the Kubernetes API server hosts were deployed, Kubernetes validated and refreshed the Ingress configurations we had previously defined. Since the most recent rolling upgrade, an additional hostname was needed for PyPI’s Ingress. PyPI administrators created a new ACM TLS certificate including that hostname and updated the Ingress managed ELB to use this new certificate. As a result, the new Kubernetes API servers were unable to find the previous ACM TLS certificate and disabled the HTTPS listener for the Ingress configuration that serves PyPI and associated services as they came online. Once identified, the PyPI admins updated the Ingress configuration to point to the new ACM TLS certificate and Kubernetes restored the HTTPS listener on the Ingress managed ELB, restoring access to all services. ## Mitigation We will investigate mechanisms by which the hostnames needed on ACM TLS certificates for PyPI’s Ingress configurations can be managed via Kubernetes resources rather than manually via the AWS console. By managing resources all via the Kubernetes API, drift between desired state and reality will be less likely to occur and surface during similar maintenance in the future.

    Resolved

Oct 21, 2022

  1. OpsgenieElevated error rate in all Opsgenie services

    Between 2022-10-21 09:01 UTC and 2022-10-21 09:12 UTC, in the US region, we started to see elevated error rate in our infrastructure due to faulty deployment. We have deployed a fix to mitigate the issue and have verified that all services have recovered without data loss. Thanks to quick reaction of our engineering team, issue has been resolved and the service is operating normally.

    Resolved

Oct 20, 2022

  1. OpsgenieSchedule overrides were not editable/deletable

    Between 2022-10-20 14:23 (UTC) and 2022-10-20 17:39 (UTC), due to latency in Opsgenie system, we were not allowing any edit or delete for schedule overrides and returning error to some of our users. After detecting the problem, our engineering team worked towards getting Opsgenie service back up as quickly as possible.

    Resolved

Oct 19, 2022

Oct 18, 2022

Oct 16, 2022

Oct 12, 2022

  1. MezmoDegraded performance for WebUI, Ingestion, Alerting, Searching, Live Tail, Graphing, and Timelines

    **Dates:** Start Time: Wednesday, October 5, 2022, at 14:27 UTC End Time: Wednesday, October 5, 2022, at 14:45 UTC Duration: 00:18 **What happened:** The ingestion of logs was partially halted. The WebUI was mostly unresponsive and most API calls failed. Because many newly submitted logs were not being ingested, new logs were not immediately available for Alerting, Searching, Live Tail, Graphing, Timelines, and Archiving. **Why it happened:** We recently added a new API gateway - Kong - to our service, that acts as a proxy for all other services. We had gradually increased the amount of traffic directed through the API gateway over several weeks and seen no ill effects. Prior to the incident, only some of the traffic for ingestion wen through the gateway. Kong was restarted after a routine configuration change. After the restart, all traffic for our ingestion service began to go through Kong. Our monitoring quickly revealed the Kong service did not have enough pods to keep up with the increased workload, causing many requests to fail. **How we fixed it:** We manually added more pods to the Kong service. Ingestion, the WebUI, and API calls began to work normally again. Once ingestion had resumed, LogDNA agents running on customer environments resent all locally cached logs to our service for ingestion. No data was lost. **What we are doing to prevent it from happening again:** We updated Kubernetes to always assign enough pods for the Kong API gateway service to be able to handle all traffic. We’ll update the Kong gateway to more evenly distribute ingestion traffic across available pods. We will adjust our deployment processes so pods are restarted more slowly, which will reduce the impact in a similar scenario. We’ll explore autoscaling policies so more pods could be added automatically in a similar situation.

    Resolved

Oct 11, 2022

  1. PyPIPyPI UI missing CSS

    All CSS appears to be loading, we will investigate what occurred that caused CSS to fail to load during deployment.

    Resolved

Oct 10, 2022

Sep 29, 2022

Sep 28, 2022

Sep 27, 2022

  1. Rev AIStreaming API Connection Errors

    We opened the incident out of an abundance of caution. Our automated systems fixed the issue before it reached any customers. No impact.

    Resolved

Sep 23, 2022

  1. PyPITestPyPI Maintenance

    This maintenance is complete. Please file an issue at https://github.com/pypa/pypi-support/issues/new?assignees=&labels=network&template=access-issues.yml if you experience issues with test.pypi.org after this migration.

    Resolved

Sep 22, 2022

  1. PandaDocunable to pull information from Zoho

    We're all set! Users can now connect to Zoho CRM. If you continue to experience any issues with this, please reach out to us at support@pandadoc.com.

    Resolved

Sep 21, 2022

Sep 14, 2022

Sep 12, 2022

Sep 9, 2022

Sep 2, 2022

Older incidents