MailjetPassport Builder Inaccessible
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Due to Wordpress maintenance, the http://www.workramp.com/ and http://blog.workramp.com/ pages were down for 2-3 minutes but was restored once the maintenance was completed.
Resolved
The new settings have resolved most of the issues. Note that you may still experience problems if the package only exists on JCenter and was never mirrored until now.
Resolved
Impacts from the JCenter issue should be resolved. We will keep monitoring the situation.
Resolved
We're all set! Users can now create documents. If you continue to experience any issues with this, please reach out to us at support@pandadoc.com.
Resolved
This incident has been resolved.
Resolved
We've worked around the issue, and will be performing a larger maintenance overnight but at this time the issue appears to be resolved.
Resolved
## Summary The cluster that hosts PyPI’s backends as well as multiple ancillary services experienced an outage during maintenance that interrupted access to services over HTTPS. ## Details From 2022-10-24 14:43 UTC until 2022-10-24 15:03 UTC, PyPI’s backends were not accessible over HTTPS. This interfered with our CDNs ability to fetch pages, made uploads to [uploads.pypi.org](http://uploads.pypi.org) impossible, and interrupted other services such as our legacy file redirect service. PyPI services run as deployments in a Kubernetes cluster and are exposed via Ingress with the AWS Elastic Load Balancer \(ELB\) integration. The TLS certificate that this load balancer uses is managed via Amazon’s Certificate Manager \(ACM\). When initially deploying our Kubernetes cluster, the Ingress managed ELB was configured to use an existing ACM TLS certificate. Earlier today, regular maintenance of our Kubernetes cluster required a rolling restart of all nodes in the cluster to distribute upgrades and new configurations to all nodes. During this rolling restart as the Kubernetes API server hosts were deployed, Kubernetes validated and refreshed the Ingress configurations we had previously defined. Since the most recent rolling upgrade, an additional hostname was needed for PyPI’s Ingress. PyPI administrators created a new ACM TLS certificate including that hostname and updated the Ingress managed ELB to use this new certificate. As a result, the new Kubernetes API servers were unable to find the previous ACM TLS certificate and disabled the HTTPS listener for the Ingress configuration that serves PyPI and associated services as they came online. Once identified, the PyPI admins updated the Ingress configuration to point to the new ACM TLS certificate and Kubernetes restored the HTTPS listener on the Ingress managed ELB, restoring access to all services. ## Mitigation We will investigate mechanisms by which the hostnames needed on ACM TLS certificates for PyPI’s Ingress configurations can be managed via Kubernetes resources rather than manually via the AWS console. By managing resources all via the Kubernetes API, drift between desired state and reality will be less likely to occur and surface during similar maintenance in the future.
Resolved
Between 2022-10-21 09:01 UTC and 2022-10-21 09:12 UTC, in the US region, we started to see elevated error rate in our infrastructure due to faulty deployment. We have deployed a fix to mitigate the issue and have verified that all services have recovered without data loss. Thanks to quick reaction of our engineering team, issue has been resolved and the service is operating normally.
Resolved
Between 2022-10-20 14:23 (UTC) and 2022-10-20 17:39 (UTC), due to latency in Opsgenie system, we were not allowing any edit or delete for schedule overrides and returning error to some of our users. After detecting the problem, our engineering team worked towards getting Opsgenie service back up as quickly as possible.
Resolved
This incident has been resolved.
Resolved
The problematic node has been replaced in our search cluster and performance should be returning to normal levels.
Resolved
This incident has been resolved.
Resolved
**Dates:** Start Time: Wednesday, October 5, 2022, at 14:27 UTC End Time: Wednesday, October 5, 2022, at 14:45 UTC Duration: 00:18 **What happened:** The ingestion of logs was partially halted. The WebUI was mostly unresponsive and most API calls failed. Because many newly submitted logs were not being ingested, new logs were not immediately available for Alerting, Searching, Live Tail, Graphing, Timelines, and Archiving. **Why it happened:** We recently added a new API gateway - Kong - to our service, that acts as a proxy for all other services. We had gradually increased the amount of traffic directed through the API gateway over several weeks and seen no ill effects. Prior to the incident, only some of the traffic for ingestion wen through the gateway. Kong was restarted after a routine configuration change. After the restart, all traffic for our ingestion service began to go through Kong. Our monitoring quickly revealed the Kong service did not have enough pods to keep up with the increased workload, causing many requests to fail. **How we fixed it:** We manually added more pods to the Kong service. Ingestion, the WebUI, and API calls began to work normally again. Once ingestion had resumed, LogDNA agents running on customer environments resent all locally cached logs to our service for ingestion. No data was lost. **What we are doing to prevent it from happening again:** We updated Kubernetes to always assign enough pods for the Kong API gateway service to be able to handle all traffic. We’ll update the Kong gateway to more evenly distribute ingestion traffic across available pods. We will adjust our deployment processes so pods are restarted more slowly, which will reduce the impact in a similar scenario. We’ll explore autoscaling policies so more pods could be added automatically in a similar situation.
Resolved
This started at about 8:25 AM UTC. We identified and addressed the issue at about 2:36 PM UTC. The issue has been resolved.
Resolved
All CSS appears to be loading, we will investigate what occurred that caused CSS to fail to load during deployment.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
AWS is showing that the incident has been resolved. Our engineering team is continuing to monitor this situation, but we have observed improvements on transcription turn around times.
Resolved
This incident has been resolved.
Resolved
Between 5:09 to 5:13 PM PST, public status pages experienced elevated errors due to increased traffic.
Resolved
We opened the incident out of an abundance of caution. Our automated systems fixed the issue before it reached any customers. No impact.
Resolved
This maintenance is complete. Please file an issue at https://github.com/pypa/pypi-support/issues/new?assignees=&labels=network&template=access-issues.yml if you experience issues with test.pypi.org after this migration.
Resolved
We're all set! Users can now connect to Zoho CRM. If you continue to experience any issues with this, please reach out to us at support@pandadoc.com.
Resolved
This incident has been resolved and users should no longer be experiencing increased response times from our APIs.
Resolved
This incident has been resolved.
Resolved
We're all set! If you continue to experience any issues with this, please reach out to us at support@pandadoc.com.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
All mitigations have now been implemented and completed.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved