PyPIFile Uploads returning 500 responses
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
# What happened On May 1st, 2023, between the hours of 08:23 UTC and 15:08 UTC, imgix experienced intermittent errors affecting a small percentage of non-cached renders. # How were customers impacted? During the affected period, a small percentage of requests to the Rendering API returned a `502` or `503` error for non-cached requests. Errors slowly and gradually increased, with <.5% of requests returning an error at the height of the incident. # What went wrong during the incident? Our upstream provider experienced communication issues between CDN POPs, causing intermittent `502`/`503` responses in a small percentage of requests to our Rendering API. The increase in errors was so minor that it did not meet our monitoring thresholds for triggering alerts. One of our engineers observed a slow increase in errors and alerted other team members to a potential issue with our service. After tracing the issue to our upstream provider, we pushed a patch to mitigate intermittent connectivity issues, resolving the incident. # What will imgix do to prevent this in the future? We have refined our alerting to better catch the slowly increasing error rates. We have also ensured that the root cause of this incident has been fixed by our upstream provider. We are also updating our traffic routing in the case that the upstream issue occurs again.
Resolved
After working with our vendor we believe things stabilized overnight. We will be monitoring throughout the day on Thursday.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
**Dates:** Start Time: Monday, May 1, 2023, at 19:55 UTC End Time: Monday, May 1, 2023, at 20:11 UTC Duration: 16 minutes **What happened:** The WebUI was unresponsive, returning an error of “failure to get a peer from the ring-balancer.” **Why it happened:** All Mezmo services run within a service mesh. The portion of the mesh dedicated to the pods running our Mongo database began receiving many connection requests, more than its allocated CPU and memory could handle at once. This portion of the mesh \(which itself runs on pods\) quickly ran out of memory. This made the Mongo database unavailable to other services. The WebUI relies entirely on Mongo for account information and therefore became unresponsive, returning an error of “failure to get a peer from the ring-balancer.” While the immediate reason for the incident is clear, the root cause is still unknown. We suspect there was a change in user usage patterns \(e.g. increased traffic, login attempts, etc\) which triggered the incident. **How we fixed it:** We removed the WebUI from the service mesh. The Mongo service has more CPU and memory resources allocated to it and was able to accept the high level of connection requests successfully. WebUI usage immediately returned to normal. **What we are doing to prevent it from happening again:** We will change the default settings for the service mesh to allocate more CPU and memory resources, permanently. Afterwards, we will add the Mongo service back to the service mesh.
Resolved
This incident had been resolved, the issue was identified and currently marked as resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We're all set! If you continue to experience any issues with the application, please reach out to us at support@pandadoc.com.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
### **SUMMARY** On April 4, 2023, between 13:32 and 14:50 UTC, Atlassian customers using Opsgenie faced significant delays while receiving Android push notifications. This was caused by an incident in a third party messaging service, which is responsible for Android push notification delivery. This in turn affected our systems. The incident was immediately detected by our monitoring tools, our on-call engineers were paged, and at 14:50 UTC our systems recovered successfully. The total time to resolution was about 80 minutes. ### **IMPACT** The overall impact was between April 4, 2023, 13:32 - 14:50 UTC in Opsgenie_._ The incident only resulted in delays in Android push notifications only, and these notifications were delivered successfully after FCM service was restored and no data loss occurred. ### **ROOT CAUSE** The issue was caused by an incident in a third party messaging service, which is responsible for delivering push notifications to Android devices. **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. The impact was immediately caught by our monitoring tools, and the responsible team immediately started analysis of incident. We value transparency with our customers and will continue to notify you and take any necessary actions promptly during an incident. In order to handle degradation or outage of messaging channels, Opsgenie recommends that users configure multiple channels of message delivery - including push notifications, mobile SMS, phone calls, and email. In order to improve our response for the future, we will also be analyzing whether we can employ autoscaling solutions for our systems in case of an outage/high load related to one notification channel. We apologize to customers whose services were impacted during this incident. Thanks, Atlassian Customer Support
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This issue has been resolved following the mitigation rollout. All DNS resolutions are stable in the us-west region, and runtime issues are confirmed to be resolved.
Resolved
## A summary of what happened At **14:01 PDT Friday, April 7th** our monitoring indicated that our Public API request rate dropped and health checks didn’t pass through. The situation deteriorated rapidly and we noticed that some of our API endpoints became unresponsive, which impacted the availability of the PandaDoc platform. We followed our protocol and immediately started our incident response procedure, rolled back recent updates, and involved engineers in multiple investigation paths. After we had dismissed some initial theories, we understood that we had an issue connected with something on the infrastructure level and started investigating this together with our cloud provider \(AWS\). After a deep investigation that lasted several hours, we were able to track down the issue to network problems: several pods on a specific Kubernetes node were experiencing intermittent low-level network issues that caused connection leaks - repeatedly opening connections without closing them, or at least closing only some of them - which eventually led to increased latency and memory consumption and resulted in some of our core services entering a chain of crashes. As a consequence, Application, and API were not available during the downtime. Once the root cause was identified, the broken machine was removed from the cluster and the system started operating normally. The issue was fully resolved by **01:23 PDT, April 08**. ## A deep dive - how we investigated the root cause When the incident started, we noticed a spike in the number of connections in our database pool and many API calls waiting for connections to be released so they could process incoming requests. We quickly figured out that what was stopping connections from being released was a large series of uncommitted transactions that were just waiting idle. We then started analyzing database locks and deadlocks because it is usually what might lead to this behavior and wrote a hotfix to one of our API endpoints to reduce the number of processed events expecting this would release connections faster. Soon after this, we understood that the database was not a bottleneck, although we still had stalled transactions growing and the connections in the pool being taken and not released. We ran a deeper analysis of API endpoints metrics which revealed that external calls within the transactions could be the culprit. After more investigation, we found similarities in the API calls that were not responsive - they all interacted with our message queue \(RabbitMQ HA cluster\). The RabbitMQ cluster was working without any disruptions for the last 1.5 years, and monitoring was not showing anything suspicious. It did not seem like a cause since queues are processing messages independently in async mode \(that’s why they are used to offload tasks to be executed asynchronously later\), but we still decided to look into it closer. After analyzing machines in the clusters and connecting to them directly we saw that they were shutting down and reloading periodically, although this was not visible in the cluster monitoring in our Grafana dashboards, nor did we get any alerts. Since the message queue was unresponsive it led API calls to sit and wait for connection which led to blocking transactions which led to increasing in blocked connections in the database connection pool which led to waiting for other API requests for a new connection forever, in a loop that caused a chain of failures. We immediately started addressing the situation by scaling the cluster up vertically and upgrading the machines it's running on with more processing power and networking capabilities. After the upgrade, we’ve added additional monitoring metrics to the cluster. In parallel, we were leading an investigation into a probable root cause: intermittent networking issues on a kubernetes node that were causing pods on that node to repeatedly open connections without closing them. We investigated deeper and realized that an underlying networking issue was the most probable root cause after we observed and correlated several facts: * We had randomly missing metrics in our Prometheus monitoring relative to several systems, coinciding in time with the degradation of the RabbitMQ cluster metrics \(the number of sockets started growing linearly\) * We found that all pods on one particular Kubernetes node \(that was added to the cluster on Friday morning\) were having trouble connecting to other parts of the system \(our NATS cluster\). We also noticed error patterns in logs related to closed network connections or client timeouts, in numbers higher than normal. At the same time, we also observed that the number of slow NATS consumers was growing abnormally since the start of the incident * Most of the connections to the RabbitMQ nodes during the incident period were coming from pods that were residing in the faulty node. Once the broken machine was removed from the cluster, the system started operating normally. To sum up: we consider the main cause of the incident to be a problem with an AWS EC2 instance provisioned as part of our EKS \(managed k8s cluster\) that occurred during the normal process of a release. There were network-related errors that caused a number of connection issues on the RabbitMQ cluster leading to a chain failure. ## What we have done and will be doing next As our investigation wraps up, we want to highlight our continuous improvement mindset, and to provide clarity on what we are doing to improve our systems: * We have improved the robustness and scale of our rabbitMQ cluster to reduce the likelihood of failure in case of a growing number of network connections and reviewed the HA RabbitMQ setup and its replication settings * We’ve added additional logging and metrics to our RabbitMQ cluster, as well as early detection alarms for any deviation in network traffic patterns for the cluster * We’ve engaged AWS in the investigation and resolution of this outage. AWS support is running their own investigation about the issue * We’ll do further improvements in our Observability stack, with a review of which additional metrics we can add to improve the detection of underlying problems in AWS-managed services \(e.g. EKS\), reduce alerting noise and ensure certain alerts are highlighted \(RabbitMQ / failing pods\) * As an additional step to prevent this in the future, we’re planning to review all the external calls in our API handlers and move them to a transactional outbox to avoid blocking transactions if external services become unavailable.
Resolved
During the time window from 11:15 AM (IST) to 06:15 PM (IST), team members' access to team resources may have been affected.
Resolved
This incident has been resolved.
Resolved
Some EdgeCloud customers would have encountered "datastore" related error responses yesterday for API runtime related requests in the US-Central1 region. All other products and services were unaffected. We were able to identify and mitigate the impact, and have monitored service stability since the incident. If you are experiencing continued impact related to this issue, please file a case with Apigee Support for further investigation. We apologize for the impact this may have had on your business, and will take the necessary actions to prevent similar issues in the future. Impact start time: 2023-04-12 17:03 PDT Impact end time: 2023-04-12 17:48 PDT
Resolved
### **SUMMARY** On April 3, 2023, from 1:15 pm UTC to 5:20 pm UTC Atlassian customers using Opsgenie product to integrate with a separate Jira Service Management Cloud instance faced significant delays while creating and updating alerts from Jira Cloud and Jira Service Management Cloud integrations in the US region. The issue was reported by our customers and also detected via internal monitoring tools. The reason for the incident was that one of the Opsgenie integration components could not scale to the high volume of requests from Jira. This caused delays in creating alerts or Jira issues by up to 30 minutes. The incident was mitigated by scaling the integration component, which put Atlassian systems into a known good state. The total time to resolution was about four hours and 30 minutes. ### **IMPACT** The overall impact was on April 3, 2023, from 1:15 pm UTC to 5:20 pm UTC. The Incident caused degradation to customers hosted in the US region only. This caused delays of up to 30 min, in creating Opsgenie alerts from Jira issues for customers who have the Jira to Opsgenie integration enabled. ### **ROOT CAUSE** The issue was caused by the sudden spike in the volume of messages, due to bulk actions. This requires scaling up the instances manually. Our proactive monitoring prevents delays by alerting early enough to allow manual scaling. A misconfiguration in this threshold and escalation policy, in our monitoring system, prevented us from scaling up instances well in time. **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Improving auto-scaling for integration components to take care of sudden spikes in the volume of incoming messages for creating alerts via integration * Adding additional monitoring mechanisms to raise an alarm when volume thresholds are breached We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the product’s performance and availability. Thanks, Atlassian Customer Support
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved
Resolved
This incident has been resolved.
Resolved
We changed the code to send the payload exactly the same way as it's sent to us (before, we were compacting it before sending). This meant that people that relied on the payload to be compact in order to verify webhooks (i.e they were verifying incorrectly) had verification failing. We reverted this immediately once it was reported signatures were failing for customers. While not a bug in Svix, as people verifying webhooks correctly wouldn't have had issues, it still caused disruption to our customers and we've added tests to ensure that we now always compact the payloads going forward. We are also working on making it even harder to get wrong for customers, but please refer to https://docs.svix.com/receiving/verifying-payloads/how for the correct way of verifying webhooks.
Resolved
This incident is resolved.
Resolved
Our engineers have been closely monitoring the platform and are declaring this incident resolved. Thank you for your patience.
Resolved
The issue has been resolved
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We were notified early in the morning of an SSL certificate expiring for the webhooks.io domain. This certificate is utilized on both the API servers as well as some of the front-end portals that are on the webhooks.io domain. Around 7:30 AM CT, we began the renewal process of renewing and installing the updated SSL certificate on all the required infrastructure components starting with the API. At 8:50 AM CT all components were updated successfully with the active SSL certificate and verified by our engineers. In response, our engineers and team have begun a process of reviewing our after-hours notification workflow for support tickets an automated alerts that occur in critical times. This system obviously failed and we have made it our top priority to find a solution as soon as possible.
Resolved
We have seen errors subside over the last 16 hours, and uploads are succeeding as usual.
Resolved