Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Jun 23, 2022

Jun 21, 2022

  1. RadarGeocoding APIs degraded

    Approximately 50% of geocode autocomplete queries incorrectly returned empty results. This issue has been resolved.

    Resolved

  2. GradlePlugin portal issues

    The root cause was the Cloudflare outage - see https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj - which is now resolved.

    Resolved

Jun 19, 2022

Jun 16, 2022

Jun 10, 2022

  1. PyPIRedirect Loops on JSON API endpoints.

    ## Summary PyPI's JSON API experienced an incident that caused redirect loops to occur for all clients when requesting many of the endpoints available on the service. These endpoints provide JSON documents describing projects and specific releases of projects hosted on [pypi.org](http://pypi.org). This incident was initiated by changes intended to make the JSON API more performant and less of a burden on the PyPI backends. Specifically, the combination of [this change to the warehouse codebase](https://github.com/pypa/warehouse/pull/11546) and [this change to the CDN configuration for PyPI](https://github.com/python/pypi-infra/pull/87) update the canonical locaiton for accessing the JSON documents deterministic. This allows for deeper cache efficiency and the ability to move redirects to the canonical URL out to the CDN edge. The result is faster response times for clients and reduced load on the PyPI backends. The redirect updates specifically had an unintended consequence that led to this outage. Namely the existing elements in cache for the "normalized" project names redirected to the verbatim names of projects. Example: * Before * `/pypi/pyOpenSSL/json` "verbatim" project name url returns a `200` with the document and is canonical. * `/pypi/pyopenssl/json` "normalized" project name url `301` redirects to canonical URL. * After * `/pypi/pyOpenSSL/json` "verbatim" project name url returns a `301` redirect to canonical URL. * `/pypi/pyopenssl/json` "normalized" project name url returns a `200` with the document and is canonical. The issue at hand is that what is intended to the new canonical URL using the "normalized" project name was cached for _many_ projects, and that cache included a redirect to the "verbatim" project name, which... redirected back. ## Impact This impacted any client of the PyPI JSON API. These include the `poetry` tool for installing and managing Python dependencies, mirroring tools such as `bandersnatch`, and even our own internal service that redirects legacy file URLs on [files.pythonhosted.org](http://files.pythonhosted.org) to their new locations. The outage began with the deployment of [this change to the warehouse codebase](https://github.com/pypa/warehouse/pull/11546) and [this change to the CDN configuration for PyPI](https://github.com/python/pypi-infra/pull/87) at approximately 2022-06-10T10:50 UTC and continued to impact some URLs in a "long-tail" fashion through to 2022-06-10T16:00 UTC. Most notably project names beginning with `p` were the last to be affected as it is the most common first letter for projects uploaded to PyPI. ## Mitigation The changes described that led to this outage have been actively being attempted for over 6 weeks after more than a year of being aware of the problem. When faced with a system that was functioning correctly, but required purging of cache to be fully established the PyPI administrator managing the deployment chose to roll forward rather than roll back, knowingly extending the impact of this incident in favor of getting PyPI to a more maintainable state for serving the JSON API into the future. Because of the massive scale of PyPI's caches it was untenable to specifically purge bad URLs, leading to a need to clear the entire PyPI cache. This was undertaken by kicking off pools of processes that would iterate over individual first letters of project names to purge the entire project cache in parallel over a 1-2 hour duration. This duration was required as purging the entire cache would have overloaded the backends for PyPI in such a way that even with massive temporary scale up/out the load would have been too much for our backend and led to _many more hours_ of outage across the entire service. While most of the purges completed within the two hour estimate, the letter `p` was the final sgement, taking nearly 4 hours due to the popularity of the `py` prefix on the index. While the purges were ongoing PyPI's backends even in a scaled out state provided lackluster response times and performance as the caches were slowly refilled. ## Future work Issues have been filed to create better tooling to safely and expediently purge the caches and to limit the blast radius of purges in the future. We will also begin to research ways to build more confidence and expose these kinds of errors in our review process for changes to the configuration of our CDN.

    Resolved

Jun 9, 2022

Jun 7, 2022

Jun 6, 2022

Jun 3, 2022

  1. MezmoLog lines not ingested

    **Dates:** Start Time: Thursday, June 2, 2022, at 20:25 UTC End Time: Thursday, June 2, 2022, at 20:50 UTC Duration: 00:25 **What happened:** The ingestion of logs was halted for about 25 minutes. During that time, newly submitted logs were never ingested and therefore not available for Alerting, Searching, Live Tail, Graphing, Timelines, and Archiving. **Why it happened:** We manually reverted our ingester service to an older version \(to solve a minor problem unrelated to this incident\). During the procedure, the version of the container was reverted, but not the container’s configuration. Because of this versioning mismatch, logs from the ingester stopped being accepted by a downstream service \(the “buzzsaw broker”\). The ingester is currently not designed to confirm logs are accepted by downstream services; therefore it returned http 200 messages to our customer’s agents, indicating logs had been successfully received. At this point the agent discarded any locally cached log files. Consequently, all log lines sent during the incident \(25 minutes\) were never ingested. **How we fixed it:** We reverted the container’s configuration correctly, so it matched the version of the container itself. Ingestion began working normally again. **What we are doing to prevent it from happening again:** We will review and update our runbooks for reverting services to earlier versions to prevent similar mistakes. We also plan to automate the reversion process. We will add internal confirmations to the ingester so it is always certain log lines were received by downstream services. This will prevent the ingester from sending erroneous 200 messages back to the agent, should the ingester be unable to pass log lines downstream.

    Resolved

  2. PyPINewly uploaded files are returning 403 from files.pythonhosted.org

    ## Summary The cloud infrastructure that runs PyPI is managed via an automation tool which allows for configuration changes to be version controlled, reviewed, and deployed automatically. This tool includes multiple backends for state sharing and locking to facilitate collaboration between multiple contributors. While migrating the remote state backend and locking features to a more robust hosted platform an error was made when setting the credentials that grant our CDN access to the object storage where uploaded files to PyPI are hosted. The result was that newly uploaded files as well as files that had expired from our CDNs cache were unavailable. The outage began 2022-06-022 20:28 UTC and was initially mitigated 2022-06-02 22:38 UTC by rolling back the configuration to a previous version. The configuration issue in the hosted platform was confirmed and corrected 2022-06-02 12:00 UTC and a corrected new version of the CDN configuration was deployed. Future deploys will include this value and will not regress in the same manner. ## Details As the Infrastructure team at the PSF onboards a new hire and the clunkiness of collaborating on our terraform configuration led to only two people having the ability to deploy changes, a decision was made to migrate all state, sensitive variables, and locking to a hosted backend for terraform. This will allow all those with appropriate access \(PyPI Admins and PSF Infrastructure Staff\) access without having to pass around plaintext secrets necessary to plan and apply changes to the infrastructure. During the migration from S3 state storage and DynamoDB locking to Hashicorp Terraform Cloud, the Infrastructure staff lead made an error in copying the value for the GCS secret access key that grants Fastly, our CDN, access to read the Cloud Storage bucket where PyPI uploads are stored. Because the value was marked as sensitive in Terraform Cloud via a setting, it wasn’t obvious to the staff member when planning the changes what would change, and it was assumed to be an artifact of that setting. As a confirmation and for assurances, the staff member configured the changes to Fastly configuration to _not_ auto activate the new configuration. After applying the changes which would not be automatically activated in Fastly it is unclear why the incorrect change was not identified when diffing service configurations, but ultimately it was missed and the services were activated. Impact lasted 130 minutes until enough signal from users was raised to alert PyPI admins and affected all new uploaded files as well as any files that had expired from our CDN caches. ## Future Mitigation This outage highlighted two key areas where improvements can be made to ensure timely response 1. **More thorough automated monitoring of file access via our CDN.** Due to the caching nature of our CDN, it was not detected that this error was occurring as the file checked never expired from cache. We will implement a monitor that checks for a file that is explicitly not cached to ensure that the entire system from client to object-store is working appropriately 2. **Improvements to secrets handling in our infrastructure automation.** We carried some legacy practices for storing secrets/configuration in our systems that led to pertinent configuration value being obscured in a complex object rather than an individual key. We’ll refactor this to make detecting changes to values more obvious to contributors to the project.

    Resolved

May 30, 2022

  1. AppsFlyerLTV data availability

    The issue is resolved. Data is available as usual on the dashboards and reports. Thank you for your patience. If you're still experiencing any related issues, please reach out to customer support.

    Resolved

May 27, 2022

  1. Engine YardFailure at running Chef

    The deployed fix addressed the issue, allowing Chef runs to happen smoothly once again. As such we're marking the incident as Resolved. Feel free to open a ticket with Engine Yard Support for any questions, comments, or concerns that may arise.

    Resolved

May 24, 2022

  1. PyPIMalicious takeover of ctx project on PyPI.

    Takeover of the ctx project was reported on multiple channels overnight and was mitigated as of 6:07 AM Eastern. We confirmed via investigation that this compromise was of a single user account due to re-registration over an expired domain. The domain that hosted the users email address was re-registered 2022-05-14T18:40:05Z and a password reset completed successfully for the user at 2022-05-14T18:52:40Z. Original releases were then deleted and malicious copies uploaded. PyPI itself was not directly compromised. Read the full incident report at [https://python-security.readthedocs.io/pypi-vuln/index-2022-05-24-ctx-domain-takeover.html](https://python-security.readthedocs.io/pypi-vuln/index-2022-05-24-ctx-domain-takeover.html).

    Resolved

May 23, 2022

  1. MezmoWebapp Unavailable

    We have identified and resolved the issue that was making https://app.logdna.com unavailable.

    Resolved

  2. MezmoService Unavailable

    **Dates:** The incident was opened on December 17, 2020 - 23:29 UTC. Our service was fully operational by December 18, 2020 - 12:30 UTC. The incident was officially closed on December 20, 2020 - 03:49 UTC. ‌ **What happened:** All services were unavailable for about eight hours. For an additional four hours, services were available but there were significant delays in searching, graphing, and timelines for newly submitted logs. Additionally, all logs submitted during the first six hours of the incident were never processed by our service and were unavailable in the UI, even after our service was fully operational. ‌ **Why it happened:** Our hosting provider had a major power failure that lasted almost five hours. The hardware that our service runs on was unavailable and none of our services could operate. More details: [https://status.equinixmetal.com/incidents/pfgmgy1fnjcp](https://status.equinixmetal.com/incidents/pfgmgy1fnjcp) ‌ **How we fixed it:** Once our provider was back online, we gradually restarted all our services. This took time and manual intervention because all our services had been taken down ungracefully by the outage. Around December 18, 2020 - 07:54 UTC, services became operational and logs began to be ingested again. Since no logs had been ingested for about eight hours, our service had a large backlog to process. As it caught up, users experienced delays in searching, graphing, and timelines for newly submitted logs. The backlog was fully processed around December 18, 2020 - 12:30 UTC and services were once again fully operational. Logs submitted during the first six hours of the incident \(around December 17, 2020, 23:00 UTC to December 18, 2020, 5:00 UTC\) remained unavailable in the UI. Normally, if our service is temporarily unavailable, logs can be resubmitted and successfully processed. In this case, the sudden loss of power brought down our services ungracefully, abruptly interrupting write operations as we processed logs. This resulted in partial writes and bad writes, which made our service unable to determine, for the resubmitted logs, where log lines began. In effect, this made logs resubmitted from that six hour period of time unreadable and unable to be processed. The incident was kept open as we made attempts to read and process these logs, but these efforts were ultimately unsuccessful. After the incident was closed, we developed the means to restore archives of these logs to all customers with version 3 of archiving enabled. The restoration of archives is expected to begin on the week of January 18th. ‌ **What we are doing to prevent it from happening again:** We are developing changes to how we write logs so that in a similar event our service will not lose track of the start of log lines and be able to read and process them.

    Resolved

  3. MezmoWeb UI unavailable and ingestion has stopped

    **Dates:** Start Time: Thursday, January 14, 2021, at 19:42 UTC End Time: Thursday, January 14, 2021, at 20:27 UTC Duration: 0:45:00 **What happened:** Our WebUI became unavailable and ingestion of new logs stopped for 45 minutes. Logs were automatically resent later and ingested successfully for customers using our ingestion client agent. **Why it happened:** The certificate used by all our services expired. Consequently, all API calls to our service failed, which caused our WebUI to fail and ingestion of new logs to stop. **How we fixed it:** We renewed the certificate and applied it to all affected services. Our WebUI became responsive again and ingestion resumed. Since no logs had been ingested for about 45 minutes, our service had a moderately large backlog to process. As it caught up, users experienced delays in searching, graphing, and timelines for newly submitted logs. **What we are doing to prevent it from happening again:** We’re tightening our internal notifications of upcoming expiration dates for all certificates our service relies upon.

    Resolved

  4. MezmoIndexing Delay

    ## Dates: Start Time: Friday, February 26, 2021, at 06:43 UTC End Time: Friday, February 26, 2021, at 20:42 UTC ## What happened: The insertion of newly submitted logs stopped entirely for all accounts for about 3 hours. Logs were still available in Live Tail but not for searching, graphing, and timelines. The ingestion of logs from clients was not interrupted and no data was lost. For more than 95% of newly submitted logs, log processing returned to normal speeds within 3 hours. All logs submitted during the 3 hour pause were available again about 30 minutes later. For less than 5% of newly submitted logs, log processing returned to normal speeds gradually. Logs submitted during the 3 hour pause also gradually became available. This impact was limited to about 12% of accounts. The incident was closed when logs from all time periods for all accounts were entirely available. ## Why it happened: Our service ran out of a set of resources that manage pre-sharding on the clusters that store logs, an operation that ensures new logs are promptly inserted into the clusters. This happened because of several simultaneous changes to our infrastructure that didn’t account for the need for more resources, particularly on clusters with a relatively large number of shards relative to their overall storage capacity. The insertion of new logs slowed down and the backlog of unprocessed logs grew. Eventually, the portion of our service that processes new logs was unable to keep up with demand. ## How we fixed it: We restarted the portion of our service that processes newly submitted logs. During the recovery, we prioritized restoring logs submitted in the last day. 95% of accounts were fully recovered after 3.5 hours. ## What we are doing to prevent it from happening again: We’ve increased the scale of the set of resources that ensure logs are processed promptly by adding more servers for these resources to run upon. We’ve also added alerting for when these resources are reaching their limit.

    Resolved

  5. MezmoLogs are not searchable in web app

    **Dates:** Start Time: Thursday, March 4, 2021, at ~03:45 UTC End Time: Thursday, March 4, 2021, at ~08:20 UTC Duration: ~4:36:00 ‌ **What happened:** Our Web UI returned an error message "Request returned an error. Try again?" when users tried to perform a search query or use Live Tail in the Web UI. ‌ **Why it happened:** The pods that run our searching and Live Tail services were automatically terminated by our Kubernetes orchestration system. Upon investigation, we discovered we had inadvertently classed these services as low priority. The incident occurred when a large number of other services that were classed as higher priority needed to run to meet usage demands. The orchestration system automatically terminated the lower priority services to make resources available for the higher priority services. More specifically, these pods were put into a “terminating” state. Normally this state is temporary -- a transition between “running” and “terminated”. During this incident, the pods remained in the “terminating” state permanently. Our monitoring detects services that have been “terminated”, but not ones that are in the temporary “terminating” state. Consequently, our infrastructure team was not notified. ‌ **How we fixed it:** We increased the priority of the pods that run our searching and Live Tail services to match the priority of other services. We updated the configuration of our orchestration system to make the change permanent. ‌ **What we are doing to prevent it from happening again:** We’ve already updated the configuration of our orchestration system to give services the correct priority. These changes are permanent and should prevent similar problems in the future.

    Resolved

  6. MezmoSearch performance degraded.

    **Start Time:** April 28, 2021 at 12:37 UTC **End Time:** May 3, 2021 at 00:54 UTC **What happened:** Newly submitted log lines from all customers were significantly delayed before being available in our WebUI for searching, graphing, and timelines. Alerting, Live Tail, and the uploading of archives to their destinations were significantly delayed as well. The incident was opened on April 28, 12:37 UTC. Typical mitigation steps were taken, but unsuccessful. Live Tail and alerting -- which were also significantly degraded -- were halted, about 14 hours after the start of the incident. This step was taken to keep other services, such as Search, functioning and give more resources to processing log lines. Logs submitted before the incident continued to be searchable. By May 1, 19:17 UTC, about 99% of newly submitted logs were again available in our WebUI at normal rates. Other essential services needed more time and manual intervention to recover. The incident was closed on May 3, 00:54 UTC. Ingestion of new logs lines from clients continued normally throughout the incident. ‌ **Why it happened:** We deployed an updated version of our proprietary messaging bus / parsing pipeline. This version had been tested in staging and multiple production regions beforehand and worked as expected. It was deployed and worked normally in production for four days. The cumulative traffic to our service over those four days revealed a performance issue that affected the processing of new log lines: logs were processed, but at a very slow rate. We’ve identified the cause of the slow performance as an update to node.js \(version 14\) that was part of the new version of our messaging bus. ‌ **How we fixed it:** Once the source of the failure had been identified, we reverted our messaging bus to its last stable version, which kept the delays in processing from degrading further. Our services still needed to process logs ingested up to that point, which required time, manual intervention, and more resources. We temporarily increased the number of servers dedicated to processing logs by about 60%. We also halted Live Tail and alerting, which were degraded almost to the point of being non-functional. Through the combination of these efforts, all logs were eventually processed and our service was again entirely operational. ‌ **What we are doing to prevent it from happening again:** During the incident, the new version of our messaging bus was reverted to its previous version. The version in production today does not contain the upgrade to node.js 14, which caused the performance degradation. We’ve removed node.js 14 from any future upgrades until we’ve had time to carefully examine its performance issues.

    Resolved

  7. MezmoSearch and Live Tail Intermittent Failures

    **Start Time:** Thursday, August 19, 2021, at, 13:56 UTC **End Time:** Thursday, August 19, 2021, at, 20:48 UTC **Duration:** 6:52:00 **What happened:** Searches in our Web UI and Live Tail were intermittently failing.  Additionally, for a small set of customers \(about 12%\), there were delays in newly submitted logs being available for searching, graphing, and timelines. **Why it happened:** Our service uses the Calico networking solution to ensure network level connectivity between all nodes and the pods running on them.  On several nodes, Calico stopped running.  This put one of our ElasticSearch clusters into an unhealthy red state.  For customers using this cluster \(about 12% of all our customers\), there were delays in newly submitted logs being available for searching, graphing, and timelines.  When Calico stopped running on some nodes, it also led to failures with tribe nodes, which make searching across multiple clusters possible.  This caused intermittent failures in searching and live tail, for all customers. ‌ **How we fixed it:** We took remedial action by restarting Calico and restoring networking connections between all nodes.  We also restarted our tribe nodes, repaired the ElasticSearch cluster that was in a red state, and then provided temporary resources so our service could more quickly process the backlog of logs sent by customers since the beginning of the incident. ‌ **What we are doing to prevent it from happening again:** We are investigating why Calico stopped working on several nodes.  We’re also updating our runbooks to recover more quickly in similar situations and limit any customer impact.

    Resolved

  8. MezmoEmail Alerts Are Not Being Sent

    Start Time: Wednesday, October 6, 2021, at 17:30:06 UTC End Time: Wednesday, October 6, 2021, at 21:30:27 UTC Duration: 4:00:21 ## What happened: Email notifications from alerts were partially halted for about 4 hours. Notifications sent by Slack and Webhooks were not affected. ## Why it happened: Our email service provider’s daily limit of 250,000 email messages was exceeded. All email message notifications from triggered alerts bounced and could not be resent. Further investigation revealed that a HackerOne Security Analyst looking for flaws in our service to report to us had made our system send 450,000 emails. This was accomplished by manually adding an array of many email addresses into the “Change Owner” request field within our Web UI. Adding multiple email addresses in that field is not permitted through normal usage of the Web UI. The Security Analyst intercepted the http request sent when the form was submitted and manually inserted a json list in the field, effectively sending an array of email addresses rather than a string. LogDNA had no server-side \(e.g. backend\) validation to ensure only a string could be accepted. ## How we fixed it: We took remedial action by contacting our email provider, who temporarily increased our daily email sending limit to 625,000 messages. This allowed email notifications from alerts to resume. We then added server-side validation for the “Change Owner” field in our Web UI so that only strings are accepted, even if the request is manually intercepted and an array of email addresses is added. ## What we are doing to prevent it from happening again: We will audit our Web UI to find all places where multiple email inputs can be added. We’ll then add server-side validation, so only strings are accepted. We’ll emphasize to our HackerOne Security Analysts that they should not take potentially damaging actions as they proactively search for vulnerabilities in our service.

    Resolved

  9. MezmoSome customers are not able to login to our UI

    Start Time: Thursday, October 7, 2021, at 17:52 UTC End Time: Thursday, October 7, 2021, at 18:46 UTC Duration: 0:54:00 ## What happened: Our Web UI returned the error message “This site can’t be reached” when some users tried to login or load pages. The ingestion of logs was unaffected. ## Why it happened: The Telia carrier service in Europe experienced a major network routing outage caused by a faulty configuration update. The routing policy contained an error that impacted traffic to our service hosting provider, Equinix Metal. The Washington DC data center that houses our services was impacted. During this incident the [app.logdna.com](http://app.logdna.com) site was unreachable for some customers, depending on their location. ## How we fixed it: No remedial action was possible by LogDNA. We waited until the incident from Equinix Metal, our service hosting provider, was resolved. ## What we are doing to prevent it from happening again: For this type of incident, LogDNA cannot take proactive preventive measures.

    Resolved

  10. MezmoEmails Are Not Being Sent

    **Start Time:** Thursday, October 28, 2021, at 16:56:52 UTC **End Time:** Thursday, October 28, 2021, at 22:17:24 UTC **Duration:** 5:20:32 ‌ **What happened:** Email notifications of all kinds, including from alerts, were delayed for about 5 hours.  Notifications sent by Slack and Webhooks were not affected.  **Why it happened:** Our email service provider \(Sparkpost\) experienced an incident that caused delays for all emails from the LogDNA service.  We rely on this service to deliver email of all kinds, including notifications for alerts.  Email messages were delayed and queued until our email service provider was able to recover.   More information on the incident can be found at Sparkpost’s Status Page: [https://status.sparkpost.com/incidents/bwl8dr6gwmts?u=ydzrh5x205pf](https://status.sparkpost.com/incidents/bwl8dr6gwmts?u=ydzrh5x205pf) **How we fixed it:** No remedial action was possible by LogDNA. We waited until the incident from Sparkpost, our email hosting provider, was resolved. **What we are doing to prevent it from happening again:** For this type of incident, LogDNA cannot take proactive preventive measures.

    Resolved

  11. MezmoWeb App is Unreachable

    **Start Time:** Monday, November 8, 2021, at 23:28 UTC **End Time:** Tuesday, November 9, 2021, at 00:16 UTC **Duration:** 0:48:00 ‌ **What happened:** Our Web UI returned the error message “This site can’t be reached” when users tried to login or load pages. The ingestion of logs was unaffected. ‌ **Why it happened:** The node our web service was running on had a failure with its network management software and became unreachable. Furthermore, the web service was only running on a single node, which is atypical – usually it runs on multiple nodes at once to improve performance and allow for redundancy. Both conditions were necessary for the Web UI to become unavailable. ‌ **How we fixed it:** We moved the web service to another node with functioning network management software, which made the Web UI available again. Later, we restarted the unreachable node, which restored it to normal usage. ‌ **What we are doing to prevent it from happening again:** We expect both necessary conditions – the failure of the network management software and that the web service was running on a single node – to be resolved by an already planned migration of our entire service to a new cloud-based environment. We are currently building monitoring of the availability of our Web UI so we can learn of any future failures as soon as possible.

    Resolved

  12. MezmoIndexing, Livetail Performance, Search, and Alerting Delays

    **Dates:** Start Time: Monday, November 22, 2021, at 19:01 UTC End Time: Tuesday, November 23, 2021, at 02:04 UTC Duration: 7:03:00 ‌ **What happened:** Newly submitted logs were not immediately available for Alerting, Searching, Live Tail, Graphing, and Timelines.  Some accounts \(about 25%\) were affected more than others. For all accounts, the ingestion of logs was not interrupted and no data was lost. ‌ **Why it happened:** Upon investigation, we discovered that the service which parses all incoming log lines was working very slowly.  This service is upstream to all our other services, such as alerting, live tail, archiving, and searching; consequently, all those services were also delayed. We isolated the slow parsing to the specific content of certain log lines.  These log lines exposed an inefficiency in our line parsing service which resulted in exponential growth in the time needed to parse those lines; this in turn created a bottleneck that delayed the parsing of other log lines.  The inefficiency has been present for some time, but went undetected until one account started sending a large volume of these problematic lines. ‌ **How we fixed it:** The line parsing service was updated to use a new algorithm that avoids the worst-case behaviors of the original, as well as improving performance for line parsing in general. From then on, the parsing service just needed time to process the backlog of logs sent to us by customers.  Likewise, the downstream services – alerting, live tail, archiving, searching – needed time to process the logs now being sent to them by the parsing service.  The recovery was quicker for about 75% of our customers and slower for the other 25%. ‌ **What we are doing to prevent it from happening again:** The new parsing methodology has improved our overall performance significantly.  We are also actively pursuing further optimizations.

    Resolved

Older incidents