Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

May 23, 2022

  1. MezmoAlerting, Searching, Live Tail, Graphing, and Timelines Delays

    **Dates:** Start Time: Tuesday, November 23, 2021, at 16:42 UTC End Time: Wednesday, November 24, 2021, at 17:00 UTC Duration: 24:18:00 ‌ **What happened:** Newly submitted logs were not immediately available for Alerting, Searching, Live Tail, Graphing, and Timelines.  Some accounts \(about 25%\) were affected more than others. For all accounts, the ingestion of logs was not interrupted and no data was lost. ‌ **Why it happened:** Upon investigation, we discovered that the service which parses all incoming log lines was working very slowly.  This service is upstream to all our other services, such as alerting, live tail, archiving, and searching; consequently, all those services were also delayed. We isolated the slow parsing to the specific content of certain log lines.  These log lines exposed an inefficiency in our line parsing service which resulted in exponential growth in the time needed to parse those lines; this in turn created a bottleneck that delayed the parsing of other log lines.  The inefficiency has been present for some time, but went undetected until one account started sending a large volume of these problematic lines. ‌ **How we fixed it:** The line parsing service was updated to use a new algorithm that avoids the worst-case behaviors of the original, as well as improving performance for line parsing in general. From then on, the parsing service just needed time to process the backlog of logs sent to us by customers.  Likewise, the downstream services – alerting, live tail, archiving, searching – needed time to process the logs now being sent to them by the parsing service.  The recovery was quicker for about 75% of our customers and slower for the other 25%. ‌ **What we are doing to prevent it from happening again:** The new parsing methodology has improved our overall performance significantly.  We are also actively pursuing further optimizations.

    Resolved

  2. MezmoWeb UI and Ingestion are intermittently unavailable. Alerting is halted and new Live Tail sessions can't be started.

    **Dates:** Start Time: Tuesday, January 18, 2022, at 21:00:00 UTC End Time: Wednesday, January 19, 2022, at 05:30:00 UTC Duration: 8:30:00 ‌ **What happened:** Our Web UI returned an error when customers tried to login or load pages. The errors persisted for short intervals – about 1-2 minutes each – then returned to normal usage. There were about 20 such intervals over the course of 4\+ hours. The ingestion of logs was also halted during these 1-2 minute intervals. All LogDNA agents running on customer environments quickly resent the logs. Alerting was halted for the duration of the incident and new sessions of Live Tail could not be started. ‌ **Why it happened:** We updated our parser service, which required scaling down all pods and restarting them. A new feature of the parser is to flush memory to our Redis database upon restart. The new flushing worked as intended, but also overwhelmed the database and made it unavailable to other services. This caused the pods running our Web UI and ingestion service to go into a “Not Ready” state; our API gateway then stopped sending traffic to these pods. When customers tried to load pages in the Web UI, the API gateway returned an error. When the Redis database became unresponsive, our alerting service stopped working and new sessions of Live Tail could not be started. Our monitoring of these services was inadequate and we were not alerted. ‌ **How we fixed it:** Restarting the parser server was unavoidable. We split the restart process for the parser into small segments to keep the intervals of unavailability as short as possible. In practice, there were 20 small restarts over 4\+ hours, each causing 1-2 minutes of unavailability. The WebUI and the ingestion service were fully operational by January 19, 01:21:00 UTC. On January 19, 5:30 UTC we manually restarted the Alerting and Live Tail services, which then returned to normal usage. ‌ **What we are doing to prevent it from happening again:** We’ve added code to slow down the shutdown process for the parser service to stagger the impact on our Redis database over time. Restarting the parser is uncommon; we intend to run load tests of restarts before any future updates of the parser in production are necessary, to confirm Redis is no longer affected by the new flushing behavior. We will improve our monitoring to alert us when services like Live Tail and Alerting are not functioning.

    Resolved

  3. MezmoMany services briefly halted due to Cloud Provider incident

    **Dates:**  Start Time: Thursday, January 20, 2022, at 19:13:00 UTC End Time: Thursday, January 20, 2022, at 21:24:00 UTC Duration: 02:11:00 **What happened:** Ingestion was halted and our Web UI was unresponsive for about 5-10 minutes.  Newly submitted logs were not immediately available for Alerting, Searching, Live Tail, Graphing, and Timelines. **Why it happened:** Our service hosting provider Equinix Metal had an outage that was caused by the failure of one of their main switches \(more details at [https://status.equinixmetal.com/incidents/gjmh37y6rkjp](https://status.equinixmetal.com/incidents/gjmh37y6rkjp)\).  The outage impacted traffic and global network connectivity to the LogDNA service. During the Equinix Metal incident, Ingestion, Alerting, and Live Tail were halted and our Web UI was unresponsive for a period of 5-10 minutes.  Multiple ElasticSearch \(ES\) clusters went into an unhealthy state which caused delays for about one hour in newly submitted logs being immediately available for Searching, Graphing, and Timelines. **How we fixed it:** No remedial action was possible by LogDNA.  We waited until the incident from Equinix Metal, our service hosting provider, was resolved.  The ES clusters were repaired and the backlog of newly submitted logs was processed in about one hour. **What we are doing to prevent it from happening again:** For this type of incident, LogDNA cannot take proactive preventive measures.

    Resolved

  4. MezmoIngestion, Searching, Live Tail, Alerting, Graphing, and Timelines Delays

    **Dates:** Start Time: Wednesday, January 26, 2022, at 15:45:00 UTC End Time: Wednesday, January 26, 2022, at 16:30:00 UTC Duration: 00:45:00 ‌ **What happened:** Ingestion was halted and newly submitted logs were not immediately available for Alerting, Live Tail, Searching, Graphing, and Timelines. Some alerts were never triggered. Once ingestion had resumed, LogDNA agents running on customer environments resent all locally cached logs to our service for ingestion. No data was lost. ‌ **Why it happened:** Our Redis database had a failover and the services that depend on it were unable to recover automatically. Normally, the pods running our ingestion service deliberately crash until they are able to access Redis again. However, these pods were in a bad state and unable to reconnect when Redis returned. Since ingestion was halted, newly submitted logs were not passed on to many downstream services, such as Alerting, Live Tail, Searching, Graphing, and Timelines. ‌ **How we fixed it:** We manually restarted all the pods of our ingestion service, then restarted all the sentinel pods of Redis. The ingestion service became operational again and logs were passed on to all downstream services. Over a short period of time, these services processed the backlog of logs and newly submitted logs were again available without delays. ‌ **What we are doing to prevent it from happening again:** The ingestion pods were in a bad state because they had not been restarted after a configuration change made several days earlier, for reasons unrelated to this incident. The runbook for making such configuration changes has been updated to prevent this procedural failure in the future. We’re also in the middle of a project to make all services more tolerant of Redis failovers.

    Resolved

  5. MezmoDelays in Alerting, Searching, Live Tail, Graphing, and Timelines. WebUI intermittently unavailable.

    **Dates:** Start Time: Tuesday, February 8, 2022, at 13:17 UTC End Time: Tuesday, February 8, 2022, at 14:21 UTC Duration: 1:04:00 ‌ **What happened:** Our Web UI was unresponsive for about 10 minutes. Newly submitted logs were not immediately available for Alerting, Searching, Live Tail, Graphing, and Timelines. No data was lost and ingestion was not halted. ‌ **Why it happened:** Our Redis database had a failover and the services that depend on it were unable to reconnect after it recovered, including the Parser. This service is upstream of many other services. Consequently, newly submitted logs were not passed on to many downstream services, such as Alerting, Live Tail, Searching, Graphing, and Timelines. The WebUI was also intermittently unavailable because it requires a connection to Redis. ‌ **How we fixed it:** We manually restarted the Redis service which allowed a new master to be elected. After Redis recovered, the Parser, Web UI and other services were restarted which were then able to reestablish a connection to Redis. This restored the Web UI and allowed newly submitted logs to pass from our Parser service to all downstream services. Over a short period of time, these services processed the backlog of logs and newly submitted logs were again available without delays. ‌ **What we are doing to prevent it from happening again:** We recently added functionality to track the flow rate of newly submitted logs. This new feature requires more memory than expected in the event of a Redis failover, which is why services could not reconnect to Redis. We’ve increased the limits of the memory buffer for the relevant portions of our service. We will also add additional Redis monitoring to more quickly detect unhealthy sentinels and continue to work on an ongoing project to make all services more tolerant of Redis failovers.

    Resolved

  6. MezmoSome logs submitted over syslog using custom ports are not being correctly formatted and are not available for Alerting, Searching, Timelines, Graphs, and Live Tail.

    **Dates:** Start Time: Wednesday, February 16, 2022 at 19:58 UTC End Time: Wednesday, February 16, 2022 at 21:10 UTC Duration: 1:12:00 **What happened:** Some logs being sent to our service over syslog using custom ports were not being correctly parsed and were not available for Alerting, Searching, Timelines, Graphs, and Live Tail. Unparsable log lines showed the error “Unidentifiable Syslog Source” and “Unsupported syslog format.” Logs being sent over syslog that do not use custom ports were working normally. ‌ **Why it happened:** We introduced a bug into our production environment, specifically in a new service called Syslog Forwarder. The bug prevented Syslog lines from being parsed. As a result, any newly submitted Syslog lines sent using custom ports were not parsed. The lines displayed an error “Unidentifiable Syslog Source” and “Unsupported syslog format.” ‌ **How we fixed it:** We created a hot fix that corrected the bug. **What we are doing to prevent it from happening again:** We added to our test suite to guard against regressions in the Syslog Forwarder.

    Resolved

  7. MezmoIngestion of new logs — for Syslog only - is intermittently failing

    **Dates:** Start Time: Wednesday, February 17, 2022, at 20:56 UTC End Time: Thursday, February 18, 2022, at 02:15 UTC Duration: 5:19:00 ‌ **What happened:** The ingestion of new logs to our Syslog endpoint was intermittently failing. ‌ **Why it happened:** We made a code change to the area of our service \(Syslog Forwarder\) that handles the ingestion of logs sent by Syslog and inadvertently changed how memory is managed. Routine memory garbage collection stopped and memory usage increased on the pods that accept newly submitted log lines over Syslog. Eventually, the increase in memory caused the pods to crash. Any log lines held on those pods were lost and never ingested. ‌ **How we fixed it:** We reverted to the previous version of the Syslog Forwarder service. This stopped the pods from crashing. We then resolved the memory management issue in our code. The new, fixed version was released to production shortly thereafter and performed as expected. ‌ **What we are doing to prevent it from happening again:** We have added regression tests to the Syslog Forwarder service to prevent a similar mistake in the future.

    Resolved

  8. MezmoIngestion of new logs — for Syslog only - Partial Outage

    **Dates:** Start Time: Thursday, February 18, 2022, at 00:10 UTC End Time: Thursday, February 24, 2022, at 23:43 UTC Duration: 167:33:00 ‌ **What happened:** The ingestion of new logs to our Syslog endpoint was intermittently failing. ‌ **Why it happened:** We recently introduced a new service \(Syslog Forwarder\) to handle the ingestion of logs sent over Syslog. As the name implies, it forwards logs to downstream services. It was designed to send all logs submitted for each account to a single port opened on the downstream services. No load balancing was implemented in our original design, which performed well in our advance testing. Once put into production, however, it became apparent that some customer accounts submit logs at a volume higher than the downstream services could process. When this happened, logs lines were buffered in memory by the Syslog Forwarder. Memory increased until the pods crashed. Any log lines held on those pods were lost and never ingested. ‌ **How we fixed it:** We improved the design of the Syslog Forwarder by adding a pool of connections to the downstream services. In effect, we added traffic shaping to the Syslog Forwarder. ‌ **What we are doing to prevent it from happening again:** The new architecture has been incorporated and proven resilient in production. No further work is needed to prevent this kind of incident from happening again.

    Resolved

  9. MezmoIngestion of new logs to our Syslog endpoint – for logs sent using a custom port, only – is intermittently delayed

    **Dates:** Start Time: Saturday, February 26, 2022, at 19:51 UTC End Time: Sunday, February 27, 2022, at 22:13 UTC Duration: 26:22:00 ‌ **What happened:** Ingestion of new logs to our Syslog endpoint – for logs sent using a custom port, only – was intermittently delayed. ‌ **Why it happened:** We recently introduced a new service \(Syslog Forwarder\) to handle the ingestion of logs sent over Syslog. As the name implies, it forwards logs to downstream services. Logs are sent from a range of ports on Syslog Forwarder to a range of ports used by clients running on downstream services. This design worked well in our advance testing, using a limited number of custom ports. Once running in production, however, the Syslog Forwarder needed to connect to a much larger number of custom ports. We then saw that the ephemeral port ranges of the clients running on downstream services overlapped with the port ranges used by the Syslog Forwarder. This led to occasional port conflicts when services and/or clients tried to start. The services and/or clients would attempt to start again until they found an open port without conflicts. This created delays in ingestion. ‌ **How we fixed it:** We changed the ephemeral port ranges of the clients running on downstream services so they no longer overlapped with the port ranges used by the Syslog Forwarder. ‌ **What we are doing to prevent it from happening again:** The new ephemeral port range has been incorporated and proven resilient in production. No further work is needed to prevent this kind of incident from happening again.

    Resolved

  10. MezmoAlerts not triggering

    **Dates:** Start Time: Tuesday, April 5, 2022, at 13:20:00 UTC End Time: Tuesday, April 5, 2022, at 18:20:00 UTC Duration: 5:00:00 ‌ **What happened:** Alerting was halted for all accounts for the entire duration of the incident. Most alerts – any whose trigger date was more than 15 minutes in the past – were discarded. ‌ **Why it happened:** We restarted our parser service, for reasons unrelated to this incident. Any restart of the parser service should be followed by a restart of the alerting service. This second step was overlooked and didn’t happen. Subsequently, all alerts stopped triggering. The need to restart alerting after a restart of the parser is already documented and well-known to our infrastructure team. However, the restart of the parser was performed by a team less familiar with the correct procedure. ‌ **How we fixed it:** We manually restarted the alerting service, which then returned to normal operation. ‌ **What we are doing to prevent it from happening again:** The proper documented restart procedure has been discussed with all teams allowed to restart services. We will add monitoring of our alerting service and automated notifications so we learn more quickly of any similar incidents in the future.

    Resolved

May 20, 2022

May 17, 2022

  1. AppsFlyertest2

    This incident has been resolved.

    Resolved

May 16, 2022

May 14, 2022

May 11, 2022

May 6, 2022

Apr 29, 2022

  1. Atlassian StatuspageMultiple sites showing down/under maintenance

    Earlier this month, several hundred Atlassian customers were impacted by a site outage. We have published a Post-Incident Review which includes a technical deep dive on what happened, details on how we restored customers sites, and the immediate actions we’ve taken to improve our operations and approach to incident management. [https://www.atlassian.com/engineering/post-incident-review-april-2022-outage](https://www.atlassian.com/engineering/post-incident-review-april-2022-outage)

    Resolved

Apr 27, 2022

  1. HoneybadgerStatus pages down

    We had a faulty deploy at 11 am yesterday that caused us to stop serving customer status pages. We weren’t immediately aware of the problem due to insufficient monitoring of this recently-launched feature. We have restored the status pages, identified and resolved the problem that caused the faulty deploy, and added more monitoring.

    Resolved

Apr 19, 2022

Apr 5, 2022

  1. SvixElevated 5xx errors affecting a small fraction of the requests for ~10 seconds

    Due to a race condition in a complex deployment, newly deployed API endpoints were trying to access resources using a stale configuration. Only a few API calls were affected as it only affected a few API endpoints before the deployment completed and updated the configuration for all of them. We have taken precautions for preventing this from ever happening again.

    Resolved

Apr 1, 2022

  1. HoneybadgerCheck-in failures

    We had a period of about 30 minutes where our check-ins and sourcemaps API endpoints were not accepting requests, which resulted in some check-in missing alerts that should not have been sent and in failures to upload sourcemaps. This issue has been resolved.

    Resolved

Mar 30, 2022

Mar 29, 2022

Mar 28, 2022

  1. GradleIssue resolving artifacts

    For 2.5 hours, the Gradle plugin portal was having issue serving artifacts from JCenter. This also caused maven-metadata.xml, used for dynamic version ranges, to fail to resolve.

    Resolved

Mar 22, 2022

  1. Atlassian StatuspageIssues With Login

    ### **SUMMARY** On March 14, 2022, between 01:05pm and 01:47pm UTC, some Atlassian customers were unable to login to our products including Trello and Statuspage, and could not access some services including the ability to create support tickets. The underlying cause was a newly introduced configuration data store that did not scale up properly due to a misconfiguration of autoscaling. The incident was detected by Atlassian's automated monitoring system and mitigated by disabling the use of the new configuration datastore which put our systems into a known good state. The total time to resolution was approximately 42 minutes. ### **IMPACT** The overall impact was between March 14 2022, 01:05 PM UTC and March 14, 2022, 01:47 PM UTC across seven products and services. The bug impacted several of the key dependent services which resulted in an outage for end users, leading to failed logins across the following products and services: * [**getsupport.atlassian.com**](http://getsupport.atlassian.com) * [**confluence.atlassian.com**](http://confluence.atlassian.com) * [**jira.atlassian.com**](http://jira.atlassian.com) * [**partners-jira.atlassian.com**](http://partners-jira.atlassian.com) * [**community.atlassian.com**](http://community.atlassian.com) * [**manage.statuspage.io**](http://manage.statuspage.io) * [**trello.com**](http://trello.com) * [**university.atlassian.com**](http://university.atlassian.com) ### **ROOT CAUSE** The issue was caused by an underlying configuration data store based on AWS DynamoDB failing to scale up. During post-setup fine-tuning it was identified that initial values for the read capacity units \(RCUs\) and write capacity units \(RCUs\) were over-provisioned. As a result a decision was made to decrease them however the resulting values proved to be insufficient to handle the increased traffic in our system. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We're prioritizing the following improvement actions to avoid repeating this type of incident: * Fix the configuration so that the new configuration data store dynamically scales-up regardless of the size of the incoming traffic. * Conduct more thorough capacity planning and load testing. * Improve the resilience of the system by adding fallbacks to our secondary data store. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support

    Resolved

Older incidents