Resolved incident

Replication bug causing slow recoveries

Elastic · Major · Started

Duration
708h 28m
Severity
Major
Status
Resolved

Summary

Major · Resolved

This issue was resolved in the 9.5.3 release of Elasticsearch.  Customers running versions 9.5.0, 9.5.1, or 9.5.2 of Elasticsearch are advised to update to a newer version as soon as possible.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    Elastic

    We are aware of a bug affecting customers on Elasticsearch versions 9.5.0, 9.5.1 and 9.5.2 where bulk index operations processed on the primary shard can result in divergence on the replicas resulting in high disk usage and long-running recoveries. The team is working on a fix. More information can be found on https://www.elastic.co/docs/release-notes/elasticsearch/known-issues#elasticsearch-9.5.2-known-issues The next update will be in the next 2 hours or sooner if needed.

  2. Identified

    Elastic

    We have identified the root cause and the team is working on a fix scheduled to be released in Elasticsearch version 9.5.3 What you can do: - If you have not yet upgraded to 9.5.*, you are not affected as upgrades to 9.5 have been disabled until 9.5.3 is released - If you are already running 9.5 and experience the symptoms of this issue, please contact Elastic Support. More information can be found on https://www.elastic.co/docs/release-notes/elasticsearch/known-issues#elasticsearch-9.5.2-known-issues

  3. Identified

    Elastic

    The fix has been identified and backported to the Elasticsearch 9.5 patch branch. The teams are performing the needed validations before scheduling and rolling out the 9.5.3 release.

  4. Monitoring

    Elastic

    Elasticsearch 9.5.3 has been released and all customers currently running 9.5.0, 9.5.1 or 9.5.2 are advised to update to this new version as soon as possible.

  5. Monitoring

    Elastic

    Elasticsearch 9.5.3 has been released and all customers currently running 9.5.0, 9.5.1 or 9.5.2 are advised to update to this new version as soon as possible. Impacted customers are advised to follow the workaround steps from https://github.com/elastic/elasticsearch/issues/158212 after the upgrade.

  6. Resolved

    Elastic

    This issue was resolved in the 9.5.3 release of Elasticsearch.  Customers running versions 9.5.0, 9.5.1, or 9.5.2 of Elasticsearch are advised to update to a newer version as soon as possible.