[fr-par-1] Latencies spikes during the day following capacity increase in one of block storage cluster

Resolved

Minor impact

Started

Lasted 171h 35m

View official incident ↗

Updates

  1. Resolved

    This incident has been resolved.

  2. Monitoring

    All new servers have been fixed. We are still increasing data density in the servers and monitoring any latency spikes.

  3. Identified

    We are continuing to work on a fix for this issue.

  4. Identified

    The fix improved nvme behavior during the day and today we applied it on half of the concerned rack. As data must be removed first to apply the fix, migrations operations are processing slowly to avoid worsening impact on latencies. Migration operations will continue through the week-end.

  5. Identified

    We applied a fix on one of the concerned servers and monitoring it, we will have more significant metrics tomorrow.

  6. Identified

    Since monday 26 october on the morning, we noticed increased periodic spike of latencies following the integration of a new rack in one of our cluster. Those spikes are more noticeable during the day and are linked to writes done on some of the new disks. This issue also impact MRDB product. We limited the impact and currently we are working on a fix. An update will be posted at the end of the day.