Search Quality Degredation

Resolved

Started

Lasted 28h 17m

View official incident ↗

Updates

  1. Resolved

    We have identified all indices which are half complete and ran manual updates to repair them.

  2. Monitoring

    We have identified the root cause of fragmented indexes as a step in our ingestion pipeline when embing documents. A fix has been rolled out over night and we have not noticed any further ingestion failures. We identified and fixed 50 deployment's indexes and are doing a final scan to ensure all indexes are healthy.

  3. Identified

    We have identified a fix that will repair most broken indexes and are working on deploying it now.

  4. Investigating

    We are investigating reports of specific search indexes being partially degraded. We are identifying the root cause and formulating a solution to repair all broken indexes. Any customers that feel that they are affected should run a manual update which will repair their index automatically.