perf: share IVF partition search across deltas

[Xuanwo]

This avoids re-ranking the same IVF centroids once per delta index during ANN search by sharing a single partition search across deltas when their metric and partition layout match.

When the active deltas disagree on metric type or partition count, the query falls back to the existing per-delta path so older inconsistent layouts continue to behave safely.

The knn test suite now covers the guard behavior and the updated partition-ranking metric semantics.

---

I believe this is the key reason why delta indexes perform so poorly for us.

I understand that can_share_partition_search is not an exact solution. If this approach does not work, we may need to add more metadata in our index proto to indicate that those segments can be searched together.

Also, I think @wjones127's #6207 could be helpful.

[github-actions]

PR Review: perf: share IVF partition search across deltas

Overall this is a clean, well-structured optimization. The approach of sharing a single find_partitions call across deltas is sound given that delta indices always reuse centroids from the original index.

P1: can_share_partition_search guard is a heuristic, not exact

The guard checks metric_type and total_partitions, but these don't strictly guarantee the same centroids. In current practice this is safe because delta indices always clone their IVF model from the first existing index (optimize_vector_indices_v2 uses existing_indices[0].ivf_model()). However, a future code change could break this assumption silently.

Consider either:

• Adding a brief comment in can_share_partition_search explaining why matching metric + partition count is sufficient (i.e., "delta indices share centroids by construction"), or
• Comparing the actual centroid data (e.g., pointer equality on the FixedSizeListArray) for a stronger guarantee. The ivf_model() method is already on the VectorIndex trait.

Minor: streaming → collect tradeoff

The old path streamed batches via .buffered() as they were produced. The new shared path collects all batches into a Vec before streaming them out. This is fine given each batch is a single row (one per delta), but worth noting for anyone extending this pattern to larger payloads.

---

Tests cover the three guard-condition branches (same layout, partition mismatch, metric mismatch) and the metric assertion update (100 * num_deltas → 100) correctly validates the optimization is applied. Looks good.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 55.91837% with 108 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/io/exec/knn.rs	55.91%	101 Missing and 7 partials ⚠️

📢 Thoughts on this report? Let us know!

[Xuanwo]

This PR is converted into a draft because we are not sure about the route we wanna go.