Add vec0 optimize command: compact sparse chunks after deletions#269
Draft
asg017 wants to merge 1 commit intoasg017/delete-clear-implfrom
Draft
Add vec0 optimize command: compact sparse chunks after deletions#269asg017 wants to merge 1 commit intoasg017/delete-clear-implfrom
asg017 wants to merge 1 commit intoasg017/delete-clear-implfrom
Conversation
Implements FTS5-style INSERT INTO v(v) VALUES ('optimize') command that
packs live entries from newer/sparser chunks into free slots of older
chunks, then deletes emptied chunks. Adds hidden command column to vtab
schema, command dispatcher in xUpdate, and two-pointer compaction
algorithm that handles vectors, all metadata types, and partitioned tables.
Includes 16 Python tests, 7 C unit tests, and a libFuzzer target.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #268
Adds a FTS5-style 'optimize' command to vec0 virtual tables.
Vectors in vec0 tables are stored in contiguous blocks next to each other. When vectors are deleted, their validity bit and data are zero'ed on.
On future inserts, sometimes vec0 will re-fill that space with new data, but not always. Meaning, a vec0 table can be larger than it needs to be, with "holes" across various chunks.
The new
'optimize'command allows a developer to start an "optimize" procedure on a specific vec0 table. It will re-arrange vectors/chunks to compact space (a defragmenter, if you will).I'm a little less sure of this change. Updating/deleting data is super important so I want to make sure I get this right.