[feature](inverted index) support multiple tokenize index in one column #60415
+3,924
−275
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…mn (apache#59117) This PR implements **Multi-Analyzer Inverted Index** feature, which allows creating multiple inverted indexes with different analyzers on a single column. 1. **Multiple Indexes on Single Column**: Create multiple inverted indexes with different analyzers (standard, keyword, chinese, custom) on the same column 2. **USING ANALYZER Syntax**: Query with specific analyzer using `MATCH ... USING ANALYZER analyzer_name` 3. **Smart Index Selection**: When specified analyzer's index is not built, automatically falls back to non-index path (correct results guaranteed) 4. **Analyzer Identity Detection**: Prevents duplicate indexes with same analyzer configuration - Multi-language search on same text column - Precision vs. recall trade-off (exact match vs. fuzzy search) - Autocomplete with edge_ngram while keeping standard search ```sql -- Create table with multiple indexes CREATE TABLE articles ( id INT, content TEXT, INDEX idx_std (content) USING INVERTED PROPERTIES("analyzer" = "std_analyzer"), INDEX idx_kw (content) USING INVERTED PROPERTIES("analyzer" = "kw_analyzer") ) ...; -- Query with specific analyzer SELECT * FROM articles WHERE content MATCH 'hello' USING ANALYZER std_analyzer; SELECT * FROM articles WHERE content MATCH 'hello' USING ANALYZER kw_analyzer; ```
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
- Add getAnalyzerIdentity() method to IndexDef (branch-4.0 uses IndexDef, not IndexDefinition) - Fix MatchPredicate to use setNullableFromNereids() instead of direct field assignment - Change IndexDefinition.IndexType to IndexDef.IndexType in Index.java - Remove unrelated ColumnSeqMapping methods from OlapTable (not present in branch-4.0)
Member
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Feb 2, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Cherry-pick of #59117 to branch-4.0.
This PR adds support for multiple tokenize indexes on a single column, allowing users to create multiple inverted indexes with different analyzers (e.g., chinese, english, standard) on the same text column and query using specific analyzers.
Key changes:
USING ANALYZERsyntax for MATCH predicatesfilterIndexesByAnalyzermethod in OlapTable for index selectionConflicts Resolved
MatchPredicate.java- Updated constructor signatures for analyzer supportOlapTable.java- Added imports and new filtering methodsExpressionTranslator.java- Updated to use new MatchPredicate constructorTest plan
Check List (For Author)
Test
Behavior changed:
USING ANALYZERsyntax for MATCH predicates.Does this need documentation?
Check List (For Reviewer who merge this PR)