Feat: add dictionaries as a supported group column type - #23187
Open
Rich-T-kid wants to merge 5 commits into
Open
Feat: add dictionaries as a supported group column type#23187Rich-T-kid wants to merge 5 commits into
Rich-T-kid wants to merge 5 commits into
Conversation
Rich-T-kid
force-pushed
the
rich-T-kid/dictionary-groupValuesColumn-impl
branch
2 times, most recently
from
July 1, 2026 04:25
3f7ff57 to
e6b6dce
Compare
Contributor
Author
|
@kumarUjjawal could you run the dictionary benchmarks on this PR? Thx |
Rich-T-kid
commented
Jul 1, 2026
| } | ||
| } | ||
| DataType::Dictionary(key_dt, value_dt) => { | ||
| let new_field = Field::new("", *value_dt.clone(), true); |
Contributor
Author
There was a problem hiding this comment.
Since this field is never read again it may be fine to ignore the name field.
should be weary of similar issues to #21765 (comment)
Contributor
There was a problem hiding this comment.
Kind of annoying that make_group_column takes a field instead of a DataType. Maybe we can change that in a follow up PR?
Rich-T-kid
commented
Jul 1, 2026
Rich-T-kid
commented
Jul 1, 2026
Contributor
Author
|
@kumarUjjawal wanted to bump this 😄 |
Contributor
|
run benchmark dictionary_group_values |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/dictionary-groupValuesColumn-impl (eb41915) to 01bf68c (merge-base) diff using: dictionary_group_values File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-T-kid/dictionary-groupValuesColumn-impl
branch
2 times, most recently
from
July 1, 2026 15:32
770abfe to
243a557
Compare
Rich-T-kid
changed the title
Contributor
Author
|
@codex review |
Contributor
@Rich-T-kid Thank you! I have been sick so I won't be available for review. I will probably get back next week. |
Contributor
Author
@kumarUjjawal Sorry to hear that. I hope you feel better! no rush on the review! |
Rich-T-kid
commented
Jul 2, 2026
Rich-T-kid
force-pushed
the
rich-T-kid/dictionary-groupValuesColumn-impl
branch
from
July 2, 2026 19:18
4ee52da to
7af7080
Compare
Rich-T-kid
commented
Jul 4, 2026
Rich-T-kid
force-pushed
the
rich-T-kid/dictionary-groupValuesColumn-impl
branch
from
July 5, 2026 03:59
152c1f0 to
f3387c5
Compare
Contributor
Author
Latest PR speeds up the low cardinality case
|
Contributor
Author
|
@geoffreyclaude could you run the benchmarks command again when you get a chance. Thanks 🚀 |
Contributor
|
run benchmark dictionary_group_values |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/dictionary-groupValuesColumn-impl (3d1e1c9) to 01bf68c (merge-base) diff using: dictionary_group_values File an issue against this benchmark runner |
Rich-T-kid
commented
Jul 6, 2026
Rich-T-kid
left a comment
Rich-T-kid
left a comment
Contributor
Author
There was a problem hiding this comment.
Revision #4 Last revision before making this open for review
| cached_combined: Option<ArrayRef>, | ||
| /// Per-call group equality cache for the low-cardinality path. `Mutex` | ||
| /// because `vectorized_equal_to` takes `&self`; always uncontended. | ||
| group_eq_cache: Mutex<Vec<Option<bool>>>, |
Contributor
Author
There was a problem hiding this comment.
This is a work around needed because vectorized_equal_to takes in a &self as opposed to &mut self. This means the alternative is needing to allocate target_batch_size arrays for each intern() call instead of re-using the vector.
Originally planned to use RefCell but its not Send + Sync
Contributor
Author
There was a problem hiding this comment.
this shouldn't incur any overhead since there no contention but I'm happy to explore other ideas.
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
Contributor
Author
|
🤔 benchmarks show good improvement in every case but it should be much larger. Currently every |
Contributor
Author
|
seems to have failed @kumarUjjawal |
Contributor
Merged main into the pr. Let's see if that works. Will run benchmark |
Contributor
|
run benchmark dictionary_group_values |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/dictionary-groupValuesColumn-impl (2808d5f) to 774f0bc (merge-base) diff using: dictionary_group_values File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
Rich-T-kid
force-pushed
the
rich-T-kid/dictionary-groupValuesColumn-impl
branch
from
July 21, 2026 17:37
2808d5f to
aed9807
Compare
add schema support for dictionarys introduce high level GroupValuesColumn test introduce groupColumn trait test git issues introduce edge case/ regression section inital impl working implementation of dictionary for groupValuesCOlumns benchmarks show perf boost over groupvaluerows, TODO:dedupe items before inner append fix clippy errors & inline final builder add cache for arc ptr trim down test trim test LOC again trim PR revision 3 speed up low cardinlaity case working version introduce inter-batch caching break complex types into seperate parts fixed breaking test, re-allocate hashtable on each intern() call re-introduce cache remove mutex add cache to concat pointers to avoid un-needed allocations remove ptr caches and concat call reduce LOC revised PR comments add regression test to align with GroupValueRows & re-order overflow check add test to assert de-duplicated output dictionary tmp low card speed up optimize low-card case wip re-use allocations across calls add test final clean up
Rich-T-kid
force-pushed
the
rich-T-kid/dictionary-groupValuesColumn-impl
branch
from
July 21, 2026 17:41
aed9807 to
be4bc89
Compare
Contributor
Author
|
Okay finished with the PR updates. multi_col_groupby_dict_many_groups_two_values & append_only_stores_referenced_values This provides with better performance for low cardinality but this becomes pure overhead for the 100% unique case (8k/8k & 65k/65k). if a key types stores too many unique values it will exist with an error instead of panicking at the latest PR also avoid re-allocations for hashes and other vectors so this should future bridge the gap for the high cardinality case. There have been tested added in separate PR to avoid making this PR look much larger than it is. (half the LOC of the diff are test) sql logic test & advanced test @kumarUjjawal could you take another look when you get a chance. |
kumarUjjawal
left a comment
kumarUjjawal
left a comment
Contributor
There was a problem hiding this comment.
@Rich-T-kid I have left some comments for your consideration.
| let old_inner_len = self.inner.len(); | ||
| let all_inner_values = self.inner.take_n(old_inner_len); | ||
|
|
||
| let emitted = |
Contributor
There was a problem hiding this comment.
take_n emits all stored dictionary values, even when only a small group batch is requested, then copies the remaining values back into the builder. Repeated partial output can become O(groups² / batch_size) and can double peak memory.
Contributor
Author
There was a problem hiding this comment.
This is true but its a constraint of the GroupColumn trait. to keep this type agnostic we can only look at the actual values through GroupColumn::take_n()/GroupColumn::build(). The remaining groups' inner slots are not a contiguous prefix of inner, they can be scattered at any position, so there is no way to partially drain inner without losing values still needed. We must drain everything into a plain array first, then re-insert only what the remaining groups still reference.
This is also a known pattern:
Contributor
Author
|
@kumarUjjawal I've added test to address your comments in 7566a13 Let me know if you have any other suggestions |
Contributor
|
@zhuqi-lucas @alamb do you have any thoughts on this? |
Contributor
|
run benchmark dictionary_group_values |
|
Benchmark for this request failed. Last 20 lines of output: Click to expandFile an issue against this benchmark runner |
Contributor
|
Some regressions before, need to verify again: dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_65536_null_0.10 1.00 17.3±0.17ms 14.4 MElem/sec 1.27 22.0±0.25ms 11.4 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_8192_null_0.10 1.00 1459.5±8.00µs 21.4 MElem/sec 1.18 1721.9±6.11µs 18.1 MElem/sec |
Contributor
|
run benchmark dictionary_group_values |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/dictionary-groupValuesColumn-impl (38d656f) to e8a65f2 (merge-base) diff using: dictionary_group_values File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
Contributor
|
Latest run, still one regression need to investigate: dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_65536_null_0.10 1.00 16.8±0.07ms 14.9 MElem/sec 1.19 19.9±0.10ms 12.5 MElem/sec |
zhuqi-lucas
reviewed
Jul 27, 2026
zhuqi-lucas
reviewed
Jul 27, 2026
zhuqi-lucas
reviewed
Jul 27, 2026
zhuqi-lucas
reviewed
Jul 27, 2026
Rich-T-kid
force-pushed
the
rich-T-kid/dictionary-groupValuesColumn-impl
branch
from
July 27, 2026 14:43
38d656f to
fb60ab3
Compare
Contributor
Author
|
thank you for the review @zhuqi-lucas, the two latest PR's should address your comments + test. @kumarUjjawal @zhuqi-lucas can we run the benchmarks one more time to make sure theres no meaningful regressions |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
This PR introduces a specialized
GroupColumnimplementation for dictionary-typed columns insideGroupValuesColumn, allowing dictionary columns to participate in the columnar, vectorized aggregation path instead of the row-based fallback.The Implementation is only about 175+ lines of code. the remaining LOC is adding extensive test at the
GroupColumntrait level as well as testing theGroupValuesColumnGroupValues trait and how it inter-opts with multi-dictionary group by's.What changes are included in this PR?
DictionaryGroupValueBuilderstruct implementing theGroupColumntrait forDictionary-typed group-by columns, supporting a configurable subset of value typesGroupValuesColumn::try_new(thematches!block) to acceptDictionary(_, value_type)wherevalue_typeis already supported.emitAre these changes tested?
yes. a majority of this PR is test
Are there any user-facing changes?
no. this is a pure perf boost for users.