fix: preserve aggregate filter pushdown order - #22926
Conversation
|
Hi, this PR fixes #22925 and is still waiting for review. The only failing check is Thanks! |
05e5a77 to
3626a96
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #22926 +/- ##
==========================================
+ Coverage 80.70% 80.75% +0.04%
==========================================
Files 1095 1096 +1
Lines 372554 373588 +1034
Branches 372554 373588 +1034
==========================================
+ Hits 300665 301677 +1012
+ Misses 53921 53906 -15
- Partials 17968 18005 +37 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
neilconway
left a comment
There was a problem hiding this comment.
Thank you for this PR! Overall, this is excellent work: clear diagnosis, minimal and correct fix, and good test coverage. I'm sorry it didn't get reviewed sooner -- a few minor suggestions but it should be good to land shortly.
3626a96 to
4b0e6c6
Compare
neilconway
left a comment
There was a problem hiding this comment.
Looking good! Can you update the PR description, please? Currently refers to the old implementation; would also be good to elaborate on the pushdown change for global aggregates?
|
Updated the PR description to reflect the current |
Which issue does this PR close?
Rationale for this change
FilterPushdownmaps each child pushdown result back to its original parentfilter by position.
AggregateExec::gather_filters_for_pushdownpreviouslysplit parent filters into safe and unsafe buckets and then concatenated the
results, which changed their order.
For example, in
cnt@2 = 1 AND b@1 = bar, the grouping-column predicate onbcan cross the aggregate, while the predicate on aggregate outputcntmust remain above it. Reordering the returned results could make the optimizer
associate the pushed-down
bresult with thecntpredicate and incorrectlyremove the latter.
Aggregate pushdown also needs to account for empty-input semantics. A global
aggregate, or a grouping-sets aggregate containing
(), can emit a row evenwhen its input is empty. Moving any parent predicate below such an aggregate —
including a column-free predicate such as
false— can therefore change theresult.
What changes are included in this PR?
with
ChildFilterDescription::from_child_with_allowed_indices.aggregate-result columns remain above the aggregate.
containing an empty grouping set, while retaining aggregate-generated
dynamic filters.
before positional remapping.
global-aggregate name collisions, constant predicates, and grouping sets.
Are these changes tested?
Yes:
cargo test -p datafusion --test core_integration physical_optimizer::filter_pushdowncargo fmt --all -- --checkcargo check -p datafusion-physical-plan -p datafusion-physical-optimizergit diff --check upstream/mainAre there any user-facing changes?
There are no public API changes. Filter pushdown now preserves mixed aggregate
predicates correctly and avoids moving predicates across aggregates when doing
so could change empty-input results.