[MAJOR] Equivalence System Overhaul by ozankabak · Pull Request #16217 · apache/datafusion

ozankabak · 2025-05-30T13:17:03Z

Which issue does this PR close?

Closes Exponential planning time (100s of seconds) with UNION and ORDER BY queries #13748.

Rationale for this change

DF's theoretical approach to representing orderings, equivalences and constants is sound, but it suffers from many implementation problems that lead to various issues. Some of these issues include:

High-complexity planning times for queries involving UNION and ORDER BY clauses (see the referred issue).
A continuous stream of subtle bugs that arise from the fact that our objects allow multiple representations of the same underlying state (e.g. a non-existing ordering is sometimes represented by a None value for a type Option<LexOrdering>, and sometimes with a Some value with the payload being "empty").
The inability to relay ordering preference information to optimizer rules such as EnforceSorting, which results in missed optimization opportunities. For example, a windowing operator prefers its input to be sorted w.r.t. both PARTITION BY and ORDER BY keys, but if the former is not available, it is still preferable to have just the latter instead of a fully unsorted input.

What changes are included in this PR?

The equivalence system is rewritten to solve the computational complexity problem. The new system avoids handling constants separately from other equivalences and treats them on an equal footing within the same mechanism, avoiding unnecessary work during ordering_satisfy checks. Also, the new system caches normalization results for equivalent orderings, which was one of the major sources of excessive computational complexity during sort enforcement.
The LexOrdering and LexRequirement objects do not allow degenerate orderings anymore, whenever you see this object, you can be sure there will be some ordering. Possible absence of an ordering will henceforth be properly represented by an Option<LexOrdering>.
The OrderingRequirements object is introduced to capture ordering preferences of operators.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes -- this is a major API change and will break old code. We spent months looking for ways to (1) break this PR down into smaller ones, (2) yet lump all API changes in one PR -- but couldn't find an effective way to do this.

Therefore, I invite all contributors who are also downstream stakeholders to help by participating in a "pre-scream" test. Please test and report how this PR breaks your code, so we can collectively prepare an upgrade guide for the community at large.

Tips and Tricks for Reviewing

The "meat" of the code is in the following files:

datafusion/physical-expr-common/src/sort_expr.rs: Definition of OrderingRequirements, changes to LexOrdering and friends.
datafusion/physical-plan/src/execution_plan.rs: API change to required_input_ordering.
datafusion/physical-expr/src/equivalence: Re-implementation of the equivalence system.
datafusion/functions-aggregate/src: Effects of the above on aggregate function implementations.

The rest of the changes are "mandatory" propagative effects of fixing a fundamental low-level mechanism in a large codebase, and mostly trivial.

@alamb, @andygrove, @berkaysynnada -- feel free to tag others.

remove usage of prefer_existing_sort as default set requirements Hard set soft on AggregateExec and BoundedWindowExec since they have InputOrderMode functionalities

simplify indentation

remove prefer_existing_sort based test cases

# Conflicts: # datafusion/core/src/datasource/listing/table.rs # datafusion/physical-optimizer/src/enforce_sorting/mod.rs

# Conflicts: # datafusion-examples/examples/custom_file_format.rs # datafusion/core/src/datasource/file_format/csv.rs # datafusion/core/src/datasource/file_format/json.rs # datafusion/core/src/datasource/file_format/mod.rs # datafusion/core/src/datasource/file_format/parquet.rs # datafusion/core/tests/physical_optimizer/enforce_distribution.rs # datafusion/core/tests/physical_optimizer/test_utils.rs # datafusion/physical-optimizer/src/enforce_sorting/sort_pushdown.rs

return alternative on BoundedWindowAggExec

# Conflicts: # datafusion/catalog/src/stream.rs # datafusion/core/tests/physical_optimizer/enforce_distribution.rs # datafusion/physical-plan/src/joins/sort_merge_join.rs # datafusion/physical-plan/src/joins/symmetric_hash_join.rs

simplify sort_pushdown.rs

add requirements compatible test cases

viirya

I left some comments after (roughly) reviewing followed after @alamb's call on dev. 🙂
It is roughly because the size of the changes. A 7000+ lines PR is too hard to review very in details.

Overall the change looks good and it solves the complexity issue reported. 👍

Although the PR claims that it couldn't find an effective way to split it to smaller ones. I'm not very sure but I can see there are changes that are good to have but maybe not so urgent in just one shot, e.g., a lot of code refactors seems not core change, code comment rephrasing, API change that seems can be in separate works.

For such big overhaul, I still prefer and suggest we can it in incremental approach. It will help reviewing a lot.

ozankabak · 2025-06-04T16:30:44Z

Thanks for all the reviews. I went through each and sent a commit that addresses all the suggestions, I also tried to answer any questions I saw.

I will later resolve the conflicts and get this ready to go. I see that we are close to getting ready for the 48 release too, so it seems like we are making a good effort to align them in the way we want.

Thanks all 🚀

ozankabak · 2025-06-04T21:31:29Z

I removed the conflicts, this is in a good state now. Did we get to the 48 cut-off point yet?

Omega359 · 2025-06-05T12:27:18Z

I plan on running my test suite against this side branch hopefully tomorrow or early next week to verify it.

alamb · 2025-06-06T20:33:06Z

I removed the conflicts, this is in a good state now. Did we get to the 48 cut-off point yet?

~~We are close I think~~

Update yes I think main is open for new features and we can merge this one (though maybe it would be a good idea for @Omega359 to test before we do so)

ozankabak · 2025-06-06T21:59:36Z

I think there are some incoming big PRs, so I'd like to go ahead with this and help with fixing any issues @Omega359 finds after merge. In addition to the time cost of going through merges, I am fearful of accidentally introducing bugs every time I merge.

ozankabak · 2025-06-06T22:29:28Z

OK, I resolved the conflicts and all seems OK. Should we go ahead after CI passes?

Omega359 · 2025-06-06T23:51:53Z

I'll be able to do my tests Sat afternoon or Sun morning. I'm good with doing that against either main or the branch.

ozankabak · 2025-06-07T06:40:49Z

Thanks @Omega359. Given that we didn't hear any concerns yesterday, I will go ahead and merge this.

Thanks everyone for all the reviews!

alamb · 2025-06-07T10:33:47Z

🚀 woohoo!

Omega359 · 2025-06-08T13:08:08Z

fyi I did run this through my test suite and it seemed to work correctly :)

* introduce Soft & Hard RequiredInputOrderings remove usage of prefer_existing_sort as default set requirements Hard set soft on AggregateExec and BoundedWindowExec since they have InputOrderMode functionalities * add documentation to replace_with_partial_sort simplify indentation * add documentation to analyze_immediate_sort_removal simplify indentation * remove prefer_existing_sort effects remove prefer_existing_sort based test cases * remove prefer_existing_sort configuration * remove prefer_existing_sort configuration * add documentation * add documentation * add documentation * fix imports and test cases * fix imports and test cases * implement RequiredInputOrdering as vectors * implement RequiredInputOrdering as vectors return alternative on BoundedWindowAggExec * fix test cases * change doc * revert prefer_existing_sort flag * fix changes * fix test case * make LexRequirement private * ensure RequiredInputOrdering inner requirement can not be empty simplify sort_pushdown.rs * add default test cases add requirements compatible test cases * doc fixes * fix clippy and docs * format code * format code * doc fix * add TODO test cases with test_soft_hard_requirements prefix * Review Part 1 * Review Part 2 * Review Part 3 * Review Part 4 * Review Part 5 * Review Part 6 * Enforce non-degeneracy for LexRequirement * Enforce non-degeneracy for LexOrdering (Part 1) * Enforce non-degeneracy for LexOrdering (Part 2) * fix first phase of merge conflicts and other bugs * Fix sqllogictests except the schema mismatch * Cleanup Part 1 * Cleanup Part 2 * Cleanup Part 3 * do not initialize Trivial accumulators if ordering is set * initialize TrivialFirstPrimitiveGroupsAccumulator struct and return * fix clippy * fix merge conflicts * fix typos remove TrivialFirstPrimitiveGroupsAccumulator make groups accumulator available only when order requirement is set * format code * Add requirement_satisfied back in * Replace AsRef with ordinary & for LexOrdering * Further cleanup * add OutputRequirementExec fetches to sort adding * Simplify remove_redundant_entries * Work with iterators in ordering_satisfy_requirement * Fix doctests * Cleanup LexOrdering APIs * Cleanup LexOrdering APIs 2 * Add reverse_each to LexOrdering * Use LexOrdering instead of Arc<[PhysicalSortExpr]> * Use PhysicalSortExpr slices in contexts where we simply list sort expressions * Generalize add_new_ordering APIs * Simplifications * More cleanups * API Simplifications * Improve comments * Use vector in Expr structs * Fix doctests * Simplify sort * Simplify the get_finer_aggregate_exprs_requirement function * Avoid hidden clones * bugfix * Simplify the get_finer_aggregate_exprs_requirement function * Simplify the function with_reorder * Fix with_reorder bug * Simplify the function with_reorder (Part 2) * Simplify * DRY * Simplifications * Improve add_equal_condition * Improve docs * Simplifications * Simplifications * RequiredInputOrdering -> OrderingAlternatives * Simplify new_with_orderings * Transition to fallible LexOrdering constructor * Transition to fallible LexOrdering constructor - 2 * Transition to fallible LexOrdering constructor - 3 * Transition to fallible LexOrdering constructor - 4 * Transition to fallible LexOrdering constructor - 5 * Transition to fallible LexOrdering constructor - 6 * Transition to fallible LexOrdering constructor - 7 * Transition to fallible LexOrdering constructor - 8 * Transition to fallible LexOrdering constructor - 9 * Transition to fallible LexOrdering constructor - 10 * Transition to fallible LexOrdering constructor - 11 * Simplify constant expressions * Simplify constant expressions - 2 * Simplify constant expressions - 3 * Simplify constant expressions - 4 * Simplify constant expressions - 5 * Simplify constant expressions - 6 * Simplify constant expressions - 7 * Simplify constant expressions - 8 * Simplify constant expressions - 9 * Fix imports * Remove explicit constant tracking from equivalences * Resolve logical conflict * Remove the unusual take API, instead use the from trait * Simplify projection mapping - 1 * Use a map instead of a vector in ProjectionMapping * Simplify DependencyMap * Simplify DependencyMap - 2 * Simplify DependencyMap - 3 * Incorporate Jay's suggestions * Simplifications * Fix doctest * Improve docstrings * Update/cast the constant value accordingly when schema changes * Improve ProjectionMapping * Remove DerefMut from ProjectionTargets to preserve non-emptiness * Docstring * Optimize project_expr by fetching equivalence classes only once * Project multiple expressions more efficiently at once * Project multiple expressions more efficiently at once - 2 * Project multiple expressions more efficiently at once - 3 * Project multiple expressions more efficiently at once - 4 * Move normalization of sort expressions to equivalence group * Improve comments * Improve display for EquivalenceProperties * More idiomatic code * More succinct code * Remove extend_orderings from EquivalenceProperties * Simplify with_reorder * Store normalized orderings - 1 * Reduce time complexity of normalization w.r.t. number of equivalence classes * Simplify bridge_classes logic * Remove TODOs * Simplify generate_dependency_orderings * normalized orderings - 2 * normalized orderings - 3 * undo normalized orderings * Fix logical conflicts * Fix imports * Remove noop code * Move add_offset_to_expr * Remove mutation from LexOrdering * Remove unwraps * Remove unwraps - 2 * Remove unwraps - 3 * Remove unwraps - 4 * Remove collapse from LexOrdering * Remove unwraps - 5 * Remove unwraps - 6 * Remove unwraps - 7 * Remove unwraps - 8 * Remove unwraps - 9 * Remove unwraps - 10 * Remove collapse from LexRequirement * Simplify ordering_satisfy * Enforce uniqueness in LexOrdering * Fix with_reorder * Use tee * Fix reorder api * Comment grammar * Remove unwraps * Cache normalized orderings * Minor: remove an unecessary clone in common_sort_prefix_length * Address reviews --------- Co-authored-by: mertak-synnada <mertak67+synaada@gmail.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Berkay Şahin <124376117+berkaysynnada@users.noreply.github.com>

alamb · 2025-08-07T19:54:13Z

We found a regression related to this PR for string_agg not respecting ORDER BY

string_agg does not respect ORDER BY on 49.0.0 #17011

@nuno-faria and @findepi have prepared PRs to fix this, and it would be great if @ozankabak you or someone on your team could also help review it

[branch-49] fix: string_agg not respecting ORDER BY #17058

berkaysynnada · 2025-08-08T06:36:26Z

We found a regression related to this PR for string_agg not respecting ORDER BY

string_agg does not respect ORDER BY on 49.0.0 #17011

@nuno-faria and @findepi have prepared PRs to fix this, and it would be great if @ozankabak you or someone on your team could also help review it

[branch-49] fix: string_agg not respecting ORDER BY #17058

I'll take a look at that today

…t whether they keep the same ordering of the input (apache#23807) ## Which issue does this PR close? - Closes apache#23798 Related to: - apache#16217 ## Rationale for this change To be able to keep the same sorting order allowing for more optimizations Now comet own `cast` implementation can be recognized as not modifying sort order in the same cases that datafusion cast does. ## What changes are included in this PR? added `strictly_order_preserving` property to `ExprProperties` + varius other places and replaced the hard coded logic for cast (`substitute_cast_ordering`) about keeping input order with more generic approach that now any expression can implement and have the same advantage of sort elimination also marked from_unixtime as keeping ordering to show a case of this optimization ## Are these changes tested? yes ## Are there any user-facing changes? yes, breaking change, added `strictly_order_preserving` property to `ExprProperties` and to `FFI_ExprProperties` this property means that given expression `f` and 2 values from the input column `a` and `b` the following variants are kept: 1. `a.cmp(b) == f(a).cmp(f(b))` 2. nulls maps to nulls Example of satisfying expression: `cast(col_a as BIGINT)` where `col_a` is `INT` it is keeping the properties Example of not satisfying: `floor` - floor can not `array_repeat(my_col, 2)` which might look like at first glance as keeping the property as well but in fact it does not. the reason is that `array_repeat(null, 2)` will output list of 2 nulls which breaks the 2nd property that nulls must be kept as nulls how to migrate: Option 1 - keeping the old behavior (safest but least performant) set `strictly_order_preserving` to false Option 2 - Using the new optimization that this opens up: set `strictly_order_preserving` only if the expression keep both variants

mertak-synnada added 30 commits February 12, 2025 15:22

introduce Soft & Hard RequiredInputOrderings

5df66e7

remove usage of prefer_existing_sort as default set requirements Hard set soft on AggregateExec and BoundedWindowExec since they have InputOrderMode functionalities

add documentation to replace_with_partial_sort

7c9f6d3

simplify indentation

add documentation to analyze_immediate_sort_removal

3c76bdc

simplify indentation

remove prefer_existing_sort effects

265e00e

remove prefer_existing_sort based test cases

remove prefer_existing_sort configuration

23e0bf6

remove prefer_existing_sort configuration

ee626d1

add documentation

0b98382

add documentation

a41dfec

add documentation

f6dae9d

Merge branch 'refs/heads/apache_main' into soft-required-input-ordering

419e8ab

# Conflicts: # datafusion/core/src/datasource/listing/table.rs # datafusion/physical-optimizer/src/enforce_sorting/mod.rs

fix imports and test cases

d551623

fix imports and test cases

9a1b5b5

implement RequiredInputOrdering as vectors

77e97fd

implement RequiredInputOrdering as vectors

1d90a71

return alternative on BoundedWindowAggExec

fix test cases

55ef9aa

change doc

ea2583b

revert prefer_existing_sort flag

c83862f

fix changes

87dd95e

fix test case

24dba16

make LexRequirement private

8635d5e

ensure RequiredInputOrdering inner requirement can not be empty

ab28421

simplify sort_pushdown.rs

add default test cases

cb8f763

add requirements compatible test cases

doc fixes

1c1f116

fix clippy and docs

7817b4f

format code

6b43982

format code

55b9cf1

Merge branch 'refs/heads/apache_main' into required-input-ordering

156b84c

doc fix

41e4292

alamb mentioned this pull request Jun 3, 2025

Improve sql planing performance (optimize try_process_unnest) #16242

Closed

viirya reviewed Jun 3, 2025

View reviewed changes

Comment thread datafusion/physical-plan/src/sorts/sort.rs

viirya reviewed Jun 3, 2025

View reviewed changes

Comment thread datafusion/physical-expr-common/src/sort_expr.rs

viirya reviewed Jun 3, 2025

View reviewed changes

Address reviews

74d023b

ozankabak added 2 commits June 4, 2025 23:59

Merge branch 'apache_main' into required-input-ordering

0d4e8b2

Merge branch 'apache_main' into required-input-ordering

32de761

Merge branch 'apache_main' into required-input-ordering

bde28fc

ozankabak merged commit 1daa5ed into apache:main Jun 7, 2025

comphead mentioned this pull request Jul 25, 2025

fix: TrivialValueAccumulators to ignore null states for ignore nulls #16918

Closed

vegarsti mentioned this pull request Jul 31, 2025

fix: UnnestExec preserves relevant equivalence properties of input #16985

Merged

This was referenced Aug 5, 2025

string_agg does not respect ORDER BY on 49.0.0 #17011

Closed

[branch-49] fix: string_agg not respecting ORDER BY #17058

Merged

Omega359 mentioned this pull request Aug 20, 2025

Planning time for queries with many columns with union and order by is very slow #17261

Open

crepererum mentioned this pull request Aug 29, 2025

FileScanConfig::output_ordering must be vector of optionals #17354

Open

rluvaton mentioned this pull request Jul 22, 2026

ScalarUdfImpl::strictly_order_preserving: Allow expression to report whether they keep the same ordering of the input #23807

Merged

Uh oh!

Conversation

ozankabak commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Tips and Tricks for Reviewing

Uh oh!

Uh oh!

Uh oh!

viirya left a comment

Choose a reason for hiding this comment

Uh oh!

ozankabak commented Jun 4, 2025

Uh oh!

ozankabak commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Omega359 commented Jun 5, 2025

Uh oh!

alamb commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ozankabak commented Jun 6, 2025

Uh oh!

ozankabak commented Jun 6, 2025

Uh oh!

Omega359 commented Jun 6, 2025

Uh oh!

ozankabak commented Jun 7, 2025

Uh oh!

alamb commented Jun 7, 2025

Uh oh!

Omega359 commented Jun 8, 2025

Uh oh!

alamb commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

berkaysynnada commented Aug 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ozankabak commented May 30, 2025 •

edited

Loading

ozankabak commented Jun 4, 2025 •

edited

Loading

alamb commented Jun 6, 2025 •

edited

Loading

alamb commented Aug 7, 2025 •

edited

Loading