refactor(physical-plan): Simplify ExecutionPlan API with replace_children

[JSOD11]

Which issue does this PR close?

• Closes refactor(physical-plan): simplify ExecutionPlan children-replacement API #23441

Deprecating with_new_children and with_new_children_and_same_properties to reduce API surface area

As noted here, while the addition of with_new_children_and_same_properties has the benefit of skipping potentially expensive computation in the case that replacement children have the same properties as the original children, it widens the API surface area of ExecutionPlan in a way that could be confusing for users.

replace_children and with_new_children_if_necessary

Thus, to rectify this, we unify these methods by introducing replace_children, and we shift towards with_new_children_if_necessary as the universal entry point for replacing the children of an ExecutionPlan. replace_children simplifies the interface for users by taking an enum called ChildrenPropertiesHint as an argument. The enum has two variants, SameProperties and Recompute, which function as a hint to replace_children from the caller as to whether or not the properties need to be recomputed.

Trait implementation migration

To migrate from with_new_children and with_new_children_and_same_properties to replace_children, I went through all 93 implementations of with_new_children and implemented replace_children with a match statement matching on the ChildrenPropertiesHint. In the case that the properties match the children, ChildrenPropertiesHint::SameProperties, and we have an implementation of with_new_children_and_same_properties, then we follow the body of with_new_children_and_same_properties. In the case that the properties do not match the children, ChildrenPropertiesHint::Recompute, we follow the body of with_new_children. In the cases in which there was no implementation of with_new_children_and_same_properties, I simply move the body of with_new_childrenintoreplace_children` and ignore the hint.

Example

For example, here is what the implementation looks like for FilterExec after this change:

    fn replace_children(
        self: Arc<Self>,
        mut children: Vec<Arc<dyn ExecutionPlan>>,
        hint: ChildrenPropertiesHint,
    ) -> Result<Arc<dyn ExecutionPlan>> {
        validate_child_count!(self, children);
        match hint {
            ChildrenPropertiesHint::SameProperties => Ok(Arc::new(Self {
                input: children.swap_remove(0),
                metrics: ExecutionPlanMetricsSet::new(),
                ..Self::clone(&*self)
            })),
            ChildrenPropertiesHint::Recompute => {
                let new_input = children.swap_remove(0);
                FilterExecBuilder::from(&*self)
                    .with_input(new_input)
                    .build()
                    .map(|e| Arc::new(e) as _)
            }
        }
    }

We see here that in the case that the hint suggests the properties are the same, we can simply swap the children without having to recompute the properties. In the case that the properties are not the same, we create a new node from scratch. We achieve this functionality by moving the hint calculations definitively into with_new_children_if_necessary rather than having them scattered around many methods. However, for this to all work we must ensure that users actually do use with_new_children_if_necessary by making it obvious to them somehow. I feel replace_children is a step in the right direction, but it could still be easy for a user to miss with_new_children_if_necessary and just jump to using replace_children instead.

Usage Migration

replace_children is called from with_new_children_if_necessary, which is the standard entry point that should be used for replacing the children of a node.

To model the intended behavior for our users, I took the time here to migrate usages of with_new_children and with_children_and_same_properties to with_new_children_if_necessary where it made sense to do so, and I migrated with_new_children_if_necessary to use replace_children with the correct hint filled in at each branch.

Testing

[codecov-commenter]

Codecov Report

❌ Patch coverage is 45.86173% with 556 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.63%. Comparing base (88365dd) to head (3c9d8e4).
⚠️ Report is 39 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/physical-plan/src/test/exec.rs	0.00%	48 Missing ⚠️
datafusion/core/src/physical_planner.rs	40.98%	36 Missing ⚠️
datafusion/physical-plan/src/execution_plan.rs	63.41%	29 Missing and 1 partial ⚠️
...ysical-plan/src/joins/piecewise_merge_join/exec.rs	34.78%	30 Missing ⚠️
datafusion/physical-plan/src/sorts/partial_sort.rs	0.00%	23 Missing ⚠️
datafusion/physical-plan/src/buffer.rs	0.00%	22 Missing ⚠️
datafusion/physical-plan/src/async_func.rs	0.00%	21 Missing ⚠️
...usion/physical-plan/src/operator_statistics/mod.rs	0.00%	17 Missing ⚠️
datafusion/physical-plan/src/sorts/sort.rs	19.04%	16 Missing and 1 partial ⚠️
datafusion/physical-plan/src/coalesce_batches.rs	36.00%	16 Missing ⚠️
... and 39 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23903      +/-   ##
==========================================
- Coverage   80.65%   80.63%   -0.02%     
==========================================
  Files        1093     1096       +3     
  Lines      371619   374158    +2539     
  Branches   371619   374158    +2539     
==========================================
+ Hits       299730   301708    +1978     
- Misses      53993    54456     +463     
- Partials    17896    17994      +98     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v54.1.0 (current)
       Built [ 110.017s] (current)
     Parsing datafusion v54.1.0 (current)
      Parsed [   0.037s] (current)
    Building datafusion v54.1.0 (baseline)
       Built [ 106.974s] (baseline)
     Parsing datafusion v54.1.0 (baseline)
      Parsed [   0.038s] (baseline)
    Checking datafusion v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.943s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 219.909s] datafusion
    Building datafusion-catalog v54.1.0 (current)
       Built [  39.222s] (current)
     Parsing datafusion-catalog v54.1.0 (current)
      Parsed [   0.027s] (current)
    Building datafusion-catalog v54.1.0 (baseline)
       Built [  38.909s] (baseline)
     Parsing datafusion-catalog v54.1.0 (baseline)
      Parsed [   0.025s] (baseline)
    Checking datafusion-catalog v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.155s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  79.381s] datafusion-catalog
    Building datafusion-datasource v54.1.0 (current)
       Built [  38.032s] (current)
     Parsing datafusion-datasource v54.1.0 (current)
      Parsed [   0.033s] (current)
    Building datafusion-datasource v54.1.0 (baseline)
       Built [  38.119s] (baseline)
     Parsing datafusion-datasource v54.1.0 (baseline)
      Parsed [   0.034s] (baseline)
    Checking datafusion-datasource v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.365s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  78.607s] datafusion-datasource
    Building datafusion-ffi v54.1.0 (current)
       Built [  60.034s] (current)
     Parsing datafusion-ffi v54.1.0 (current)
      Parsed [   0.063s] (current)
    Building datafusion-ffi v54.1.0 (baseline)
       Built [  59.884s] (baseline)
     Parsing datafusion-ffi v54.1.0 (baseline)
      Parsed [   0.066s] (baseline)
    Checking datafusion-ffi v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.322s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 122.012s] datafusion-ffi
    Building datafusion-physical-optimizer v54.1.0 (current)
       Built [  39.352s] (current)
     Parsing datafusion-physical-optimizer v54.1.0 (current)
      Parsed [   0.023s] (current)
    Building datafusion-physical-optimizer v54.1.0 (baseline)
       Built [  39.332s] (baseline)
     Parsing datafusion-physical-optimizer v54.1.0 (baseline)
      Parsed [   0.024s] (baseline)
    Checking datafusion-physical-optimizer v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.158s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  80.005s] datafusion-physical-optimizer
    Building datafusion-physical-plan v54.1.0 (current)
       Built [  36.921s] (current)
     Parsing datafusion-physical-plan v54.1.0 (current)
      Parsed [   0.152s] (current)
    Building datafusion-physical-plan v54.1.0 (baseline)
       Built [  36.716s] (baseline)
     Parsing datafusion-physical-plan v54.1.0 (baseline)
      Parsed [   0.150s] (baseline)
    Checking datafusion-physical-plan v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.947s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure trait_method_marked_deprecated: trait method #[deprecated] added ---

Description:
A trait method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/trait_method_marked_deprecated.ron

Failed in:
  method with_new_children in trait datafusion_physical_plan::execution_plan::ExecutionPlan in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/execution_plan.rs:98
  method with_new_children_and_same_properties in trait datafusion_physical_plan::execution_plan::ExecutionPlan in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/execution_plan.rs:98
  method with_new_children in trait datafusion_physical_plan::ExecutionPlan in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/execution_plan.rs:98
  method with_new_children_and_same_properties in trait datafusion_physical_plan::ExecutionPlan in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/execution_plan.rs:98

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [  76.122s] datafusion-physical-plan
    Building datafusion-proto v54.1.0 (current)
       Built [  59.952s] (current)
     Parsing datafusion-proto v54.1.0 (current)
      Parsed [   0.019s] (current)
    Building datafusion-proto v54.1.0 (baseline)
       Built [  59.900s] (baseline)
     Parsing datafusion-proto v54.1.0 (baseline)
      Parsed [   0.021s] (baseline)
    Checking datafusion-proto v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.361s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 121.429s] datafusion-proto