Skip to content

feat: add array_scale scalar function - #22466

Merged
Dandandan merged 1 commit into
apache:mainfrom
crm26:feat/array-scale
May 26, 2026
Merged

feat: add array_scale scalar function#22466
Dandandan merged 1 commit into
apache:mainfrom
crm26:feat/array-scale

Conversation

@crm26

@crm26 crm26 commented May 22, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Partial of #21536 — array_scale (the list+scalar arithmetic function in the vector math series).

Rationale for this change

Continues the per-function split requested by @alamb on #21536. Three sibling PRs already merged: cosine_distance (#21542), inner_product (#21861), array_normalize (#22013). array_add is in flight as #22459 by @SubhamSinghal.

Adds element-wise scalar multiplication for numeric arrays, returning a list of the same shape. Aliased as list_scale to match the array_X / list_X precedent in this crate.

What changes are included in this PR?

  • New scalar UDF array_scale(array, scalar) in datafusion/functions-nested/src/array_scale.rs
  • Module wire-up + registration in datafusion/functions-nested/src/lib.rs
  • SLT tests at datafusion/sqllogictest/test_files/array_scale.slt
  • Auto-generated function docs entry in docs/source/user-guide/sql/scalar_functions.md

Signature: first arg List/LargeList/FixedSizeList<numeric>, second arg numeric scalar. Both coerce to Float64. Same list-widening rules as the binary-op siblings.

NULL semantics:

  • NULL row in array → NULL row out
  • NULL scalar → NULL row out (whole-row, because the scalar applies uniformly)
  • NULL element at position `i` → NULL element at `i` out (per-element propagation)
  • Empty array → empty array

Builders: uses `OffsetBufferBuilder` + `NullBufferBuilder` per the pattern adopted in the round-1 review of #22013.

Are these changes tested?

Yes. `array_scale.slt` covers:

  • Happy paths (positive, negative, zero, fractional, single-element)
  • NULL propagation at all three levels (NULL row, NULL scalar, NULL element)
  • All list type variants (`List`, `LargeList`, `FixedSizeList`)
  • Numeric inner type coercion (Float32, Int64, integer literals)
  • Multi-row queries with both constant-scalar broadcast and per-row column scalar
  • Error paths (non-numeric scalar, non-list first arg, wrong arity)
  • Empty array
  • `list_scale` alias

Are there any user-facing changes?

Yes — new SQL scalar function `array_scale(array, scalar)` and its alias `list_scale`. Documented in `docs/source/user-guide/sql/scalar_functions.md`.

@github-actions github-actions Bot added documentation Improvements or additions to documentation sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels May 22, 2026
Jefffrey
Jefffrey reviewed May 23, 2026
View reviewed changes
Comment thread datafusion/functions-nested/src/array_scale.rs Outdated
Comment thread datafusion/functions-nested/src/array_scale.rs Outdated

let mut value_builder = Float64Array::builder(values.len());
let mut new_offsets = OffsetBufferBuilder::<O>::new(list_array.len());
let mut row_nulls = NullBufferBuilder::new(list_array.len());

@Jefffrey Jefffrey May 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't need to calculate per row nulls, can just use NullBuffer::union

let nulls = NullBuffer::union(list_array.nulls(), scalar_array.nulls());

@crm26 crm26 May 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Replaced the NullBufferBuilder with let row_nulls = NullBuffer::union(list_array.nulls(), scalar_array.nulls()) computed once before the loop, then passed directly into GenericListArray::try_new. The is_null check inside the loop is kept since it's still load-bearing for the offset buffer construction (we need to push zero-length offsets for null rows).

@Jefffrey Jefffrey May 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use the new row_nulls for the null check inside the loop now instead of still querying list + scalar separately

e.g.

row_nulls.is_some_and(|nb| nb.is_null(i))

@crm26 crm26 May 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Loop check now uses row_nulls.as_ref().is_some_and(|nb| nb.is_null(row)) instead of querying list_array and scalar_array separately. Same semantics (NullBuffer::union ORs the two null buffers), one source of truth.

Comment thread datafusion/functions-nested/src/array_scale.rs Outdated
@crm26
crm26 force-pushed the feat/array-scale branch 2 times, most recently from a29be46 to bf052a9 Compare May 23, 2026 16:05
@crm26
feat: add array_scale scalar function
172c35d 
Adds `array_scale(array, scalar)` returning a new array with each element
multiplied by a scalar. Aliased as `list_scale`. Part of the per-function
split sequence on tracking issue apache#21536, following the pattern of the
already-merged PRs in this series.

Semantics:
- NULL row in array -> NULL row out
- NULL element at position i in array -> NULL element at i out
  (per-element propagation)
- NULL scalar -> NULL row out (whole-row, because the scalar applies
  uniformly to every element; the entire operation is undefined)
- Empty array -> empty array

First argument is List/LargeList/FixedSizeList of any numeric type.
Second argument is a numeric scalar. Both coerce to Float64. List-like
inputs follow the same widening rules as the binary-op siblings:
LargeList wins, FixedSizeList coerces to List.
@crm26
crm26 force-pushed the feat/array-scale branch from bf052a9 to 172c35d Compare May 24, 2026 15:09
@Dandandan
Dandandan added this pull request to the merge queue May 26, 2026
Merged via the queue into apache:main with commit bdf8a6d May 26, 2026
36 checks passed
@crm26

crm26 commented May 26, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @Jefffrey for the review, @Dandandan for the merge.

crm26 added a commit to crm26/datafusion that referenced this pull request May 26, 2026
@crm26
feat: add array_sum scalar function
8fb1875 
Adds `array_sum(array)` returning the sum of elements in a numeric array.
Aliased as `list_sum`. Part of the per-function split sequence on
tracking issue apache#21536, following the pattern of the already-merged PRs
in this series (cosine_distance apache#21542, inner_product apache#21861,
array_normalize apache#22013, array_scale apache#22466).

Semantics:
- NULL row in array -> NULL row out
- NULL elements are skipped (SQL aggregate convention; matches
  PostgreSQL array_sum, DuckDB list_sum, Spark aggregate). A row whose
  every element is NULL yields NULL.
- Empty array -> 0.0 (additive identity, matches SQL SUM over no rows
  conceptually, and DuckDB list_sum([]) = 0)

Input is List/LargeList/FixedSizeList of any numeric type; elements
are coerced to Float64. Output is Float64.
crm26 added a commit to crm26/datafusion that referenced this pull request May 26, 2026
@crm26
feat: add array_sum scalar function
4f04bfa 
Adds `array_sum(array)` returning the sum of elements in a numeric array.
Aliased as `list_sum`. Part of the per-function split sequence on
tracking issue apache#21536, following the pattern of the already-merged PRs
in this series (cosine_distance apache#21542, inner_product apache#21861,
array_normalize apache#22013, array_scale apache#22466).

Semantics:
- NULL row in array -> NULL row out
- NULL elements are skipped (SQL aggregate convention; matches
  PostgreSQL array_sum, DuckDB list_sum, Spark aggregate). A row whose
  every element is NULL yields NULL.
- Empty array -> 0.0 (additive identity, matches SQL SUM over no rows
  conceptually, and DuckDB list_sum([]) = 0)

Input is List/LargeList/FixedSizeList of any numeric type; elements
are coerced to Float64. Output is Float64.
crm26 added a commit to crm26/datafusion that referenced this pull request May 26, 2026
@crm26
feat: add array_sum scalar function
c35f176 
Adds `array_sum(array)` returning the sum of elements in a numeric array.
Aliased as `list_sum`. Part of the per-function split sequence on
tracking issue apache#21536, following the pattern of the already-merged PRs
in this series (cosine_distance apache#21542, inner_product apache#21861,
array_normalize apache#22013, array_scale apache#22466).

Semantics:
- NULL row in array -> NULL row out
- NULL elements are skipped (SQL aggregate convention; matches
  PostgreSQL array_sum, DuckDB list_sum, Spark aggregate). A row whose
  every element is NULL yields NULL.
- Empty array -> 0.0 (additive identity, matches SQL SUM over no rows
  conceptually, and DuckDB list_sum([]) = 0)

Input is List/LargeList/FixedSizeList of any numeric type; elements
are coerced to Float64. Output is Float64.
crm26 added a commit to crm26/datafusion that referenced this pull request May 26, 2026
@crm26
feat: add array_sum scalar function
886ba0b 
Adds `array_sum(array)` returning the sum of elements in a numeric array.
Aliased as `list_sum`. Part of the per-function split sequence on
tracking issue apache#21536, following the pattern of the already-merged PRs
in this series (cosine_distance apache#21542, inner_product apache#21861,
array_normalize apache#22013, array_scale apache#22466).

Semantics:
- NULL row in array -> NULL row out
- NULL elements are skipped (SQL aggregate convention; matches
  PostgreSQL array_sum, DuckDB list_sum, Spark aggregate). A row whose
  every element is NULL yields NULL.
- Empty array -> 0.0 (additive identity, matches SQL SUM over no rows
  conceptually, and DuckDB list_sum([]) = 0)

Input is List/LargeList/FixedSizeList of any numeric type; elements
are coerced to Float64. Output is Float64.
@crm26 crm26 mentioned this pull request May 26, 2026
Merged
crm26 added a commit to crm26/datafusion that referenced this pull request Jun 8, 2026
@crm26
feat: add array_sum scalar function
d4049bf 
Adds `array_sum(array)` returning the sum of elements in a numeric array.
Aliased as `list_sum`. Part of the per-function split sequence on
tracking issue apache#21536, following the pattern of the already-merged PRs
in this series (cosine_distance apache#21542, inner_product apache#21861,
array_normalize apache#22013, array_scale apache#22466).

Semantics:
- NULL row in array -> NULL row out
- NULL elements are skipped (SQL aggregate convention; matches
  PostgreSQL array_sum, DuckDB list_sum, Spark aggregate). A row whose
  every element is NULL yields NULL.
- Empty array -> 0.0 (additive identity, matches SQL SUM over no rows
  conceptually, and DuckDB list_sum([]) = 0)

Input is List/LargeList/FixedSizeList of any numeric type; elements
are coerced to Float64. Output is Float64.
martin-augment pushed a commit to martin-augment/datafusion that referenced this pull request Jun 8, 2026
@crm26 @claude
feat: add array_sum scalar function (apache#22542)
1d740ed 
## Which issue does this PR close?

Partial of apache#21536 — `array_sum` (first of the array aggregates in the
series).

## Rationale for this change

Continues the per-function split sequence requested by @alamb on apache#21536.
Four sibling PRs already merged: `cosine_distance` (apache#21542),
`inner_product` (apache#21861), `array_normalize` (apache#22013), `array_scale`
(apache#22466). `array_add` is in flight as apache#22459 by @SubhamSinghal.

`array_sum` is the first of the three array-aggregate functions (sum,
product, avg). Its semantics set the pattern for the other two
aggregates.

## What changes are included in this PR?

- New scalar UDF `array_sum(array)` in
`datafusion/functions-nested/src/array_sum.rs`
- Module wire-up + registration in
`datafusion/functions-nested/src/lib.rs`
- SLT tests at `datafusion/sqllogictest/test_files/array_sum.slt`
- Auto-generated docs entry in
`docs/source/user-guide/sql/scalar_functions.md`

**Signature:** \`List/LargeList/FixedSizeList<numeric>\` in, \`Float64\`
out (one scalar per row). Numeric inner types coerced to \`Float64\`.

**NULL semantics — SQL aggregate convention (deliberate divergence from
binary-op siblings):**
- NULL row → NULL row out
- NULL elements are **skipped**, matching PostgreSQL \`array_sum\`,
DuckDB \`list_sum\`, Spark \`aggregate\`. Binary-op siblings
(\`inner_product\`, \`array_normalize\`) null-row on NULL element
because their per-element operation is undefined on NULL; aggregates
conventionally skip NULLs in SQL.
- All-NULL row → NULL out (matches \`SUM(...)\` over an all-NULL column)

- **Empty array → NULL** (matches sibling `array_product` apache#22703,
PostgreSQL, DuckDB `list_sum`, SQL Standard SUM-of-empty-set)

**Alias:** \`list_sum\` (matches the precedent of
\`array_normalize\`→\`list_normalize\`, \`array_scale\`→\`list_scale\`).

## Are these changes tested?

Yes. SLT covers happy paths, empty arrays, NULL row, NULL elements (mix
+ all-NULL), all list variants (List/LargeList/FixedSizeList), numeric
coercion (Float32/Int64/integer literals), multi-row composition, error
paths, return type, and the \`list_sum\` alias.

## Are there any user-facing changes?

Yes — new SQL scalar function \`array_sum(array)\` and its alias
\`list_sum\`.

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
AdamGS pushed a commit to AdamGS/arrow-datafusion that referenced this pull request Jun 11, 2026
@crm26 @claude @AdamGS
feat: add array_sum scalar function (apache#22542)
e42fea6 
## Which issue does this PR close?

Partial of apache#21536 — `array_sum` (first of the array aggregates in the
series).

## Rationale for this change

Continues the per-function split sequence requested by @alamb on apache#21536.
Four sibling PRs already merged: `cosine_distance` (apache#21542),
`inner_product` (apache#21861), `array_normalize` (apache#22013), `array_scale`
(apache#22466). `array_add` is in flight as apache#22459 by @SubhamSinghal.

`array_sum` is the first of the three array-aggregate functions (sum,
product, avg). Its semantics set the pattern for the other two
aggregates.

## What changes are included in this PR?

- New scalar UDF `array_sum(array)` in
`datafusion/functions-nested/src/array_sum.rs`
- Module wire-up + registration in
`datafusion/functions-nested/src/lib.rs`
- SLT tests at `datafusion/sqllogictest/test_files/array_sum.slt`
- Auto-generated docs entry in
`docs/source/user-guide/sql/scalar_functions.md`

**Signature:** \`List/LargeList/FixedSizeList<numeric>\` in, \`Float64\`
out (one scalar per row). Numeric inner types coerced to \`Float64\`.

**NULL semantics — SQL aggregate convention (deliberate divergence from
binary-op siblings):**
- NULL row → NULL row out
- NULL elements are **skipped**, matching PostgreSQL \`array_sum\`,
DuckDB \`list_sum\`, Spark \`aggregate\`. Binary-op siblings
(\`inner_product\`, \`array_normalize\`) null-row on NULL element
because their per-element operation is undefined on NULL; aggregates
conventionally skip NULLs in SQL.
- All-NULL row → NULL out (matches \`SUM(...)\` over an all-NULL column)

- **Empty array → NULL** (matches sibling `array_product` apache#22703,
PostgreSQL, DuckDB `list_sum`, SQL Standard SUM-of-empty-set)

**Alias:** \`list_sum\` (matches the precedent of
\`array_normalize\`→\`list_normalize\`, \`array_scale\`→\`list_scale\`).

## Are these changes tested?

Yes. SLT covers happy paths, empty arrays, NULL row, NULL elements (mix
+ all-NULL), all list variants (List/LargeList/FixedSizeList), numeric
coercion (Float32/Int64/integer literals), multi-row composition, error
paths, return type, and the \`list_sum\` alias.

## Are there any user-facing changes?

Yes — new SQL scalar function \`array_sum(array)\` and its alias
\`list_sum\`.

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Dandandan Dandandan Dandandan approved these changes
@Jefffrey Jefffrey Jefffrey approved these changes

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@crm26 @Dandandan @Jefffrey