perf: vectorize float/decimal to narrow-integer casts#4941
Open
andygrove wants to merge 4 commits into
Open
Conversation
The four macros for float-to-int and decimal-to-int narrowing casts (cast_float_to_int16_down, cast_float_to_int32_up, cast_decimal_to_int16_down, cast_decimal_to_int32_up) built the output with a per-element iterator-collect over Option/Result. Replace that with Arrow's unary (legacy) and try_unary (ANSI) kernels, which map the values buffer in one pass and carry the null buffer over, following the same pattern used by cast_int_to_int_macro. The decimal macros also hoist the constant scale divisor out of the per-element loop. Overflow, NaN, saturation, and wrap-through-Int semantics are preserved unchanged; only the iteration mechanism changes. Add Rust unit tests for the float-to-Byte and decimal-to-Int/Byte legacy wrap paths and the ANSI overflow error paths (previously covered only by Scala tests), plus a benchmark. Non-overflow casts are 49-91% faster with no regression. Part of apache#4936.
15 tasks
mbutrovich
reviewed
Jul 15, 2026
mbutrovich
left a comment
mbutrovich
left a comment
Contributor
There was a problem hiding this comment.
First pass, thanks @andygrove!
…ulls invariant Extends test_cast_float64_to_int8_legacy_wraps with 3e9, +inf, and -inf so the saturate-i32-then-truncate-to-i8 chain is asserted end to end; a regression that lost the double-narrowing would fail loudly. Adds a one-line comment on each of the four legacy PrimitiveArray::unary arms noting that the closure runs on null slots too and stating why the as-cast / positive-divisor division is infallible for any bit pattern.
mbutrovich
approved these changes
Jul 22, 2026
mbutrovich
left a comment
mbutrovich
left a comment
Contributor
There was a problem hiding this comment.
This one LGTM after revision, thanks @andygrove!
Member
Author
|
@copilot resolve the merge conflicts on this branch. |
…ng-casts # Conflicts: # docs/source/contributor-guide/expression-audits/conversion_funcs.md # native/spark-expr/Cargo.toml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #4936.
Rationale for this change
The float-to-int and decimal-to-int narrowing casts built their output with a per-element
iter().map(...).collect::<Result<...>>()overOption/Result, the same slow pattern thatspark_cast_int_to_intmoved off of (that change was up to 100x faster).What changes are included in this PR?
Rewrites the four macros (
cast_float_to_int16_down,cast_float_to_int32_up,cast_decimal_to_int16_down,cast_decimal_to_int32_up) to use Arrow'sunary(legacy/try) andtry_unary(ANSI) kernels, which map the values buffer in one pass and carry the null buffer over, following the existingcast_int_to_int_macro. Each macro gains a destinationArrowPrimitiveTypeparameter, threaded through the 12 call sites. The decimal macros also hoist the constant10^scaledivisor out of the per-element loop.Overflow, NaN, saturation, and the cast-through-Int wrap semantics are preserved exactly; only the iteration mechanism changes.
How are these changes tested?
Added Rust unit tests (these paths previously had only Scala coverage): float64-to-Byte legacy wrap, float64-to-Int ANSI ok + overflow error, decimal-to-Int legacy, decimal-to-Byte legacy wrap, and decimal-to-Int ANSI overflow error. Output is bit-identical to
main.Benchmark (criterion), baseline
mainvs this branch, 8192-row columns: