IN LIST: optimize Utf8View and BinaryView filters

[geoffreyclaude]

Which issue does this PR close?

• Part of Further improve performance of IN list evaluation #19241.
• Stacked on #23015.
• Next in stack: #23018.
• Extracted from Optimize IN performance with specialized implementations #19390.

Rationale for this change

Arrow Utf8View and BinaryView arrays store each value in a 128-bit view. Values up to 12 bytes are fully inline. Longer values keep their length and four-byte prefix in the view and store the remaining bytes separately.

That representation allows cheap rejection without hashing or comparing complete strings. Inline values can be matched exactly from the view itself; long candidates are always confirmed against their full bytes before reporting a match.

What changes are included in this PR?

• Adds branchless matching for all-inline lists up to four values.
• Adds a keyed FrozenSet<u128> for larger inline lists.
• Adds a mixed byte-view filter that uses view keys for rejection and a native HashTable for exact long-value confirmation.
• Specializes only measured winning shapes and keeps all-long or inline-dominant mixed lists on the generic fallback.
• Requires exact byte-view physical types after recursively unwrapping dictionary expression types.
• Skips membership work for null needles while preserving IN and NOT IN semantics.
• Covers slices, dictionaries, Utf8View, BinaryView, nulls, long-prefix collisions, and custom hash collisions.

Are these changes tested?

Yes.

• cargo fmt --all
• cargo test -p datafusion-physical-expr --lib expressions::in_list
• cargo clippy --all-targets --all-features -- -D warnings
• ./dev/rust_lint.sh
• RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption

Are there any user-facing changes?

No. This is an internal performance optimization only.

Local benchmark snapshot

Built and run in separate target directories, using identical benchmark source and filtering to the relevant byte-view rows:

cargo bench --target-dir <target-dir> -p datafusion-physical-expr --bench in_list_strategy -- <filter>

Filters used: utf8view/short_8b, utf8view/len_12b, utf8view/mixed_len, and nulls/utf8view/short_8b.

Method: directly compared Criterion median point estimates. Filter construction remains outside the timed loop. Lower is better; changes within +/-5% are treated as noise.

Compared baselines: #23015 -> #23016

Relevant scope: all-inline Utf8View rows, mixed inline/long shapes, the inline-dominant fallback, and nullable inline views. BinaryView uses the same physical view representation and filter implementation.

Summary: 23 relevant rows, 20 faster, 0 slower, 3 within +/-5%.

The inline-dominant mixed rows remain within noise, supporting the conservative fallback for that shape.

Largest relevant deltas:

Benchmark	Before	After	Change
nulls/utf8view/short_8b/list=16/match=50%/nulls=50%	69.96 us	15.73 us	-77.5% (4.45x faster)
utf8view/short_8b/list=4/match=50%	55.03 us	12.65 us	-77.0% (4.35x faster)
nulls/utf8view/short_8b/list=16/match=50%/nulls=20%/NOT_IN	60.26 us	14.87 us	-75.3% (4.05x faster)
nulls/utf8view/short_8b/list=16/match=50%/nulls=20%	59.73 us	14.99 us	-74.9% (3.98x faster)
utf8view/short_8b/list=16/match=50%	55.43 us	14.74 us	-73.4% (3.76x faster)
utf8view/short_8b/list=64/match=50%	55.51 us	15.47 us	-72.1% (3.59x faster)
utf8view/len_12b/list=64/match=50%	54.22 us	15.51 us	-71.4% (3.50x faster)
utf8view/short_8b/list=256/match=50%	53.51 us	15.73 us	-70.6% (3.40x faster)
utf8view/len_12b/list=16/match=50%	54.70 us	16.37 us	-70.1% (3.34x faster)
utf8view/short_8b/list=4/match=0%	19.62 us	12.62 us	-35.7% (1.55x faster)

Full relevant table (23 rows)

Benchmark	Before	After	Change
utf8view/short_8b/list=4/match=0%	19.62 us	12.62 us	-35.7% (1.55x faster)
utf8view/short_8b/list=4/match=50%	55.03 us	12.65 us	-77.0% (4.35x faster)
utf8view/short_8b/list=16/match=0%	20.65 us	18.30 us	-11.4% (1.13x faster)
utf8view/short_8b/list=16/match=50%	55.43 us	14.74 us	-73.4% (3.76x faster)
utf8view/short_8b/list=64/match=0%	21.33 us	18.47 us	-13.4% (1.15x faster)
utf8view/short_8b/list=64/match=50%	55.51 us	15.47 us	-72.1% (3.59x faster)
utf8view/short_8b/list=256/match=0%	21.18 us	17.85 us	-15.7% (1.19x faster)
utf8view/short_8b/list=256/match=50%	53.51 us	15.73 us	-70.6% (3.40x faster)
utf8view/len_12b/list=16/match=0%	19.68 us	17.93 us	-8.9% (1.10x faster)
utf8view/len_12b/list=16/match=50%	54.70 us	16.37 us	-70.1% (3.34x faster)
utf8view/len_12b/list=64/match=0%	20.03 us	18.45 us	-7.9% (1.09x faster)
utf8view/len_12b/list=64/match=50%	54.22 us	15.51 us	-71.4% (3.50x faster)
utf8view/mixed_len/list=16/match=0%	40.76 us	42.26 us	+3.7% (within noise)
utf8view/mixed_len/list=16/match=50%	125.41 us	87.15 us	-30.5% (1.44x faster)
utf8view/mixed_len/list=64/match=0%	40.96 us	38.40 us	-6.3% (1.07x faster)
utf8view/mixed_len/list=64/match=50%	134.17 us	90.56 us	-32.5% (1.48x faster)
utf8view/mixed_len/inline=2%/list=64/match=0%	49.36 us	38.07 us	-22.9% (1.30x faster)
utf8view/mixed_len/inline=2%/list=64/match=50%	129.82 us	110.82 us	-14.6% (1.17x faster)
utf8view/mixed_len/inline=98%/list=64/match=0%	24.38 us	24.39 us	+0.1% (within noise)
utf8view/mixed_len/inline=98%/list=64/match=50%	63.29 us	61.24 us	-3.2% (within noise)
nulls/utf8view/short_8b/list=16/match=50%/nulls=20%	59.73 us	14.99 us	-74.9% (3.98x faster)
nulls/utf8view/short_8b/list=16/match=50%/nulls=20%/NOT_IN	60.26 us	14.87 us	-75.3% (4.05x faster)
nulls/utf8view/short_8b/list=16/match=50%/nulls=50%	69.96 us	15.73 us	-77.5% (4.45x faster)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 84.58844% with 176 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@7ff7278). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...l-expr/src/expressions/in_list/byte_view_filter.rs	84.19%	32 Missing and 41 partials ⚠️
...l-expr/src/expressions/in_list/primitive_filter.rs	83.48%	14 Missing and 39 partials ⚠️
.../physical-expr/src/expressions/in_list/strategy.rs	76.75%	31 Missing and 12 partials ⚠️
...atafusion/physical-expr/src/expressions/in_list.rs	69.23%	1 Missing and 3 partials ⚠️
...hysical-expr/src/expressions/in_list/frozen_set.rs	97.84%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #23016   +/-   ##
=======================================
  Coverage        ?   80.70%           
=======================================
  Files           ?     1091           
  Lines           ?   369879           
  Branches        ?   369879           
=======================================
  Hits            ?   298528           
  Misses          ?    53523           
  Partials        ?    17828           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.