feat(arrow): export StringArray and BinaryArray directly

[ClSlaid]

Summary

This PR adds direct export to Arrow StringArray and BinaryArray.

A caller can request one of these Arrow types:

• Utf8
• LargeUtf8
• Binary
• LargeBinary

Vortex builds a VarBinArray for the requested Arrow type. Arrow uses the buffers in the VarBinArray.

The Python API accepts the requested Arrow type:

array.to_arrow_array(arrow_type=pa.string())
vxf.to_arrow(schema=pa.schema([...]))

Vortex still exports StringViewArray and BinaryViewArray when the caller does not request a type or schema.

Direct export paths

These encodings can write directly to the VarBinArray builder:

• FSST
• OnPair
• Zstd
• Dict

The Dict path also supports compressed dictionary values. Examples include Dict<FSST> and Dict<Zstd>.

Sparse and RunEnd use the normal canonical path. The Rust benchmark showed a 55% gain for the Sparse canonical path. The Vortex compressor does not create RunEnd string arrays.

Reason for the change

The old path created a StringViewArray or BinaryViewArray. Arrow then cast the result to a StringArray or BinaryArray.

The cast copied or reordered the value bytes.

The new path creates a StringArray or BinaryArray directly.

Performance

This Python benchmark used 500,000 repeated URL strings. Each result is the median of 10 runs.

Export path	Median
Direct export to StringArray	111.232 ms
Export to StringViewArray, then cast to StringArray	273.663 ms

Direct export was 59.4% faster in this benchmark.

The Rust benchmark covers these layouts and encodings:

• View
• FSST
• Sparse
• Zstd
• Dict
• Dict<FSST>
• Dict<Zstd>
• filtered compressed arrays
• Chunked<FSST>

The Filter cases do not use the direct encoding paths. Chunked<FSST> uses the direct FSST path. Chunked passes the output builder to each child.

AI assistance

I used GPT5.6sol to help develop this change. I reviewed and tested all changes.

Verification

• cargo nextest run -p vortex-arrow -p vortex-fsst -p vortex-onpair -p vortex-runend -p vortex-sparse -p vortex-zstd passed 529 tests.
• cargo nextest run -p vortex-array --no-fail-fast passed 3,003 tests and skipped 1 test.
• cargo clippy -p vortex-arrow -p vortex-sparse -p vortex-runend --all-targets --all-features -- -D warnings passed.
• cargo +nightly fmt --all passed.
• git diff --check passed.
• The targeted Python tests passed 3 tests.
• The Sphinx doctest run passed 271 tests.

Coverage

Rust coverage used LLVM branch instrumentation for vortex-array, vortex-arrow, vortex-fsst, vortex-onpair, vortex-runend, vortex-sparse, and vortex-zstd.

cargo +nightly llvm-cov nextest --branch --all-features \
  -p vortex-array -p vortex-arrow -p vortex-fsst -p vortex-onpair \
  -p vortex-runend -p vortex-sparse -p vortex-zstd \
  --no-fail-fast --json --output-path coverage.json

The original targeted run passed 3,550 tests and skipped 2 tests. After adding focused tests, this command refreshed coverage for the changed vortex-array paths:

cargo +nightly llvm-cov nextest --branch -p vortex-array --no-fail-fast --lcov

The refreshed run passed 3,003 tests and skipped 1 test.

• Changed executable Rust lines: 495 of 547 lines, or 90.49%.
• Changed Rust branch outcomes: 19 of 26 outcomes, or 73.08%.

The changed-line results compare this branch with 57962f4df9af99ebad584bf3e59310e4bf552fc1. They intersect added lines with the LLVM LCOV line and branch records. The report combines the refreshed vortex-array LCOV with the existing LCOV for the unchanged encoding and Arrow crates. This Rust scope does not include benchmarks or the PyO3 crate.

Python branch coverage used the changed array and file test modules.

uv run --all-packages --with coverage coverage run --branch -m pytest \
  vortex-python/test/test_array.py vortex-python/test/test_file.py

The run completed with 19 passed, 1 pre-existing xfail (test_scalar_at_out_of_bounds), and 0 failed.

• vortex/file.py: 32 of 47 statements, or 68.09%.
• vortex/file.py: 0 of 2 branches. Both branches are in unchanged code.
• Changed executable Python statements: 1 of 1, or 100%.
• Changed Python branches: none.

Python coverage does not instrument the Rust code in the extension. The Python tests exercise that code through the public API.

Next steps

Pass the output builder through lazy arrays such as Filter, Slice, and Take. This change can remove more intermediate canonical arrays. Track this work in #7674.

[codspeed-hq]

Merging this PR will improve performance by 12.8%

⚡ 1 improved benchmark
✅ 1848 untouched benchmarks
⏩ 46 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	fsl_large	99.2 µs	88 µs	+12.8%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ClSlaid:feat/direct-arrow-byte-arrays (95074ff) with develop (a12c310)}

Footnotes

1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[ClSlaid]

The regression is strange, nowhere can the code be touched.

GPT blames codegen unit difference, it insists code in this patch disturbs LLVM's optimization and thus produced benchmark with different layouts.

I've benchmarked the patch against baseline under 1 CGU. Their results are close, and regression turns out to be solved.

 ┌────────┬───────────┬───────────┬────────┐                               
 │ Group  │ develop   │ #8902     │ Diff   │                                                
 ├────────┼───────────┼───────────┼────────┤                                                   
 │ 1      │ 60.10 µs  │ 60.24 µs  │ +0.23% │                                      
 ├────────┼───────────┼───────────┼────────┤                                     
 │ 2      │ 60.01 µs  │ 60.15 µs  │ +0.23% │                                      
 ├────────┼───────────┼───────────┼────────┤                                 
 │ 3      │ 60.01 µs  │ 60.01 µs  │ 0.00%  │                                   
 ├────────┼───────────┼───────────┼────────┤                                
 │ 4      │ 60.40 µs  │ 60.47 µs  │ +0.12% │                             
 ├────────┼───────────┼───────────┼────────┤                     
 │ 5      │ 60.23 µs  │ 59.98 µs  │ -0.42% │                           
 ├────────┼───────────┼───────────┼────────┤                         
 │ 6      │ 60.11 µs  │ 60.27 µs  │ +0.27% │                     
 ├────────┼───────────┼───────────┼────────┤                      
 │ MID    │ 60.105 µs │ 60.195 µs │ +0.15% │                      
 └────────┴───────────┴───────────┴────────┘ 

It is interesting, but I will keep this patch free of codegen changes.

[robert3005]

Some of the microbenchmarks are sensitive to code changes. I don't think it's related to your change. Thanks for your contribution. I will review it tomorrow!

[ClSlaid]

@robert3005 please rerun benchmark.

[robert3005]

Reviewed at 02637a4. Nice work — the composition is correct, and I tried fairly hard to break it:

• Zstd: every slice range (0..=12 × 0..=12) × 4 frame sizes, with nulls, differentially against the canonical path — all match. The value_idx_start - n_skipped_values math is equivalent to the views path.
• FSST: every slice range, plus three slices of one FSST array composed as chunks into a single builder — all match.
• Dict: empty dict, all-null codes, empty values + all-null codes, nullable values, dict-of-dict, constant values, null codes with out-of-range payloads — all match the old StringView→cast path.
• Sliced VarBin, sliced Dict, chunked-with-mixed-encodings — all match.
• Suites green: vortex-array 3143 + 72; vortex-arrow/fsst/onpair/runend/sparse/zstd 529. 0 failures.

The 4157bd9 unification is a real improvement, and I checked all four migrated call sites — FSST/OnPair's .scan() cumulative sum, append_varbin's first_offset normalization, and filter.rs's Mask::new_true(end - start) all preserve prior semantics. The Zstd change is a pure code move. It also fixes a latent bug it doesn't mention: the old append_values did its offset arithmetic in O's type, so VarBinBuilder<i32> could silently wrap in release; doing checked_add in usize first means that's now detected.

One blocking item: the offset-overflow panic (inline on builder.rs). Requesting pa.string() on >2 GiB of strings now aborts where the old StringView→cast path returned a clean Err.

One I'd push on: the new arrow_type= / schema= parameters route an unvalidated target type into a bare assert! — vx.array([1,2,3]).to_arrow_array(arrow_type=pa.binary()) panics. The assert is pre-existing (it fires on develop too for all 12 dtype×target combinations I tried), but develop gave Python callers no way to pick the target type, so this PR is what makes it reachable.

The other three inline comments are non-blocking; the finish/new_unchecked one is pre-existing and this PR narrows it rather than widening it, so it's fine as a follow-up.

Minor note not worth its own thread: append_varbin now indexes offsets[0] directly, which panics on a zero-length offsets array where windows(2) yielded empty. Builder output always has ≥1 offset and VarBinData::validate requires non-empty, so this is defensive-only — just a strictly narrower input domain than before.

I did not run the Python tests, docs doctests, clippy, or fmt; your reported results cover those and none of my findings depend on them. The >2 GiB overflow path I reasoned from code — materializing 2 GiB wasn't practical.

---

Generated by Claude Code

[robert3005]

Offset overflow should be a VortexError, not a panic.

Both this checked_add and the O::from(end) conversion below panic. For the Utf8/Binary targets (i32 offsets), that means >2 GiB of string data in one chunk aborts instead of erroring — and >2 GiB is exactly what a caller hits asking for pa.string() on a large scan.

The old path got this right for free: it built a StringViewArray and let arrow_cast::cast handle it, which returns Err(ArrowError::ComputeError) on offset overflow. So this is a behavioral regression on the failure path. In a DataFusion/DuckDB worker it's a panic rather than a surfaced error, and through PyO3 it becomes a PanicException.

Suggest making append_values return VortexResult<()> and using vortex_ensure!. Blast radius is small — filter.rs plus the four new call sites, all of which already sit in VortexResult functions.

Separately, credit where due: computing this in usize with checked_add before O::from is a real improvement over the old offset + self.data.len().as_(), which did the arithmetic in O's type and so could silently wrap for VarBinBuilder<i32> in release. That latent corruption in the filter.rs path is now detected — the only remaining gap is that detection is a panic rather than a recoverable error.

---

Generated by Claude Code

[robert3005]

Correcting the last paragraph of this comment, and the equivalent line in my review summary — I overstated it.

I claimed the old offset + self.data.len().as_() was a latent bug that this change fixes. The mechanism is real (arithmetic in O's type, so VarBinBuilder<i32> wraps negative in release rather than panicking as it would in debug), but it wasn't reachable. append_values had exactly one caller before this PR — filter.rs:157 — and that caller is size-constrained:

• builder: &mut VarBinBuilder<O> takes O from match_each_integer_ptype!(offsets.ptype(), ..), i.e. O is the input array's offset type;
• filter selects a subset, so cumulative output bytes ≤ input total bytes;
• the input's total bytes ≤ O::MAX by construction, since its own final offset is an O.

So data.len() + relative_end could never exceed O::MAX there. Defensive hardening, not a fix — apologies for the inflation.

What I should have said is that the guard matters because of the call sites this PR adds, which are not subset-constrained: to_arrow_byte_array picks i32 whenever the target is pa.string() regardless of source size, and the Dict path deliberately builds i64 values then narrows into a possibly-i32 target. That's exactly where the overflow becomes reachable, so doing the arithmetic in usize first is the right call — it just needs to surface as a VortexError rather than a panic, which is the substance of the comment above and unaffected by this correction.

---

Generated by Claude Code

[robert3005]

The new arrow_type= / schema= parameters can reach a bare assert!.

execute_arrow_naive dispatches purely on the target Arrow DataType (executor/mod.rs:151-154) and neither it nor execute_arrow checks the target against the array's dtype. So a non-byte source falls past the source_is_utf8 != target_is_utf8 guard on line 41 and lands in DynVarBinBuilder::with_capacity, whose first line is assert!(matches!(dtype, DType::Utf8(_) | DType::Binary(_))).

vx.array([1, 2, 3]).to_arrow_array(arrow_type=pa.binary())  # panics

I checked this against develop first — the panic is pre-existing internally, and all 12 dtype×target combinations I tried panic there too. But on develop a Python caller had no way to select the target type, so this PR is what makes it reachable. Worth a vortex_bail! here when array.dtype() isn't Utf8/Binary, or validation at the Python boundary.

Unrelated nit while you're in with_capacity: that assert! has no message, whereas the equivalent in VarBinViewBuilder::new reads "VarBinViewBuilder DType must be Utf8 or Binary." Worth matching.

---

Generated by Claude Code

[robert3005]

finish builds via new_unchecked with no length check — worth adding one while you're here.

VarBinArray::new_unchecked derives len = offsets.len().saturating_sub(1), so if offsets and validity desync you get an array whose len() disagrees with its validity mask, silently in release.

This is pre-existing, and the PR improves it — the old append_values had no checks at all, so a mismatched num desynced in every build profile; these two debug_assert_eq! at least catch it in debug. But VarBinViewBuilder::finish_into_varbinview already asserts this unconditionally (builders/varbinview.rs:239-243, "View and validity length must match"), and VarBinBuilder::finish doesn't. Since this PR adds a second public append path into the same builder, the asymmetry is more likely to bite now.

Cheapest fix that matches the existing convention — in finish, before new_unchecked:

assert_eq!(
    self.offsets.len() - 1,
    self.validity.len(),
    "Offset and validity length must match"
);

offsets is never empty (with_capacity pushes a leading 0), so len() - 1 is safe. That covers the pre-existing append_values path too, which a check inside this method wouldn't.

Non-blocking — happy for this to be a follow-up if you'd rather keep the PR tight.

---

Generated by Claude Code

[robert3005]

This branch preempts the existing dict fast path and drops its guards.

It runs whenever the builder is a DynVarBinBuilder, ahead of the take_canonical path at line 240 — and skips both of that path's short-circuits (!array.is_empty(), !codes.validity()?.definitely_all_null()). execute has the same two guards at lines 190 and 199, so this is the one place in the vtable without them.

It also does three full copies of the expanded data: decode all values into an i64 VarBin, take to full logical length (fresh offsets + bytes), then append_varbin into the target. For a dict that's the case the encoding exists to make cheap.

Separately, <VarBin as TakeExecute>::take is called directly, bypassing precondition(), then assumed to be Some and specifically VarBin. Both hold today — I traced every return path and they all build a VarBinArray — but the signature is Option<ArrayRef> and the doc says "The indices are guaranteed to be non-empty," which this call site violates for empty dicts. It'll break quietly if VarBin's take ever gains a constant-folding or view-returning fast path.

I tested the empty and all-null-codes cases and they happen to produce correct results, so this is brittleness rather than a live bug — but reusing the existing path (or at least its guards) seems preferable to duplicating it.

---

Generated by Claude Code

[robert3005]

Per-call overhead in the fallback path.

This is the path every encoding without a direct branch takes — Sparse, RunEnd, and anything on the VTable::append_to_builder default, which canonicalizes to VarBinView and lands here. The loop itself is irreducible (view→offset conversion has to gather scattered bytes), but three things inside it are avoidable:

1. values.bit_buffer().iter().enumerate() walks validity a bit at a time. BitBuffer::for_each_set_index (vortex-buffer/src/bit/buf.rs:450) plus null gap-filling is the same shape your Zstd AllOr::Some(valid_indices) arm already uses — per vortex-mask/benches/mask_iteration.rs it's 2-45x on the iteration itself.
2. bytes_at(index) returns an owned ByteBuffer per row when append_value only needs &[u8]. The inlined branch is the expensive one (views_handle().as_host().clone().into_byte_buffer().slice_ref(..)) and inlined means ≤12 bytes — the common short-string case. Hoisting views() and the buffer list and slicing directly avoids the per-row refcount and slice construction.
3. The byte total is computable up front from the view lengths in one pass over the fixed-width views, so self.data could be sized exactly instead of reallocating as it grows.

Your benchmark reports Sparse gaining 55%; I'd expect a chunk of the remainder to be here. Non-blocking.

---

Generated by Claude Code

[robert3005]

Thanks for the contribution. There's couple of follow ups that I have found while reviewing this code but they're independent of your changes.