[Parquet] ALP encoder/decoder support by sdf-jkl · Pull Request #9372 · apache/arrow-rs

sdf-jkl · 2026-02-07T00:01:58Z

Which issue does this PR close?

Closes [Parquet] Prototype ALP encoding #8748 .
Related spec PR: GH-533: Add ALP (Adaptive Lossless floating-Point) encoding specification parquet-format#557

Rationale for this change

check issue

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

alamb · 2026-02-08T13:32:29Z

amaaazing

sdf-jkl · 2026-02-20T23:16:14Z

@alamb I worked with codex on replicating the c++ implementation reviewing commit by commit.

I'll do one final read myself, but this should be ready for initial review.

alamb · 2026-02-26T20:29:50Z

Thank you -- I will put this on my short list to reivew I was out last week

devanbenz

A few comments

sdf-jkl · 2026-03-10T20:41:27Z

@devanbenz Sorry for taking so long to address your comments. I'll make sure to work on it this week!

alamb · 2026-03-11T18:03:12Z

The final spec is

GH-533: Add ALP (Adaptive Lossless floating-Point) encoding specification parquet-format#557

devanbenz

Comment about header regarding publication of spec.

sdf-jkl · 2026-03-12T20:37:12Z

Thanks @devanbenz, I've finally addressed your reviews!

devanbenz · 2026-03-12T22:54:40Z

Thanks @devanbenz, I finally addressed your reviews!

Please cargo fmt the code 🫡

sdf-jkl · 2026-03-12T23:12:46Z

~~I swear I did 😿~~
Had to run

cargo fmt -p parquet --  --config skip_children=true `find ./parquet -name "*.rs" \! -name format.rs`

5e5c8d2

…` marker

…h buffer

sdf-jkl · 2026-06-08T16:07:15Z

@alamb @devanbenz 🤓

alamb · 2026-06-11T14:47:05Z

Thanks @sdf-jkl -- I will try and find tiume to review this (probably not until the weekend though as I have to sort out the next object store release before

StreamingPage<F> encodes pages 2..N vector-by-vector as values arrive via put(), holding only a <1024 carry plus the compressed body instead of buffering the whole page. Page 1 still buffers to build the preset. Output is provably byte-identical to the buffered encoder (pinned by test_streaming_matches_buffered / test_streaming_irregular_puts). Adds an #[ignore]d rdtsc throughput bench (mod throughput) reporting cyc/val for encode buffered/streaming and decode. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

sdf-jkl · 2026-07-31T21:41:40Z

@alamb @devanbenz PTAL

( ~1400 locs are tests, so it's less scary than it looks like 😃 )

We might want to push apache/parquet-testing#105 first ...

alamb

I spent a bunch of a plane ride today reviewing this PR. I found it really easy to read, understand and follow. Very Nice work @sdf-jkl and @devanbenz

Only thing required: a few more tests

The only thing I think we need is some more testing, focused on end-to-end and round trips

Tests to read the other ALP files that are in apache/parquet-testing#105
Add some round trip tests for the ALP csv files (so we are testing encode path across more values)

Also, before we merge we should run llvm-cov and ensure nothing critical is uncovered.

All in all I can't emphasize enough how nice it was to read.

I will work on getting apache/parquet-testing#105 ready to merge, as you suggest

Performance

I am sure we can improve the performance in subsequent iterations, but I think we should focus on getting this implementation in first, and then collect follow on performance improvements in other tickets

High level comments:

The plumbing looks good, though I wonder if we really need the new GetEncoder trait

Ideas for follow on work (perhaps I can file a follow on epic)

Add a benchmark (perhaps bsed on the paruqet-testing dataset...) as well (as a separate PR) -- QA for performance
Support multiple vector lengths (now the encoder only supports 1024 element vectors
QA for encoding effectiveness: To test the effectiveness of the sampling / compression heuristics, we could brute force try different exponent / factor combinations for different vectors and ensure the exponent / factor chosen by the algorithm is the best (empirically verify)

alamb · 2026-08-01T20:35:06Z

+    0.000000000000000001,
+];
+
+pub(crate) trait AlpFloat:


I think we should add some comments on the AlpFloat trait explaining that it represents the floating point representation and the there is a corresponding AlpExact for each AlpFloat and cross link it. Otherwise I was confused at first trying to understand what the difference was

alamb · 2026-08-01T20:35:45Z

+    /// for negative `x` too. Values large enough to be mis-rounded just fail
+    /// the caller's round-trip check and become exceptions. The add/sub must
+    /// not be simplified away: it *is* the rounding.
+    fn fast_round(self) -> <Self::Exact as AlpExact>::Signed;


I think we could probbaly provide some more rationale about using fast_round before getting into the binade details. For example, perhaps we can explains that it is a way to round quickly using floating point math in a way that maximizes chance that multplying / dividing by factor and exponent will result in the same value (and thus avoiding exceptions)

alamb · 2026-08-01T20:36:26Z

+    );
+    assert_eq!(actual.num_rows(), expected.num_rows(), "row count mismatch");
+
+    for col_idx in 0..actual.num_columns() {


This is a lot of code to compare record batches and it will be much slower than the normal kernels.
I recommend using normal record batch comparison here rather than explicit row-by-row (aka assert_eq!(expected == actual)

alamb · 2026-08-01T20:36:40Z

+#[test]
+fn test_read_f32_alp() {
+    let data_dir = PathBuf::from(parquet_test_data());
+    let parquet_path = data_dir.join("alp_float_arade.parquet");


Also, I think we should test reading all the example ALP files (not just alp_float_arade.parquet)

alamb · 2026-08-01T20:37:18Z

+/// larger than PLAIN here, a property of the data rather than the encoder.
+/// ALP's win on arade in the paper is on the `f64` version of the dataset.
+#[test]
+fn test_write_f32_alp_roundtrip() {


As this is writing data, I think it would make make sense to move it to the arrow_writer tests either in parquet/tests/arrow_writer.rs or in parquet/src/arrow/arrow_writer/mod.rs. parquet/src/arrow/arrow_writer/mod.rs is probably the most consistent but it is already a massive module 🤔

In general I like the test pattern of round tripping the data used for alp encoding (aka read the csv and then round trip it through parquet with ALP enabled) to increase coverage. We could both:

Verify correctness

Compare compression (data page size) and that it was comparable to the checked in results

alamb · 2026-08-01T20:37:33Z

+    let actual = read_parquet_bytes(alp_bytes);
+    assert_eq!(actual.num_rows(), expected.num_rows(), "row count mismatch");
+
+    for col_idx in 0..expected.num_columns() {


this comparison logic I think can be avoided if we just compare the record batches

alamb · 2026-08-01T20:37:54Z

+/// unencoded value counts differ from the level count in two different ways at
+/// once.
+#[test]
+fn test_alp_roundtrip_page_versions_with_nulls() {


Likewise this appears to a roudnteip writer test, so it probably makes sense to move it with the other round trip writer tests

alamb · 2026-08-01T20:38:37Z

+    /// floating-point columns.
+    ///
+    /// [`ParquetValueType`]: crate::data_type::private::ParquetValueType
+    pub trait GetEncoder {


Why do we need a new GetEncoder trait? I think the ColumnDescPtr physical_type.logical_type

For example the fallback encoder depends on type
https://github.com/apache/arrow-rs/blob/8042ea288e084107b602f9e25a850314942567b6/parquet/src/column/writer/mod.rs#L1735-L1734

Alternately, maybe we could pass a &DataType as a parameter into the system

alamb · 2026-08-01T20:39:19Z

+    // positional overwrite, so it is independent of exception ordering.
+    let lo = cur.delivered;
+    let hi = cur.delivered + out.len();
+    for (pos_chunk, value_chunk) in cur


In this code, would it make sense to check / assert that the chunks_exact has no remainder (aka debug assert that remainder is null?)?

sdf-jkl added 2 commits February 6, 2026 18:48

Add alp to the thrift definitions

54bc343

Add alp to the encoding enum

622841b

github-actions Bot added the parquet Changes to the parquet crate label Feb 7, 2026

sdf-jkl and others added 10 commits February 12, 2026 16:43

Add alp header and page layout structs

bf754b2

Add Interleaved Vectors support

8337fcd

Merge branch 'main' into alp

5349a9e

Add bit unpacking, reverse FOR and exceptions patching

148b1d5

Decode page works

5f0222e

Add more checks and tests

027172f

Make decode page lazy + some tests

8ee1b7e

Find vectors using offsets at decode time, not parse time

18f885e

get decode fast path

a9fa0c3

Simplify decode_page_values

7754ebc

devanbenz suggested changes Feb 27, 2026

View reviewed changes

Comment thread parquet/src/encodings/decoding/alp.rs Outdated

Comment thread parquet/src/encodings/decoding/alp.rs Outdated

devanbenz reviewed Feb 27, 2026

View reviewed changes

Comment thread parquet/src/encodings/decoding/alp.rs Outdated

devanbenz suggested changes Mar 11, 2026

View reviewed changes

Comment thread parquet/src/encodings/decoding/alp.rs Outdated

sdf-jkl added 6 commits March 12, 2026 14:09

remove version from page header

6b7e923

change log_vetctor_size range consts

a13f244

redo the integration test using a test file from the c++ pr

6e86d6f

cache parsed vectors

ac56cc2

use Bytes for packed_values

07ca6f0

clippy too many args

def5c09

sdf-jkl added 8 commits June 7, 2026 11:22

store offsets instead of vecs in AlpEncodedVectorView

10bbc6b

pass output slice instead of allocating a vec

9978b31

fuse inverse FOR and decimal decode in one pass + remove `PhantonData…

70e35a8

…` marker

Merge branch 'main' of https://github.com/apache/arrow-rs into alp

e970919

lazy vector decode

81c14cb

even lazier decode

93eb925

use BitReader::get_batch to support autovectorization with a scratc…

ad84bde

…h buffer

update parquet-testing dep

23fac76

sdf-jkl marked this pull request as ready for review June 9, 2026 16:43

This was referenced Jun 17, 2026

[Parquet] Prototype FSST encoding #8749

Open

feat: [WIP] Adds FSST encoding (not ready for review) #10153

Draft

sdf-jkl and others added 14 commits July 14, 2026 10:33

avoid division

b0199e0

v1 header doesn't know num_value upfront

532b025

add encoder support

3b7d362

Merge branch 'main' of https://github.com/apache/arrow-rs into alp

3306fd2

skip loading each vector during skip

dc193a6

use copysign for fast_round

0dd0324

branchless exceptions encode

e5d9bf9

comment trim

b58a7e5

drop the sign in fast rounding

526b8af

drop the bench

2196346

Merge branch 'main' of https://github.com/apache/arrow-rs into alp

799d58c

fmt

e23b476

comments clean up pass

2c25e55

sdf-jkl changed the title ~~[WIp] ALP encoder/decoder support~~ [Parquet] ALP encoder/decoder support Jul 31, 2026

alamb reviewed Aug 1, 2026

View reviewed changes

Uh oh!

Conversation

sdf-jkl commented Feb 7, 2026 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Feb 8, 2026

Uh oh!

sdf-jkl commented Feb 20, 2026

Uh oh!

alamb commented Feb 26, 2026

Uh oh!

devanbenz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sdf-jkl commented Mar 10, 2026

Uh oh!

alamb commented Mar 11, 2026

Uh oh!

devanbenz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sdf-jkl commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devanbenz commented Mar 12, 2026

Uh oh!

sdf-jkl commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sdf-jkl commented Jun 8, 2026

Uh oh!

alamb commented Jun 11, 2026

Uh oh!

sdf-jkl commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Only thing required: a few more tests

Performance

High level comments:

Ideas for follow on work (perhaps I can file a follow on epic)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sdf-jkl commented Feb 7, 2026 •

edited by alamb

Loading

sdf-jkl commented Mar 12, 2026 •

edited

Loading

sdf-jkl commented Mar 12, 2026 •

edited

Loading

sdf-jkl commented Jul 31, 2026 •

edited

Loading