Introduces a new LogicalType: FILE by brkyvz · Pull Request #585 · apache/parquet-format

brkyvz · 2026-06-09T22:04:45Z

Rationale for this change

Introduces a new type called File as a typed FileReference. The design document is here.

The motivation is as follows:

Unstructured data ingestion is getting extremely popular with the advances in Generative AI. 
Today, our only means of dealing with unstructured data is to store it as a binary blob inside Parquet, 
or point to files that exist in some object store with a string. These solutions fail to address these use 
cases, because of scalability, usability, and governance issues.

We would like to introduce a new logical type annotation in Parquet called “File” for storing a struct that 
contains a path reference to a file with additional metadata. This reference may be to a file that exists 
(or expected to exist) in storage at a given path. We’d like to define the minimum required list of fields 
that would allow a client to correctly read the referenced data. Any additional metadata can be optionally 
stored by engines and table formats as necessary adjacent to this type.

What changes are included in this PR?

Introduces the specification for FileType.

Do these changes have PoC implementations?

Yes:

emkornfield

Looks reasonable to me a few minor comments for clarification.

etseidl

Just a few questions I have after reviewing the Rust implementation.

brkyvz · 2026-07-10T15:53:47Z

Addressed your comments, @rok @pitrou please take a look. There's only one unaddressed comment on the CompressionCodec, which I will address after talking to @alkis

rdblue

I left a few comments for clarification, but I think that this is ready either way.

brkyvz · 2026-07-17T21:47:49Z

@rdblue @wgtmac @mapleFU @RussellSpitzer Thank you for the feedback! Addressed your comments. Hopefully things are clearer now! Let me know if they aren't

…@ 90147dc) Align with the latest apache/parquet-format#585 (commit 90147dc): - path is now a URI-reference per RFC 3986 (absolute or relative), not an opaque location string. Resolves the path/uri debate: name stays `path`, semantics become URI. - checksum form changes from <algorithm>:base64(<digest>) to <algorithm>:<digest> with per-algorithm encoding (lowercase hex for MD5/CRC32/CRC32C/SHA-256, opaque for ETAG); add RFC references and an Encoding column. - content_type: MIME per RFC 2046, defaults to application/octet-stream when unset. - offset must not be negative. - Note that implementations are not expected to treat empty strings as null. Co-authored-by: Isaac

rok · 2026-07-22T21:25:19Z

@brkyvz can you reply on the path thread?

brkyvz · 2026-07-22T22:05:44Z

@brkyvz can you reply on the path thread?

@rok sorry that was hidden in the UI. Responded! We weren't planning on renaming the field. Just explaining that it is a URI

…@ 92228a1) Align with the latest apache/parquet-format#585 (commit 92228a1): - Update checksum RFC citations: MD5 -> RFC 1321, CRC32 -> RFC 2083, CRC32C -> RFC 3385 (SHA-256 unchanged at RFC 6234). - Readers may return a null file value for a row whose reference is invalid. Co-authored-by: Isaac

rok · 2026-07-23T13:55:49Z

@brkyvz can you reply on the path thread?

@rok sorry that was hidden in the UI. Responded! We weren't planning on renaming the field. Just explaining that it is a URI

See my reply in the thread above. I don't think "we weren't planning" sufficiently answers this.

brkyvz · 2026-07-23T15:39:44Z

@rok @pitrou Renamed path to uri!

…o#585 @ 4223744) Align with apache/parquet-format#585 (commit 4223744): the FILE locator field was renamed from `path` to `uri`. Rename the field throughout the field table, resolution table, Parquet examples, writer/reader requirements, statistics, and prose. Generic uses of "path" (field path, absolute path) are unchanged, as is the DBR-specific __FILE_TYPE_MODE storage-mode key. Co-authored-by: Isaac

rok

Thanks @brkyvz this is a great addition to the format! Two minor nits that can also be ignored.

rok · 2026-07-23T19:47:44Z

   *   MAP - undefined
   *   VARIANT - undefined
   *   GEOMETRY - undefined
   *   GEOGRAPHY - undefined


Suggested change

* GEOGRAPHY - undefined

* FILE - undefined

rok · 2026-07-23T19:57:24Z

+can be renamed or relocated as a single unit.
+


Suggested change

can be renamed or relocated as a single unit.

can be renamed or relocated as a single unit.

Parquet files containing self-references must not use Parquet modular encryption.

Self-referenced byte ranges are not Parquet encryption modules and therefore cannot

be encrypted or authenticated independently. Encryption of external files referenced

by `uri` is outside the scope of the Parquet format.

Add FILE type definitions

7d966f0

emkornfield reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

emkornfield reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

emkornfield approved these changes Jun 10, 2026

View reviewed changes

danielcweeks reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

danielcweeks reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

Address comments

b6ae0de

RussellSpitzer approved these changes Jun 10, 2026

View reviewed changes

danielcweeks approved these changes Jun 10, 2026

View reviewed changes

etseidl reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

wgtmac reviewed Jun 11, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

divjotarora reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

sfc-gh-sgrafberger reviewed Jun 12, 2026

View reviewed changes

Comment thread LogicalTypes.md

alkis reviewed Jun 17, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift Outdated

CTTY reviewed Jun 17, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

pitrou reviewed Jun 30, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread LogicalTypes.md Outdated

sfc-gh-sgrafberger reviewed Jul 2, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift

brkyvz added 2 commits July 2, 2026 16:30

update spec based on Alkis' proposal

0f05340

minor changes

bcbabcc

etseidl reviewed Jul 2, 2026

View reviewed changes

alkis reviewed Jul 6, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

This was referenced Jul 6, 2026

[PROTOCOL RFC] File Data Type delta-io/delta#7147

Open

[Protocol RFC] File data type delta-io/delta#7148

Open

brkyvz added 2 commits July 7, 2026 22:05

address comments

117b3d1

save changes

4ff444e

danielcweeks approved these changes Jul 13, 2026

View reviewed changes

rdblue reviewed Jul 13, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jul 13, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jul 13, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jul 13, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jul 13, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jul 13, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jul 13, 2026

View reviewed changes

Comment thread LogicalTypes.md

rdblue reviewed Jul 13, 2026

View reviewed changes

Comment thread LogicalTypes.md

rdblue approved these changes Jul 13, 2026

View reviewed changes

mapleFU reviewed Jul 14, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

wgtmac reviewed Jul 14, 2026

View reviewed changes

Comment thread LogicalTypes.md

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md

Comment thread LogicalTypes.md Outdated

RussellSpitzer reviewed Jul 14, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

address comments

90147dc

rok suggested changes Jul 17, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

rok reviewed Jul 18, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

wgtmac approved these changes Jul 22, 2026

View reviewed changes

brkyvz added 2 commits July 22, 2026 17:25

Address comments

989099b

added invalid read behavior

92228a1

rename path to uri

4223744

rok approved these changes Jul 23, 2026

View reviewed changes

-can be renamed or relocated as a single unit.
+can be renamed or relocated as a single unit.
+Parquet files containing self-references must not use Parquet modular encryption.
+Self-referenced byte ranges are not Parquet encryption modules and therefore cannot
+be encrypted or authenticated independently. Encryption of external files referenced
+by `uri` is outside the scope of the Parquet format.

Uh oh!

Conversation

brkyvz commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Do these changes have PoC implementations?

Uh oh!

Uh oh!

Uh oh!

emkornfield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brkyvz commented Jul 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rdblue left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brkyvz commented Jul 17, 2026

Uh oh!

Uh oh!

brkyvz commented Jun 9, 2026 •

edited

Loading