RFC: Versioning only

[emkornfield]

A smaller scoped PR then #588 to frame the first parse of the discussion around versioning (and when to enable as default).

[emkornfield]

IMO 6 months is the minimal reasonable amount of time before a feature can be enabled by default. It guarantees at least 1 quarter has passed and therefore at least one release of readers for an implementation.

[alamb]

Thanks @emkornfield -- the only thing I see missing here is how the parquet writer would communicate to readers what features were used in a partcular file.

To make this proposal an improvement over the status quo, I think we would also need to add an extra field to the footer metadata that encodes the minimum parquet-format version of forward incompatible features used (aka the minimium parquet-format version support required to successfully read the file)

FWIW I also think this scheme would work (and I would persnally be happy to support it).

[emkornfield]

Thanks @emkornfield -- the only thing I see missing here is how the parquet writer would communicate to readers what features were used in a partcular file.

@alamb I'm specifically #588 (PARX magic number is roughly what I was thinking here) but I specifically removed it from this PR because I think we are trying to build consensus on a "logical level/process" first, because getting to the physical level.

To make this proposal an improvement over the status quo, I think we would also need to add an extra field to the footer metadata that encodes the minimum parquet-format version of forward incompatible features used (aka the minimium parquet-format version support required to successfully read the file)

I'm hesitant to use version number as something implementations use to consume to determine if they could read the file (I do think it is very useful debug information and can help provide useful errors to end users,). I think it fails in two directions:

1. It potentially allows readers to read incorrect data. Consider the scenario of a new major version with 2 breaking changes. A reader implements the first of the features (the second one has been discussed yet). For the first feature the reader must "accept" the new major version (i.e. not fail). The reader is then released. The second feature is then added to the specification, this feature is not detectable by other information in the footer but changes how to interpret data on disk. When writers write out the full version, the old reader would silently return incorrect data.
2. It potentially breaks more readers then necessary. Consider a new format version that adds a compression algorithm. This new version is bumped to be the the default writer version, but the user chooses a different compression algorithm for there file. Using the new format version breaks old readers that don't understand it. This can be solved by having writers try to dynamically "downgrade" but it adds more complexity that writers need to manage. It duplicates the information already in the footer (i.e. the CompressionCodec enum).

[alamb]

1. It potentially allows readers to read incorrect data. Consider the scenario of a new major version with 2 breaking changes. A reader implements the first of the features (the second one has been discussed yet). For the first feature the reader must "accept" the new major version (i.e. not fail). The reader is then released. The second feature is then added to the specification, this feature is not detectable by other information in the footer but changes how to interpret data on disk. When writers write out the full version, the old reader would silently return incorrect data.

My understanding is that the second feature would be classified as "forward incompatible" and thus could not be written to a new file without changing the embedded version / feature listing

2. It potentially breaks more readers then necessary. Consider a new format version that adds a compression algorithm. This new version is bumped to be the the default writer version, but the user chooses a different compression algorithm for there file. Using the new format version breaks old readers that don't understand it. This can be solved by having writers try to dynamically "downgrade" but it adds more complexity that writers need to manage. It duplicates the information already in the footer (i.e. the CompressionCodec enum).

This is true -- on the other hand writers could reduce the impact by emitting the lowest version number required to read all features in the file (rather than the most recent feature level the writer supports). The feature set could be restricted as necessary.

I agree this is more coarse grained than a feature-by-feature (or buckets of features) listing, but it is simpler to communicate and implement, which is why I think it is worthc onsidering

[emkornfield]

It potentially allows readers to read incorrect data. Consider the scenario of a new major version with 2 breaking changes. A reader implements the first of the features (the second one has been discussed yet). For the first feature the reader must "accept" the new major version (i.e. not fail). The reader is then released. The second feature is then added to the specification, this feature is not detectable by other information in the footer but changes how to interpret data on disk. When writers write out the full version, the old reader would silently return incorrect data.

My understanding is that the second feature would be classified as "forward incompatible" and thus could not be written to a new file without changing the embedded version / feature listing

@alamb I think this only works with a version if you guarantee every forward incompatible feature has its own release (and therefore version). This was not intended as a strict requirement of this proposal. IMO two forward incompatible features could end up in the same release if they land close enough together or the community moves away from on-demand releases a regular time based cadence. This would not be a problem with "feature lists". Ideally, how we communicate versions at a physical level should not need to change if the release process changes.

This is true -- on the other hand writers could reduce the impact by emitting the lowest version number required to read all features in the file (rather than the most recent feature level the writer supports). The feature set could be restricted as necessary.

Definitely, this is an alternative. I think for writers that want to be smart and pick the lowest version, it ends up being extra work. The feature list already needs to be tracked, but they then need to properly calculate the lowest possible version as well.

I agree this is more coarse grained than a feature-by-feature (or buckets of features) listing, but it is simpler to communicate and implement, which is why I think it is worthc onsidering

Yes, this is a nice to have and version numbers could work here with the trade-off you mention. My perspective is colored by operating a multitenant service reading and writing parquet files with a wide variety of producers and consumers. The more things we can do to make parquet files compatible with old readers the better. My primary concern would be ensuring the first failure mode never happens.

For communication at a user/debugging level I agree version number should always be present in the footer. If implementers don't want to be smart about tracking individual features then I think they could simply keep a map of Version number to feature list. This would not add a lot of complexity but bias implementations towards allowing finer grained compatibility. In other words, users should not need to think about the subtlety here but I think it is reasonable for implementators to do so.