Support v3 `variant` data type

[zhjwpku]

[nssalian]

Thanks for creating this @zhjwpku , do you mind adding sub tasks around non shredding first, then shredding as well. You can look here under the go tracker Track A to have something similar for cpp

[zhjwpku]

Thanks for creating this @zhjwpku , do you mind adding sub tasks around non shredding first, then shredding as well. You can look here under the go tracker Track A to have something similar for cpp

Just dug into some details about Variant Shredding [1].

[1] https://parquet.apache.org/docs/file-format/types/variantshredding/

[nssalian]

@zhjwpku I would be best to add support for Variant as a type first which is usually writing unshredded variant data. Shredded writes can follow.

[wgtmac]

@nssalian Thanks for your interest! Do you have a concrete plan for the variant type support? I'm not quite sure if we need variant encoding in the iceberg library. As we depend on arrow-cpp for parquet, I'm not sure if we want to reuse variant encoding from arrow-cpp. (cc @HuaHuaY who is adding variant encoding to arrow-cpp)

[nssalian]

@wgtmac thanks for weighing in. Yes, the plan is to wire arrow-cpp's variant into iceberg-cpp's type system, not duplicate the codec. iceberg-go follows the same pattern (defers to arrow-go's variant).