[Feature Request] Supports StageOnly metadataUpdate

[yingjianwu98]

Feature Request / Improvement

Currently in java, there is a way to set stageOnly for snapshotUpdate so that only the snapshots are created but the refs are not updated.

[yingjianwu98]

We have some internal used case for this feature and If people think this is a useful feature, I am willing to contribute.

[dingo4dev]

@stevie9868
Would you share more on this use case? I think it will be useful and we can add some examples and docs on this feature.

[kevinjqliu]

ah this is used for WAP
https://github.com/apache/iceberg/blob/c3d50e177ce17a45f4dd3af7593350dd6724ca8f/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java#L221-L226

And its different from writing to branches...

Is the expected operations here to commit to the main branch but not fast forward the reference?

[kevinjqliu]

Would be interesting to get WAP feature into pyiceberg

[yingjianwu98]

Is the expected operations here to commit to the main branch but not fast forward the reference?

Yeah that's correct, only trigger an AddSnapshotUpdate in pyiceberg but don't trigger a SetSnapshotRefUpdate when it's stageOnly defined here

Would be interesting to get WAP feature into pyiceberg

I Totally agree. Here are my thoughts to support this:

For wap.id

1. Implement stageOnly in snapshotProducer
2. Integrate stageOnly into the commit path
3. implement wap.id

For wap.branch:
It should be relatively easy, as I believe py-iceberg already support writing to branches.

Also looks like we haven't implemented cherrypick and fastforward operation in py-iceberg? Would be nice to add them as well.

[gabeiglio]

It seems that there is no api for cherry-pick and fast-forward. I create this issue to track it

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'