fix: preserve DimensionValueSetShardSpec on upgraded append segments

[abhishekrb19]

With #19571, when a concurrent replace task
upgrades a still-appending streaming task, the upgraded (new-version) copy of
each append segment previously adopted the pending segment's plain NumberedShardSpec,
dropping the DimensionValueSetShardSpec stamped at publish time. This made upgraded
segments unprunable by the broker.

The upgraded copy now takes its partition number and core-partition count
from the pending segment while carrying forward the original segment's
partitionDimensionValues when it is a DimensionValueSetShardSpec. The
value-set guarantee holds because the upgraded copy serves the same rows
as the original append segment.

This does not change the fact that pending segments are always allocated as numbered
shard specs, since tasks still advertise them as such — the value-set shard spec
currently applies only to recently-published segments.

This PR has:

• been self-reviewed.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• been tested in a test Druid cluster.