feat: optimize partial loads by coalescing (or splitting) range reads

[clintropolis]

Description

This PR optimizes partial loads by coalescing range reads into a single request when the gap between them is under a threshold (default 1MiB) as well as split range reads into multiple requests when over a threshold (default 64 MiB) to attempt to achieve a balance of maximizing parallelism, but sensibly so that we don't suffer too much overhead from having too many individual requests.

new configs:

config	description	default
druid.storage.virtualStorageCoalesceGapBytes	Maximum number of unrequested bytes a partial-download range read will fetch in order to bridge two requested internal files into a single deep-storage request. Bridged bytes are whole valid internal files that are kept in the local cache (they aren't waste), so this trades at most this many extra bytes per bridged gap (one read can bridge several gaps) for one fewer deep-storage round trip each. A value less than or equal to 0 will disable this bridging behavior.	1MiB
druid.storage.virtualStorageMaxFetchRunBytes	Maximum size of a single range read in a query-driven partial download. Larger fetches split at internal-file boundaries into multiple reads of at most this size that proceed concurrently, so throughput isn't bounded by a single deep-storage connection. A value less than or equal to 0 will disable splitting.	64MiB

This transitions the prior load by side-effect behavior of partial loads into something planned and purposeful; to support this V10 segments now have some additional metadata which tracks which column internal files are associated with, making it easy to translate a CursorBuildSpec into an explicit list of the internal files needed to serve it. The prior load-by-side-effect behavior should not happen in any production paths, as this PR transitions any v10 segments written prior to this metadata change to fall-back to 'download the entire bundle' behavior.

This change also fixes a bug with partial load of nested columns, which prior to this change was missing the loading of nested field columns, so any query using them would incorrectly perform downloads on the processing thread (sad). This is fixed because now all of those internal files are associated with the nested column, so we can fetch the entire set when we intend to. This is actually not ideal in most cases, I have planned some future work to make improvements to the CursorBuildSpec -> internal files list to recognize nested field virtual columns and do a more targeted download to only fetch the internal files associated with the field instead of all fields belonging to the json column (but not in this PR).

[FrankChen021]

Severity	Findings
P0	0
P1	1
P2	0
P3	0
Total	1

Severity	Findings
P0	0
P1	1
P2	0
P3	0
Total	1

Reviewed 29 of 29 changed files.

---

This is an automated review by Codex GPT-5.6-Sol

[FrankChen021]

[P1] Keep holder construction off the caller thread

The no-op token can complete before AsyncResources.collect and addReadyCallback are wired, and callbacks registered on an already-ready resource execute synchronously. In that ordering, the callback materializes columns and may lazily download projection-parent files on the processing thread, violating the async I/O contract. Schedule the continuation itself on the download executor instead of relying on the token's completion thread.

[jtuglu1]

Question 1: Per-historical, do we track in-flight range requests so we avoid duplicate calls? E.g. query 1 calls for range A, not available locally so we initiate fetch from S3, meanwhile query 2 calls for range A. Does query 2 know about the in-flight request to avoid sending a duplicate?

Question 2: how are we balancing coalescing ranged GET calls while maximizing parallelism at the S3 connection level?

Copying from the prior PR

[clintropolis]

Question 1: Per-historical, do we track in-flight range requests so we avoid duplicate calls? E.g. query 1 calls for range A, not available locally so we initiate fetch from S3, meanwhile query 2 calls for range A. Does query 2 know about the in-flight request to avoid sending a duplicate?

yes, though sort of indirectly, there is a set of locks associated with each internal file of the segment which dedupe calls to the same range, so one caller does the work and the others just wait and pick up the result. These locks are acquired in a stable order so that competing callers dont lock each other out. The locking isn't new in this PR, the changes here are more around efficiency/optimization.

Question 2: how are we balancing coalescing ranged GET calls while maximizing parallelism at the S3 connection level?

this is tunable, virtualStorageCoalesceGapBytes defaults to 1MiB and controls the coalescing part (combining multiple small requests into 1 request) while virtualStorageMaxFetchRunBytes defaults to 64MiB and defines when we consider splitting ranges into parallel calls (splits are aligned with internal file boundaries). The default tuning aims for chunk download walltime of ~1 second. I'm not sure how good these defaults are, still need to do some testing.