verity volumes: attach verified read-only data disks to CVMs#752
Open
h4x3rotab wants to merge 23 commits into
Open
verity volumes: attach verified read-only data disks to CVMs#752h4x3rotab wants to merge 23 commits into
h4x3rotab wants to merge 23 commits into
Conversation
h4x3rotab
changed the title
h4x3rotab
force-pushed
the
verity-volumes
branch
12 times, most recently
from
July 4, 2026 23:16
b0a463a to
109c584
Compare
![github-advanced-security[bot]](https://cdn.github.com/in/57789?s=60&v=4){kind=small}
kvinwang
changed the title
Add read-only verity volumes -- extra virtio-blk disks a CVM can mount
instead of pulling and unpacking their contents. A volume is declared in
app-compose.json as `{ verity_root, target }`, and attached at deploy time
with `--volume <name>`: the vmm looks the name up under cvm.volumes_dir and
attaches it read-only. Because verity_root is part of the measured compose,
the guest can check the bytes it's handed against it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Just before dockerd starts, dstack-prepare.sh runs the seeding helper. For each volume declared in the compose it finds the matching disk by opening it with veritysetup against the measured verity_root, then either seeds docker's overlay2 store (target "docker") so the images are already present, or mounts the volume at a path (a data volume). It's fail-safe throughout: a volume that's missing or doesn't verify is skipped, and its images just pull. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`dstack verity <image>...` builds a squashfs + dm-verity volume that pre-extracts docker images (or, with `--dir`, a plain directory) so a CVM can start without pulling or unpacking them. It fetches images itself through oci-client -- no docker daemon -- and lays out the overlay2 store deterministically (each layer's directory id is its chain-id, with a fixed timestamp and salt). The same inputs always produce the same verity_root, so anyone can recompute it from the pinned image digests and confirm what's in the volume without trusting the builder. `dstack deploy --volume` attaches the result. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Adds generic read-only verity data volumes to CVMs. An operator builds or wraps a filesystem image, places it in the VMM's configured
volumes_dir, and attaches it by name at deploy time. The guest discovers the disk, verifies it against the root measured inapp-compose.json, and mounts it read-only at the requested absolute path.How it fits together
dstack verity --dir <path>packs a directory into a reproducible squashfs volume.dstack verity --fs-image <path>wraps an existing filesystem image.DSTACK_VOLUMEenvelope in partition 1, filesystem data in partition 2, and the dm-verity superblock/hash tree in partition 3.cvm.volumes_dirand attaches the raw disks read-only.dstack-volumehelper scans/sys/class/block, matches the full root, opens the disk withveritysetupusing the measured root, and mounts it before the application starts.verity_rootandtargetthroughapp-compose.jsonand therefore the measuredapp_id.Example:
See
docs/verity-volumes.md.Why Docker image seeding is not included
An earlier version injected pre-extracted layers directly into Docker's private overlay2 state. That couples the feature to Docker internals (
imagedb,layerdb,overlay2, and repository metadata) and creates unsafe lifecycle edge cases: bind mounts disappear on reboot while Docker metadata persists, pulled images may share layers with verity-seeded images, and removing or changing a volume can leave existing images or containers with dangling layer references.Safely reconciling that state would require ownership tracking and Docker-specific graph cleanup, while a custom containerd snapshotter would be a substantially larger and less mature integration. This PR therefore keeps the trust and lifecycle boundary simple: it only exposes verified, read-only data filesystems and does not modify Docker image state.
Validation
cargo clippy -p dstack-volume --all-targets -- -D warningscargo clippy -p dstack-cli --all-targets -- -D warningscargo test -p dstack-volume -p dstack-cli -p dstack-typescargo test -p dstack-vmm --no-fail-fastdstack verity --dir