backup: S3 encoder for buckets, objects, and blob reassembly (Phase 0a) - #718
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on top of #716 (DynamoDB) and #717 (s3keys.ParseBlobKey). Adds the S3 encoder for the Phase 0 logical-backup decoder — the most complex per-adapter piece because it must reassemble multipart object bodies from independent blob chunks.
Snapshot prefixes handled:
!s3|bucket|meta|<bucket>→s3/<bucket>/_bucket.json!s3|obj|head|<bucket><gen><object>→s3/<bucket>/<object>.elastickv-meta.json(sidecar)!s3|blob|<bucket><gen><object><uploadID><partNo><chunkNo>[<partVersion>]→ spilled to scratch on arrival; concatenated intos3/<bucket>/<object>at Finalize in(partNo, partVersion, chunkNo)order!s3|upload|meta|/!s3|upload|part|: excluded by default; opt in via--include-incomplete-uploadsto emit under_incomplete_uploads/records.jsonl!s3|bucket|gen|,!s3|gc|upload|,!s3route|: ignoredHow body assembly works
Each blob chunk is written atomically to a per-(bucket, object) scratch directory as it arrives, registered under the
(uploadID, partNo, chunkNo, partVersion)routing key. At Finalize, chunks for each object are sorted and concatenated into a single body file via tmp+rename. Body bytes are never held in memory — only the scratch path map. The scratch tree is removed on Finalize.Reserved-suffix collision handling
A user object key ending in
.elastickv-meta.jsonis rejected withErrS3MetaSuffixCollisionby default.WithRenameCollisions(true)appends.user-datato the body file and records the rename ins3/<bucket>/KEYMAP.jsonlwithKindMetaCollisionso the dump remains reversible. Per-bucketKEYMAP.jsonlis opened lazily and dropped on Finalize if no records were written.Test plan
go test -race ./internal/backup/...— pass.golangci-lint run ./internal/backup/...— clean.--include-incomplete-uploadsround-trip + default skip, versioned blob assembly.Stacking
Base:
feat/backup-phase0a-dynamodb(PR #716). Also depends onfeat/s3keys-parse-blob-key(PR #717).