Skip to content

backup: S3 encoder for buckets, objects, and blob reassembly (Phase 0a) - #718

Merged
bootjp merged 35 commits into
feat/backup-phase0a-dynamodbfrom
feat/backup-phase0a-s3
May 1, 2026
Merged

backup: S3 encoder for buckets, objects, and blob reassembly (Phase 0a)#718
bootjp merged 35 commits into
feat/backup-phase0a-dynamodbfrom
feat/backup-phase0a-s3

Conversation

@bootjp

@bootjp bootjp commented Apr 30, 2026

Copy link
Copy Markdown
Owner

Summary

Stacked on top of #716 (DynamoDB) and #717 (s3keys.ParseBlobKey). Adds the S3 encoder for the Phase 0 logical-backup decoder — the most complex per-adapter piece because it must reassemble multipart object bodies from independent blob chunks.

Snapshot prefixes handled:

  • !s3|bucket|meta|<bucket>s3/<bucket>/_bucket.json
  • !s3|obj|head|<bucket><gen><object>s3/<bucket>/<object>.elastickv-meta.json (sidecar)
  • !s3|blob|<bucket><gen><object><uploadID><partNo><chunkNo>[<partVersion>] → spilled to scratch on arrival; concatenated into s3/<bucket>/<object> at Finalize in (partNo, partVersion, chunkNo) order
  • !s3|upload|meta|/!s3|upload|part|: excluded by default; opt in via --include-incomplete-uploads to emit under _incomplete_uploads/records.jsonl
  • !s3|bucket|gen|, !s3|gc|upload|, !s3route|: ignored

How body assembly works

Each blob chunk is written atomically to a per-(bucket, object) scratch directory as it arrives, registered under the (uploadID, partNo, chunkNo, partVersion) routing key. At Finalize, chunks for each object are sorted and concatenated into a single body file via tmp+rename. Body bytes are never held in memory — only the scratch path map. The scratch tree is removed on Finalize.

Reserved-suffix collision handling

A user object key ending in .elastickv-meta.json is rejected with ErrS3MetaSuffixCollision by default. WithRenameCollisions(true) appends .user-data to the body file and records the rename in s3/<bucket>/KEYMAP.jsonl with KindMetaCollision so the dump remains reversible. Per-bucket KEYMAP.jsonl is opened lazily and dropped on Finalize if no records were written.

Test plan

  • go test -race ./internal/backup/... — pass.
  • golangci-lint run ./internal/backup/... — clean.
  • 11 tests covering bucket-meta round-trip, multipart out-of-order chunk reassembly, orphan-chunks warning, sidecar collision (rejected/renamed), JSON-decode rejection on both paths, ignored-prefix no-op, --include-incomplete-uploads round-trip + default skip, versioned blob assembly.

Stacking

Base: feat/backup-phase0a-dynamodb (PR #716). Also depends on feat/s3keys-parse-blob-key (PR #717).

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant