feat(database): self-service snapshot management and rollback - #134
Open
tjorri wants to merge 13 commits into
Open
feat(database): self-service snapshot management and rollback#134tjorri wants to merge 13 commits into
tjorri wants to merge 13 commits into
Conversation
Request (and its Get/Post/Put/Delete helpers) now returns a typed
*HTTPError on non-2xx responses, wrapping the status code, method,
URL, raw body, and parsed error message. Callers can use errors.As
to discriminate between 400/404/409/500 and produce user-friendly
messages. Existing callers that just bubble the error up keep
working unchanged.
The parser handles the {"error": "..."} JSON body shape used by the
backend APIs and falls back to the trimmed raw body when the shape
does not match or the body is not JSON.
The raw "Request failed with status code ...: <body>" Error-level log
was useful when callers received an opaque error and had no way to see
what went wrong. Now that Request returns a typed *HTTPError carrying
the status code, method, URL, raw body, and parsed server message,
callers can produce informative user-facing errors on top — and the
raw log becomes redundant noise when the caller does its job.
Demote the log to Debug level *only* when the response body parses
cleanly as the expected {"error": "..."} JSON shape. For opaque
bodies (non-standard JSON, plain text, HTML intercepted by a proxy,
etc.) keep the log at Error level so users still see the raw
diagnostic content even when the caller only produces a generic
wrapper error — this preserves backwards-compatible behavior for any
endpoint that does not conform to the structured error contract.
parseHTTPErrorMessage now returns (message, structured bool) so the
calling code can pick the log level. HTTPError's shape is unchanged;
structured and opaque errors both populate Message with the best
available summary of the failure.
Add Go types and client methods on *TargetEnvironment that wrap the database operations API (cloud-managed snapshots and point-in-time rollback). Covers all nine endpoints: * GetDatabaseCapabilities * GetDatabaseInfo * ListDatabaseSnapshots (with type filter and per-shard limit) * GetDatabaseSnapshot * CreateDatabaseSnapshot * DeleteDatabaseSnapshot * RollbackDatabase * ListDatabaseOperations * GetDatabaseOperation Includes a DatabaseOperation.IsTerminal helper and constants for provider ids, snapshot types, operation types, and operation statuses. Snapshot identifiers are path-escaped in URLs so values containing colons (e.g. automated snapshot ids) round-trip cleanly.
Add three read-only commands for cloud-managed database operations on a
given environment:
* metaplay database info ENVIRONMENT
Shows the managed database provider, per-shard cluster state, manual
snapshot quota usage (e.g. '3 / 5'), and the point-in-time rollback
window. Environments without a dedicated managed database cluster
report as 'not supported' rather than erroring.
* metaplay database snapshot list ENVIRONMENT [--type] [--limit]
Lists cloud-managed database snapshots across all shards, sorted
newest first. Supports filtering by type (manual/automated/backup).
* metaplay database operation list ENVIRONMENT
Lists in-progress database operations (snapshot creates/deletes and
rollbacks). Useful for discovering work after CLI disconnects.
* metaplay database operation status ENVIRONMENT OPERATION_ID [--watch]
Shows the current status of a single operation, optionally polling
until the operation reaches a terminal state.
All commands support --format=text (default) or --format=json for
scripting. Adds a shared cmd/database_helpers.go with format validation,
HTTP-error-to-CLIError mapping (with context-aware hints for quota,
concurrency, and unsupported-environment cases), shard resolution for
--shard / --all-shards, target-time parsing for upcoming rollback, and
age/time rendering helpers. Unit tests cover the pure helpers.
New 'snapshot' and 'operation' subcommand groups coexist alongside the
existing flat 'database export-snapshot' / 'database import-snapshot'
commands — those operate on ad-hoc SQL dumps and are a different feature.
Add two mutating commands for cloud-managed database snapshots:
* metaplay database snapshot create ENVIRONMENT [--shard|--all-shards]
[--name] [--description] [--no-wait]
Creates a manual snapshot on one or more shards. By default waits
and polls until the snapshot is available; --no-wait returns as
soon as the request is accepted. --all-shards fans out creation
across every shard in parallel via syncutil.ParallelMap.
* metaplay database snapshot delete ENVIRONMENT [SNAPSHOT_ID] [--yes]
[--no-wait]
Deletes a manual snapshot. In interactive mode, omitting the
SNAPSHOT_ID opens a picker listing the environment's manual
snapshots. Non-interactive mode requires both the id and --yes.
Refuses to delete automated or backup-service snapshots.
Adds shared async infrastructure in database_helpers.go:
* waitForDatabaseOperation polls an operation id to terminal state,
invoking an onStatusChange callback on each status transition.
* runShardOperation wraps a single-shard mutating call with logging,
polling, and failure reporting, returning a shardOperationResult.
* aggregateShardResults joins per-shard results from a fan-out into
a single CLIError listing the failing shards.
Add 'metaplay database rollback ENVIRONMENT [--shard|--all-shards] --target-time=... [--force] [--yes] [--confirm-production]'. The command rolls the environment's cloud-managed database back to a previous point in time via the provider's native point-in-time recovery mechanism. It is an in-place operation — no cluster delete/restore is performed. Features: * --target-time accepts either an absolute RFC3339 timestamp or a relative duration (30m, 2h) meaning 'that long ago from now'. * In interactive mode, if --target-time is omitted, the CLI prints the available rollback window for the selected shard(s) and prompts for a value. * Warns (but does not reject) when the requested target time is outside the reported rollback window — the backend is the ultimate arbiter. * Pre-flight check: refuses to run if a game server is currently deployed in the environment (split-brain risk). --force overrides with a loud warning. * Production gate: requires --confirm-production for environments of type production, matching the 'database reset' convention. * --all-shards fans out rollbacks across every shard in parallel. * --no-wait returns as soon as the request is accepted instead of polling to completion.
Add a shared resolveEnvironmentForDatabaseOps helper used by every 'metaplay database ...' command as its first step. It resolves the target environment, builds a TargetEnvironment, and probes the database capabilities endpoint as a readiness check. Any error from the probe is translated into a single clear "database operations are not available for this environment" error with the underlying cause attached. The capabilities endpoint is specifically designed as the "are database operations supported?" probe, so treating any probe error as "not available" gives consistent, actionable feedback regardless of how the infrastructure stack fails to respond. This covers the real-world case of older stacks running infrastructure versions that do not yet expose the database operation endpoints at all — users now see a clear upgrade hint instead of a cryptic protocol error. Because we now have the capabilities data on hand, add a local ensureShardsSupportCapability pre-flight check and use it to: * reject 'snapshot create' on shards where SupportsSnapshots is false * reject 'rollback' on shards where SupportsRollback is false Single-shard failures name the specific shard in the error; multi-shard fan-outs (--all-shards) report a summary listing which shards would be rejected, so the user can pick a different --shard or upgrade the stack before retrying. Every command (info, snapshot list/create/delete, rollback, operation list/status) was refactored to use the shared helper, removing per- command env-resolution boilerplate and the prior ad-hoc capabilities calls in info/create/rollback. The 400-not-supported string inference in mapDatabaseHTTPError is no longer needed and has been removed — the pre-flight probe makes it dead code. New unit tests cover the uniform error-mapping behavior across success, empty-shards, 404-empty-body, and 404-with-structured-body responses, plus single-shard and multi-shard variants of the capability pre-check.
Add a METAPLAYCLI_STACKAPI_BASEURL environment variable that, when set, overrides the StackAPI base URL normally constructed from each environment's stack domain. Intended for local development against a StackAPI running on the developer's machine (pointed at a real cluster for tenant discovery and auth) — previously the only way to hit a local StackAPI instance was to bypass the CLI entirely and curl it directly. Mirrors the existing METAPLAYCLI_PORTAL_BASEURL override pattern in pkg/common/config.go. When the override is active, NewTargetEnvironment logs an info line so the user is never confused about which backend their commands are targeting.
Snapshot create / delete and rollback can take multiple minutes, during which the backend operation stays in a stable 'in-progress' status with no new updates to render. Previously waitForDatabaseOperation only logged on status transitions, so the CLI would appear hung until the operation finished. Add a periodic heartbeat emitted every 15 seconds while the status is stable, rendered as a muted '… still <status> (elapsed 1m30s)' line (and including a progress percentage if the backend ever starts reporting one). The heartbeat clock resets on every genuine status change, so there is no double-output when statuses transition. Refactor 'database operation status --watch' to delegate polling to waitForDatabaseOperation for a single source of truth, and change the helper to accept the already-known initial operation so it can defer its first poll by one interval. This eliminates both a redundant round-trip and a duplicate render that --watch produced previously.
aggregateShardResults previously always wrapped per-shard errors in a "X failed on N of M shards" summary. For single-shard commands (which is the common case outside --all-shards fan-out) this produced awkward output like "snapshot create failed on 1 of 1 shards" followed by a nested shard-scoped error that already contained all the useful info. Return the single shard's error directly when only one shard was targeted. Multi-shard fan-outs keep the summary + details rendering since that shape is genuinely more informative when partial failures affect only some shards.
tjorri
force-pushed
the
feature/database-operations-commands
branch
from
April 11, 2026 10:46
f8edabb to
01d5115
Compare
Some operation types (notably snapshot-delete) have no reliable source for a creation timestamp on the backend — AWS retains no trace of the delete once the snapshot is gone, and StackAPI is stateless so it cannot record the initiation time itself. The backend now leaves CreatedAt unset for those operations (see platform-apps fix), which the Go value type marshals as the zero time. Skip the 'created:' line in operation status output when CreatedAt is zero, instead of printing the confusing '- (- ago)' placeholder the formatters would otherwise produce.
tjorri
marked this pull request as ready for review
April 14, 2026 11:32
…el commands Rename 3-level nested commands to flat 2-level to match the rest of the CLI (e.g. secrets list, secrets delete). The snapshot and operation parent groups are removed; all commands register directly on databaseCmd. database snapshot list → database list-snapshots database snapshot create → database create-snapshot database snapshot delete → database delete-snapshot database operation list → database list-operations database operation status → database operation-status
The StackAPI moved its tenant-scoped database endpoints from /v0/databases/<env>/* to /tenant/v1/<env>/databases/*, alongside credentials, observability, and environment. Update the CLI client to follow. No request/response shape changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #133 — the typed
metahttp.HTTPErrorintroduced there is used extensively by the new database command error mapping. Review / merge #133 first; this PR will show an overlapping diff againstmainuntil then.Summary
Adds a new set of
metaplay database ...subcommands for managing cloud-managed database snapshots and performing point-in-time rollback on an environment's managed database cluster. All commands are driven by the environment's database operations API and work with any provider the backend supports (currently AWS RDS).New commands
All commands support
--format=text(default) or--format=json.These live alongside the existing
database export-snapshot/database import-snapshot/database resetcommands, which operate on ad-hoc SQL dumps and are a different feature.Highlights
Flat 2-level command naming. All new commands follow the same
database <verb-noun>pattern as the rest of the CLI — no nested subcommand groups.Wait-by-default async. Mutating operations (
create-snapshot,delete-snapshot,rollback) are async at the backend. By default the CLI waits and polls until the operation reaches a terminal state, rendering status changes as they happen.--no-waitreturns as soon as the request is accepted, printing just the operation id for scripts to pick up later.Parallel
--all-shardsfan-out. For multi-shard environments,create-snapshotandrollbackcan fan out across every shard in parallel usingsyncutil.ParallelMap. Each shard logs its own progress with a[shard N]prefix, and a partial failure reports which shards failed without cancelling the in-flight ones.Flexible rollback target time.
--target-timeaccepts either an absolute RFC3339 timestamp (2026-04-09T15:00:00Z) or a relative Go duration (30m,2h) meaning "that long ago". In interactive mode, omitting the flag prints the available rollback window and prompts for a value.Safety gates for rollback.
database rollbackmatches the existingdatabase resetconventions:--confirm-production--yes--forceoverrides with a loud warningInteractive snapshot picker.
database delete-snapshotwithout aSNAPSHOT_IDargument shows a picker listing the environment's manual snapshots. In non-interactive mode the id is required.Capabilities-driven readiness probe. Every database command starts by probing the capabilities endpoint as a readiness check. Any failure is translated into a uniform "database operations are not available for this environment" error with the underlying cause attached, which transparently handles the real-world case of older infrastructure stacks that do not yet expose these endpoints.
Local pre-flight shard validation. Because the capabilities response is already in hand after the probe,
create-snapshotandrollbackreject unsupported shards locally ("Rollback is not supported on shard 2 (mygame-2)") instead of round-tripping to the backend.Files
pkg/envapi/database.go— types + client methods mirroring the 9 database operation endpointscmd/database_helpers.go— shared plumbing: pre-flight probe, HTTP-error mapping, shard resolution, target-time parser, capability pre-check, async polling helper, fan-out result aggregation, table rendering, age/time formatterscmd/database_info.go—database infocmd/database_snapshot_{list,create,delete}.go—database list-snapshots,create-snapshot,delete-snapshotcmd/database_rollback.go—database rollbackcmd/database_operation_{list,status}.go—database list-operations,operation-statuscmd/database_helpers_test.go,pkg/envapi/database_test.go— unit testsTest plan
go build ./...is cleango vet ./...is cleango test ./...passes (new tests for envapi client, pre-flight probe error mapping, shard capability pre-check, target-time parser, format validator, HTTP error mapping, shard resolution)go mod tidyis cleango run . database --helpshows the new subcommands alongside the existing onesgo run . database {info,list-snapshots,create-snapshot,delete-snapshot,rollback,list-operations,operation-status} --helprenders correctlydatabase inforeports provider, per-shard engine/status, quota usage, rollback windowdatabase create-snapshot --name=...waits and completes; new snapshot appears inlist-snapshotsdatabase create-snapshot --no-waitreturns the operation id immediatelydatabase list-operationsshows the in-progress createdatabase operation-status <op-id> --watchpolls to completiondatabase delete-snapshot <id>prompts, waits, confirms deletiondatabase rollback --target-time=5mrefuses with--forcehint while a game server is deployeddatabase rollback --target-time=5m --force --yesinitiates a rollback and waits--target-timevalue surfaces a usage error listing both accepted formats