Skip to content

feat(database): self-service snapshot management and rollback - #134

Open
tjorri wants to merge 13 commits into
mainfrom
feature/database-operations-commands
Open

feat(database): self-service snapshot management and rollback#134
tjorri wants to merge 13 commits into
mainfrom
feature/database-operations-commands

Conversation

@tjorri

@tjorri tjorri commented Apr 10, 2026

Copy link
Copy Markdown
Contributor

Depends on #133 — the typed metahttp.HTTPError introduced there is used extensively by the new database command error mapping. Review / merge #133 first; this PR will show an overlapping diff against main until then.

Summary

Adds a new set of metaplay database ... subcommands for managing cloud-managed database snapshots and performing point-in-time rollback on an environment's managed database cluster. All commands are driven by the environment's database operations API and work with any provider the backend supports (currently AWS RDS).

New commands

metaplay database info             ENVIRONMENT
metaplay database list-snapshots   ENVIRONMENT [--type=manual|automated|backup] [--limit=N]
metaplay database create-snapshot  ENVIRONMENT [--shard=N|--all-shards] [--name] [--description] [--no-wait]
metaplay database delete-snapshot  ENVIRONMENT [SNAPSHOT_ID] [--no-wait] [--yes]
metaplay database rollback         ENVIRONMENT [--shard=N|--all-shards] --target-time=<RFC3339|duration> [--force] [--yes] [--confirm-production] [--no-wait]
metaplay database list-operations  ENVIRONMENT
metaplay database operation-status ENVIRONMENT OPERATION_ID [--watch]

All commands support --format=text (default) or --format=json.

These live alongside the existing database export-snapshot / database import-snapshot / database reset commands, which operate on ad-hoc SQL dumps and are a different feature.

Highlights

Flat 2-level command naming. All new commands follow the same database <verb-noun> pattern as the rest of the CLI — no nested subcommand groups.

Wait-by-default async. Mutating operations (create-snapshot, delete-snapshot, rollback) are async at the backend. By default the CLI waits and polls until the operation reaches a terminal state, rendering status changes as they happen. --no-wait returns as soon as the request is accepted, printing just the operation id for scripts to pick up later.

Parallel --all-shards fan-out. For multi-shard environments, create-snapshot and rollback can fan out across every shard in parallel using syncutil.ParallelMap. Each shard logs its own progress with a [shard N] prefix, and a partial failure reports which shards failed without cancelling the in-flight ones.

Flexible rollback target time. --target-time accepts either an absolute RFC3339 timestamp (2026-04-09T15:00:00Z) or a relative Go duration (30m, 2h) meaning "that long ago". In interactive mode, omitting the flag prints the available rollback window and prompts for a value.

Safety gates for rollback. database rollback matches the existing database reset conventions:

  • Production environments require --confirm-production
  • Non-interactive mode requires --yes
  • Refuses to run if a game server is currently deployed in the environment (split-brain risk); --force overrides with a loud warning

Interactive snapshot picker. database delete-snapshot without a SNAPSHOT_ID argument shows a picker listing the environment's manual snapshots. In non-interactive mode the id is required.

Capabilities-driven readiness probe. Every database command starts by probing the capabilities endpoint as a readiness check. Any failure is translated into a uniform "database operations are not available for this environment" error with the underlying cause attached, which transparently handles the real-world case of older infrastructure stacks that do not yet expose these endpoints.

Local pre-flight shard validation. Because the capabilities response is already in hand after the probe, create-snapshot and rollback reject unsupported shards locally ("Rollback is not supported on shard 2 (mygame-2)") instead of round-tripping to the backend.

Files

  • pkg/envapi/database.go — types + client methods mirroring the 9 database operation endpoints
  • cmd/database_helpers.go — shared plumbing: pre-flight probe, HTTP-error mapping, shard resolution, target-time parser, capability pre-check, async polling helper, fan-out result aggregation, table rendering, age/time formatters
  • cmd/database_info.godatabase info
  • cmd/database_snapshot_{list,create,delete}.godatabase list-snapshots, create-snapshot, delete-snapshot
  • cmd/database_rollback.godatabase rollback
  • cmd/database_operation_{list,status}.godatabase list-operations, operation-status
  • cmd/database_helpers_test.go, pkg/envapi/database_test.go — unit tests

Test plan

  • go build ./... is clean
  • go vet ./... is clean
  • go test ./... passes (new tests for envapi client, pre-flight probe error mapping, shard capability pre-check, target-time parser, format validator, HTTP error mapping, shard resolution)
  • go mod tidy is clean
  • go run . database --help shows the new subcommands alongside the existing ones
  • go run . database {info,list-snapshots,create-snapshot,delete-snapshot,rollback,list-operations,operation-status} --help renders correctly
  • End-to-end against a real environment with a dedicated Aurora database:
    • database info reports provider, per-shard engine/status, quota usage, rollback window
    • database create-snapshot --name=... waits and completes; new snapshot appears in list-snapshots
    • database create-snapshot --no-wait returns the operation id immediately
    • database list-operations shows the in-progress create
    • database operation-status <op-id> --watch polls to completion
    • database delete-snapshot <id> prompts, waits, confirms deletion
    • database rollback --target-time=5m refuses with --force hint while a game server is deployed
    • database rollback --target-time=5m --force --yes initiates a rollback and waits
  • Negative cases:
    • Running any command against an environment whose stack predates the database operations API surfaces the "not available" error with an upgrade hint
    • Creating the 6th manual snapshot surfaces the quota error
    • Two concurrent creates on the same shard surface a concurrency conflict
    • Bad --target-time value surfaces a usage error listing both accepted formats

tjorri added 10 commits April 10, 2026 20:38
Request (and its Get/Post/Put/Delete helpers) now returns a typed
*HTTPError on non-2xx responses, wrapping the status code, method,
URL, raw body, and parsed error message. Callers can use errors.As
to discriminate between 400/404/409/500 and produce user-friendly
messages. Existing callers that just bubble the error up keep
working unchanged.

The parser handles the {"error": "..."} JSON body shape used by the
backend APIs and falls back to the trimmed raw body when the shape
does not match or the body is not JSON.
The raw "Request failed with status code ...: <body>" Error-level log
was useful when callers received an opaque error and had no way to see
what went wrong. Now that Request returns a typed *HTTPError carrying
the status code, method, URL, raw body, and parsed server message,
callers can produce informative user-facing errors on top — and the
raw log becomes redundant noise when the caller does its job.

Demote the log to Debug level *only* when the response body parses
cleanly as the expected {"error": "..."} JSON shape. For opaque
bodies (non-standard JSON, plain text, HTML intercepted by a proxy,
etc.) keep the log at Error level so users still see the raw
diagnostic content even when the caller only produces a generic
wrapper error — this preserves backwards-compatible behavior for any
endpoint that does not conform to the structured error contract.

parseHTTPErrorMessage now returns (message, structured bool) so the
calling code can pick the log level. HTTPError's shape is unchanged;
structured and opaque errors both populate Message with the best
available summary of the failure.
Add Go types and client methods on *TargetEnvironment that wrap the
database operations API (cloud-managed snapshots and point-in-time
rollback). Covers all nine endpoints:

* GetDatabaseCapabilities
* GetDatabaseInfo
* ListDatabaseSnapshots (with type filter and per-shard limit)
* GetDatabaseSnapshot
* CreateDatabaseSnapshot
* DeleteDatabaseSnapshot
* RollbackDatabase
* ListDatabaseOperations
* GetDatabaseOperation

Includes a DatabaseOperation.IsTerminal helper and constants for
provider ids, snapshot types, operation types, and operation statuses.
Snapshot identifiers are path-escaped in URLs so values containing
colons (e.g. automated snapshot ids) round-trip cleanly.
Add three read-only commands for cloud-managed database operations on a
given environment:

* metaplay database info ENVIRONMENT
    Shows the managed database provider, per-shard cluster state, manual
    snapshot quota usage (e.g. '3 / 5'), and the point-in-time rollback
    window. Environments without a dedicated managed database cluster
    report as 'not supported' rather than erroring.

* metaplay database snapshot list ENVIRONMENT [--type] [--limit]
    Lists cloud-managed database snapshots across all shards, sorted
    newest first. Supports filtering by type (manual/automated/backup).

* metaplay database operation list ENVIRONMENT
    Lists in-progress database operations (snapshot creates/deletes and
    rollbacks). Useful for discovering work after CLI disconnects.

* metaplay database operation status ENVIRONMENT OPERATION_ID [--watch]
    Shows the current status of a single operation, optionally polling
    until the operation reaches a terminal state.

All commands support --format=text (default) or --format=json for
scripting. Adds a shared cmd/database_helpers.go with format validation,
HTTP-error-to-CLIError mapping (with context-aware hints for quota,
concurrency, and unsupported-environment cases), shard resolution for
--shard / --all-shards, target-time parsing for upcoming rollback, and
age/time rendering helpers. Unit tests cover the pure helpers.

New 'snapshot' and 'operation' subcommand groups coexist alongside the
existing flat 'database export-snapshot' / 'database import-snapshot'
commands — those operate on ad-hoc SQL dumps and are a different feature.
Add two mutating commands for cloud-managed database snapshots:

* metaplay database snapshot create ENVIRONMENT [--shard|--all-shards]
        [--name] [--description] [--no-wait]
    Creates a manual snapshot on one or more shards. By default waits
    and polls until the snapshot is available; --no-wait returns as
    soon as the request is accepted. --all-shards fans out creation
    across every shard in parallel via syncutil.ParallelMap.

* metaplay database snapshot delete ENVIRONMENT [SNAPSHOT_ID] [--yes]
        [--no-wait]
    Deletes a manual snapshot. In interactive mode, omitting the
    SNAPSHOT_ID opens a picker listing the environment's manual
    snapshots. Non-interactive mode requires both the id and --yes.
    Refuses to delete automated or backup-service snapshots.

Adds shared async infrastructure in database_helpers.go:

* waitForDatabaseOperation polls an operation id to terminal state,
  invoking an onStatusChange callback on each status transition.
* runShardOperation wraps a single-shard mutating call with logging,
  polling, and failure reporting, returning a shardOperationResult.
* aggregateShardResults joins per-shard results from a fan-out into
  a single CLIError listing the failing shards.
Add 'metaplay database rollback ENVIRONMENT [--shard|--all-shards]
--target-time=... [--force] [--yes] [--confirm-production]'.

The command rolls the environment's cloud-managed database back to a
previous point in time via the provider's native point-in-time recovery
mechanism. It is an in-place operation — no cluster delete/restore is
performed.

Features:

* --target-time accepts either an absolute RFC3339 timestamp or a
  relative duration (30m, 2h) meaning 'that long ago from now'.
* In interactive mode, if --target-time is omitted, the CLI prints the
  available rollback window for the selected shard(s) and prompts for
  a value.
* Warns (but does not reject) when the requested target time is outside
  the reported rollback window — the backend is the ultimate arbiter.
* Pre-flight check: refuses to run if a game server is currently
  deployed in the environment (split-brain risk). --force overrides
  with a loud warning.
* Production gate: requires --confirm-production for environments of
  type production, matching the 'database reset' convention.
* --all-shards fans out rollbacks across every shard in parallel.
* --no-wait returns as soon as the request is accepted instead of
  polling to completion.
Add a shared resolveEnvironmentForDatabaseOps helper used by every
'metaplay database ...' command as its first step. It resolves the
target environment, builds a TargetEnvironment, and probes the database
capabilities endpoint as a readiness check. Any error from the probe is
translated into a single clear "database operations are not available
for this environment" error with the underlying cause attached.

The capabilities endpoint is specifically designed as the "are database
operations supported?" probe, so treating any probe error as "not
available" gives consistent, actionable feedback regardless of how the
infrastructure stack fails to respond. This covers the real-world case
of older stacks running infrastructure versions that do not yet expose
the database operation endpoints at all — users now see a clear upgrade
hint instead of a cryptic protocol error.

Because we now have the capabilities data on hand, add a local
ensureShardsSupportCapability pre-flight check and use it to:

* reject 'snapshot create' on shards where SupportsSnapshots is false
* reject 'rollback' on shards where SupportsRollback is false

Single-shard failures name the specific shard in the error; multi-shard
fan-outs (--all-shards) report a summary listing which shards would be
rejected, so the user can pick a different --shard or upgrade the
stack before retrying.

Every command (info, snapshot list/create/delete, rollback, operation
list/status) was refactored to use the shared helper, removing per-
command env-resolution boilerplate and the prior ad-hoc capabilities
calls in info/create/rollback. The 400-not-supported string inference
in mapDatabaseHTTPError is no longer needed and has been removed — the
pre-flight probe makes it dead code.

New unit tests cover the uniform error-mapping behavior across success,
empty-shards, 404-empty-body, and 404-with-structured-body responses,
plus single-shard and multi-shard variants of the capability pre-check.
Add a METAPLAYCLI_STACKAPI_BASEURL environment variable that, when set,
overrides the StackAPI base URL normally constructed from each
environment's stack domain. Intended for local development against a
StackAPI running on the developer's machine (pointed at a real cluster
for tenant discovery and auth) — previously the only way to hit a local
StackAPI instance was to bypass the CLI entirely and curl it directly.

Mirrors the existing METAPLAYCLI_PORTAL_BASEURL override pattern in
pkg/common/config.go. When the override is active, NewTargetEnvironment
logs an info line so the user is never confused about which backend
their commands are targeting.
Snapshot create / delete and rollback can take multiple minutes, during
which the backend operation stays in a stable 'in-progress' status with
no new updates to render. Previously waitForDatabaseOperation only
logged on status transitions, so the CLI would appear hung until the
operation finished.

Add a periodic heartbeat emitted every 15 seconds while the status is
stable, rendered as a muted '… still <status> (elapsed 1m30s)' line
(and including a progress percentage if the backend ever starts
reporting one). The heartbeat clock resets on every genuine status
change, so there is no double-output when statuses transition.

Refactor 'database operation status --watch' to delegate polling to
waitForDatabaseOperation for a single source of truth, and change the
helper to accept the already-known initial operation so it can defer
its first poll by one interval. This eliminates both a redundant
round-trip and a duplicate render that --watch produced previously.
aggregateShardResults previously always wrapped per-shard errors in a
"X failed on N of M shards" summary. For single-shard commands (which
is the common case outside --all-shards fan-out) this produced awkward
output like "snapshot create failed on 1 of 1 shards" followed by a
nested shard-scoped error that already contained all the useful info.

Return the single shard's error directly when only one shard was
targeted. Multi-shard fan-outs keep the summary + details rendering
since that shape is genuinely more informative when partial failures
affect only some shards.
@tjorri
tjorri force-pushed the feature/database-operations-commands branch from f8edabb to 01d5115 Compare April 11, 2026 10:46
Some operation types (notably snapshot-delete) have no reliable source
for a creation timestamp on the backend — AWS retains no trace of the
delete once the snapshot is gone, and StackAPI is stateless so it
cannot record the initiation time itself. The backend now leaves
CreatedAt unset for those operations (see platform-apps fix), which
the Go value type marshals as the zero time.

Skip the 'created:' line in operation status output when CreatedAt
is zero, instead of printing the confusing '- (- ago)' placeholder
the formatters would otherwise produce.
@tjorri
tjorri requested a review from petrikero April 14, 2026 11:32
@tjorri
tjorri marked this pull request as ready for review April 14, 2026 11:32
tjorri added 2 commits April 14, 2026 15:55
…el commands

Rename 3-level nested commands to flat 2-level to match the rest of the
CLI (e.g. secrets list, secrets delete). The snapshot and operation
parent groups are removed; all commands register directly on databaseCmd.

  database snapshot list    → database list-snapshots
  database snapshot create  → database create-snapshot
  database snapshot delete  → database delete-snapshot
  database operation list   → database list-operations
  database operation status → database operation-status
The StackAPI moved its tenant-scoped database endpoints from
/v0/databases/<env>/* to /tenant/v1/<env>/databases/*, alongside
credentials, observability, and environment. Update the CLI client
to follow. No request/response shape changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant