Decide & spec the rollout smoke-test (did a codeq change reach all users?)

[Kenny-Heitritter]

Part of #16

Question

Now that layer-1 auto-rebuild is locked (#19) and SHA/build-id stamping is specced as its Phase 0, how does an operator confirm a given codeq change actually reached all Lab users on an env — using the stamped SHA/build-id — without shelling into individual pods?

Decide the validation method and spec it:

• What to check and where the truth lives: manifest.json sha/build_id, codeq --version, and/or the lab-base image label/digest.
• Pass/fail criterion for "the change is live for all users on this env" — accounting for layer 2 (existing pods converge passively per Decide running-pod refresh policy (layer 2) #21, so "all users" has a convergence window, not an instant).
• Who runs it and when: e.g. a post-rebuild Cloud Build assertion step vs an operator one-liner vs a scheduled check.
• Cover staging (auto) and prod (gated).

HITL. Graduated from the map's "Rollout validation / smoke-test" fog now that #19 fixed the mechanism and folded in SHA-stamping. Output: chosen smoke-test method + an execution-ready spec.

[Kenny-Heitritter]

Resolution — rollout smoke-test: DECIDED & SPECCED

Resolves #22. Decided via /grilling with @Kenny-Heitritter. Builds on #19 (SHA/build-id stamping = Phase 0) and #21 (layer-2 = passive convergence). Cloud Build structure verified against the live build-lab-base-prod definition in qbraid-infrastructure (terraform/environments/prod/gcp/cloud-build-qbraid-dstacks.tf, main) — the :latest push is a trailing images = [...] block, i.e. deferred until every step passes.

Decision (headline)

The smoke-test asserts one claim: lab-base:latest carries the expected codeq SHA. That is what "the change reached all users on env X" resolves to — the image is the on-ramp every pod draws from. Layer-2 running-pod convergence is by-construction on restart (correct image → correct binary + re-asserted config, per #21/#20) and is deliberately not measured or gated. Staging and prod use the identical checks — only bucket/trigger/registry names differ.

What to check & where the truth lives

Side	Source	Meaning
Expected	sha in gs://qbraid-codeq[-staging]/latest/manifest.json	hop-1 output (the sha/build_id keys added by #19 Phase 0)
Observed	codeq --version on the lab-base:latest binary	what hop 2 actually baked
Pass	observed.sha == expected.sha	the bake wired up the right binary

• Pass key is sha; build_id logged for traceability, not a gate.
• No image OCI label / digest pinning. Keeps Decide & lock the layer-1 rebuild mechanism #19 Phase 0 (codeq --version + manifest.json) as the only stamping surface; codeq --version read from inside the image is the authoritative signal.

The two checks (identical both envs)

A. Automated gate — assert-codeq-sha step in build-lab-base[-prod] (fail-closed, pre-push)

• Add a final step to google_cloudbuild_trigger.qbraid_dstacks_lab_base (staging + prod), wait_for the build/tag-standard steps, running before the deferred images = ["…/lab-base:latest", "…/lab-standard:latest"] push.
• Logic: gsutil cat gs://qbraid-codeq[-staging]/latest/manifest.json → expected sha; docker run --rm <local lab-base:latest> codeq --version → observed sha; mismatch → non-zero exit.
• Because the images: push is deferred until every step passes, a failed assertion aborts the build before the push — :latest is never poisoned and stays on the last-good image. No restructuring into an explicit push step is needed.
• Extend the existing download-artifacts step (or the assert step) to also fetch …/latest/manifest.json (today it pulls only the binary).
• Net effect: the smoke-test is a fail-closed guard on Decide & lock the layer-1 rebuild mechanism #19's auto-rebuild — staging's auto-rebuild cannot publish a mis-baked image, and the same guard runs on every prod rebuild.

B. Operator one-liner — verify-codeq-rollout.sh <env> (on-demand, no pod shell)

• Pulls the live lab-base:latest from Artifact Registry, runs codeq --version, compares to the bucket manifest.json sha. Registry-read auth only — no kubectl exec, no pod access (satisfies the ticket's hard constraint).
• Answers "is env X's live image actually on SHA Y?" — the prod gated go-live confirmation step (Decide & lock the layer-1 rebuild mechanism #19 Phase 2 checklist item 2) and the universal spot-check.
• Documented as a one-liner / small script in CODEQ_DEPLOYMENT.md.

Pass/fail criterion — "the change is live for all users on env X"

• PASS ⟺ lab-base:latest reports expected.sha — check A green on the rebuild, or check B green on demand. At that point every new/restarted pod runs the correct binary + re-asserted config; already-running pods converge over the Decide running-pod refresh policy (layer 2) #21 window with no further action.
• FAIL ⟺ sha mismatch → (check A) the build aborts and :latest is untouched → investigate a stale/raced gsutil cp, broken stamping, or wrong bucket.

The convergence window is not part of pass/fail: "live for all users" means the image source is correct, not that every running pod has already recycled.

Explicitly rejected

• Scheduled drift cron — staging's in-build gate + auto-rebuild close the drift window; for prod an un-baked binary is the intended gated state (Decide & lock the layer-1 rebuild mechanism #19 Phase 2), not an alarmable error.
• Active stale-pod enumeration / image-digest sweep — running pods self-correct on restart (correct image + re-asserted config), so knowing which pods are still stale adds nothing to the rollout decision.
• Image label / digest as a SHA carrier — redundant with codeq --version, and expands Decide & lock the layer-1 rebuild mechanism #19 Phase 0.

Who runs it / when

• A runs automatically inside every build-lab-base[-prod] — staging on the Decide & lock the layer-1 rebuild mechanism #19 auto-rebuild, prod on each manual/gated rebuild. (The in-build gate is live in both envs immediately; only the prod auto-firing stays gated per Decide & lock the layer-1 rebuild mechanism #19 — independent of the check.)
• B run by an operator on demand — a mandatory tick on the prod go-live checklist, available for any spot-check.

Feeds

• CODEQ_DEPLOYMENT.md (execution effort): document check B's one-liner, the "image-correct ⟹ live" definition, and the pass/fail criterion above.
• Satisfies Decide & lock the layer-1 rebuild mechanism #19 Phase-2 prod-go-live checklist item 2 ("rollout smoke-test exists") at the planning level.

Output: chosen method (in-build fail-closed SHA assertion + operator registry-side one-liner) + the execution-ready spec above — no open questions for the implementer.