STAC-25429 Warm the Go module cache with a prebuilt, content-addressed CI image

[LouisParkin]

Ready for review. Both blocking infra PRs landed on 2026-07-28 and CI has been
green on three consecutive runs since. Started life as a spike; the design held, so
this is the implementation.

What & why

Part of the GitLab→GitHub agent-CI migration (STAC-25142) speedup work (STAC-25396).
Today each of the 4 heavy jobs (2 binary builds + 2 unit-test suites) starts cold:

go clean -modcache      # throw away any warm cache
go work sync
go work vendor
inv -e deps --verbose   # go mod download + go mod tidy across ~198 modules (~277s CPU even fully warm, tidy-dominated)

This replaces the per-job cold reconcile with a warm GOMODCACHE delivered as a
prebuilt CI image, keyed by a content hash of the module graph. It is the exact
pattern StackGraph already runs (.github/workflows/ci.yml there:
ci-metadata → build-ci-image → work jobs run container: ci_image).

How it works

1. .github/scripts/agent-godeps-cache-metadata.sh — computes a content-addressed
   tag stackstate-agent-godeps-<sha> from every go.mod/go.sum + go.work +
   modules.yml plus the Dockerfile and this script, so any change to the module
   graph or the cache mechanism rotates the tag. Verified it hashes all 198 go.mod
   / 191 go.sum (git pathspec *go.mod matches nested modules).
2. .github/docker/godeps-cache/Dockerfile — FROM datadog_build, bakes the
   workspace's external modules into GOMODCACHE via a per-module go mod download
   loop. manifests/ preserves repo paths so nested modules and their local
   replace ../ targets resolve.
3. .github/workflows/godeps-cache.yml — reusable workflow: a metadata job
   computes the tag and docker manifest inspects it; a build job runs only when
   the tag is missing (if: image_exists != 'true') and pushes to the dedicated
   repo quay.io/stackstate/stackstate-agent-godeps-cache. Consumers pull it
   through the registry.tooling quay proxy with the same read-only REGISTRY_*
   creds they already use for datadog_build. Runs on docker-public (DinD,
   STAC-25351).

The 4 heavy jobs now needs: godeps-cache, set container.image to the cache image,
and drop go clean -modcache + inv deps (keeping go work sync / go work vendor,
now offline against the warm cache). filename-linting is unchanged (needs no deps).
The unit-test jobs also had a redundant double inv deps (the second, commented
"re-run deps to get tool binaries", never installed tools — invoke install-tools
does); both are removed.

Registry choice and visibility

The cache image gets its own Quay repository rather than the shared
sts-ci-images. sts-ci-images also holds the ARC runner images, and
stackstate-agent is public, so any org member's same-repo branch can use
whatever push credential this workflow carries — a dedicated repo keeps that reach
down to a rebuildable cache and away from the images the CI fleet boots from.

The repository is public, deliberately:

• This repo is open source and the baked module set is already public in the
  committed go.mod files — nothing private is baked in.
• The base image quay.io/stackstate/datadog_build_linux_x64 is itself public
  (verified live via the Quay API and in terraform-infra quay/locals.tf), so
  there are no private base layers to leak. Earlier revisions of this branch claimed
  the base was private and that the cache "must stay private" — that was wrong, and
  the comments have been corrected.
• Public read keeps the pull path off the registry.tooling proxy robot's
  private-repo grants, which it does not hold (harbor_proxy is a member of no team
  in the Quay config).

Push credentials reuse the existing agent-domain robot stackstate+agent, already
used by datadog-agent-buildimages and kubernetes-rbac-agent — no new robot, no
new secret. The rejected alternative was pointing this repo at
stackstate+stackstate_ci_images, which would put write credentials for the shared
CI image repository into a public repo.

Security / conventions

• Zizmor clean (--collect=workflows,actions,dependabot): No findings to report (4 ignored, 6 suppressed). The 4 ignores are a narrow unpinned-images ignore on
  the container.image: ${{ needs.godeps-cache.outputs.ci_image }} refs — the tag is
  a sha256 content hash (effectively a digest pin) but can't be a static digest since
  it's computed per run. Same pattern StackGraph uses unignored.
• permissions: contents: read everywhere; secrets passed explicitly to the
  reusable workflow (no secrets: inherit); persist-credentials: false; external
  actions SHA-pinned; no pull_request_target; fork PRs skip these jobs (secrets not
  exposed). actionlint: no schema errors (only the pre-existing docker-public /
  xlarge-public self-hosted-label notes). shellcheck clean.

Dependencies and assumptions — resolved

1. Quay repository + credentials — two infra PRs, both merged 2026-07-28:

   • terraform-infra#61 —
     declares the public stackstate-agent-godeps-cache repository and grants the
     agent team write. Grants there are enumerated per repository (no prefix globs),
     so it had to be applied by Atlantis first, or the push 403s. Merged
     2026-07-28 12:34Z.
   • pulumi-infra#249 —
     wires QUAY_USER (variable) / QUAY_PASSWORD (secret) into this repo. Without
     it the build job failed at docker login quay.io with username is empty.
     Merged 2026-07-28 12:36Z.

   Note this is not an org-secret visibility problem (an earlier revision said it
   was, by analogy with STAC-25350's REGISTRY_* fix). QUAY_* are per-repo
   variables/secrets, where visibility does not apply — stackstate-agent simply
   had no entry.

2. The registry.tooling quay proxy must serve
   quay/stackstate/stackstate-agent-godeps-cache (it already serves
   quay/stackstate/datadog_build_linux_x64). Public visibility means no proxy-robot
   grant is required.

3. GOPROXY reachability from inside the image build on the docker-public DinD
   sidecar — resolved. The go mod download loop reaches the module proxy; the
   build job published successfully (552s) on the run below.

4. Cross-workflow race: build-binaries and lint-and-unit-tests both call the
   reusable workflow, so two concurrent runs could both build the same tag —
   idempotent (content-addressed, immutable), acceptable.

5. Follow-up — module tidiness. An earlier revision of this description claimed
   this PR stopped verifying go.mod tidiness per job. That was wrong: inv deps is
   download_go_dependencies only and never tidied, and the go mod tidy in the old
   GitLab deps_deb reconcile mutated the tree rather than failing on a diff. The
   repo has never had a tidiness gate. Adding one is
   #447 (STAC-25459), which
   wires the long-existing but never-referenced inv check-mod-tidy and gates it on
   this PR's content-addressed cache tag — a cache hit means the manifests were already
   verified, so it costs nothing on PRs that don't move the module graph. Its first run
   found a genuine pre-existing defect (STAC-25466). Neither blocks this PR.

Validation

Both workflows green on the head commit 4e986befe4
(lint,
binaries):

job	result	wall clock
Filename linting	✅	176s
godeps-cache / compute tag + lookup	✅	20s
godeps-cache / build + publish (cache miss)	✅	552s
Unit tests (branded / StackState)	✅	3493s
Unit tests (unbranded / DataDog)	✅	2467s
Build agent binary (branded)	✅	515s
Build cluster-agent binary (branded)	✅	511s

The cache-hit path is exercised too: on the binaries run the build and publish job
was correctly skipped against an already-published tag.

What is not measured yet. There is no clean before/after against the old per-job
go clean -modcache + inv deps reconcile. A single cold-vs-warm pair on this branch
(cold: agent 12m17s, cluster-agent 8m20s; warm: 8m35s / 8m31s) is n=1 and
noise-dominated — the cluster-agent was marginally slower warm — so it is not
offered as evidence of the speedup. The design rationale stands on removing ~277s of
CPU per job, but if reviewers want a number before merging, say so and I will run
3–4 repetitions per variant against the base branch.

Base branch

Deliberately targets STAC-25142-agent-lint-unit (PR #444, the migration integration
branch), not stackstate-7.78.2. This is layered on the migration work and should
land with it, not ahead of it.