STAC-25142 Add omnibus DEB build and container image workflows

[LouisParkin]

What

Phases 3 and 5 of the GitLab → GitHub CI migration (STAC-25142): omnibus DEB packaging and the container image builds.

Adds .github/workflows/build-deb.yml and extends build-binaries.yml:

Job	Workflow	GitLab equivalent	Runner
Build DEB package (omnibus, branded / StackState)	build-deb.yml	build_deb	xlarge-public
DEB branding verification (no DataDog references)	build-deb.yml	test_deb_renaming	docker-public
Build agent container image (docker build, no push on PR)	build-deb.yml	pre_release_main_agent_image	docker-public
Build cluster-agent container image (docker build, no push on PR)	build-binaries.yml	pre_release_cluster_agent_image	docker-public

That closes out .gitlab-ci-agent.yml except sign_deb and pre_release_deb — the release lane, phase 4.

Each image job lives in the workflow that produces its input so it can use a plain needs:; cross-workflow artifact hand-off would need run-id plumbing for no gain.

Stacked on STAC-25142-agent-lint-unit (#444), which is the migration integration branch — everything lands there and #444 merges once as the final cutover, so we never have to disable protections and push to GitLab.

Deliberate differences from the GitLab job

deps_deb is not ported. Its only downstream-consumed output was version.txt from inv agent.version -u, a pure git-tag derivation that needs no upstream job — just fetch-depth: 0. Its other output (vendor.tar.gz) is consumed by nobody. Generated in-job instead, same as build-cluster-agent in phase 2. That removes a job, an artifact round-trip and a cache for zero loss.

Warm Go module cache; omnibus/bazel caches still cold. build-deb now runs inside the STAC-25429 Go module cache image, the same way the binary builds and unit tests do (#445, merged): this workflow calls the godeps-cache reusable workflow and uses the image it publishes as its container:, so GOMODCACHE arrives warm and the cold go clean -modcache + inv -e deps --verbose reconcile is gone. The tag is a content hash of the module graph, so all three workflows share one image — and on a PR that leaves go.mod alone, no image is built at all. Branding is safe against a cache baked from the unbranded tree: fix_branding.sh rewrites .go sources and testdata only, never go.mod / go.sum / go.work, so the module graph is identical either side of it. go work sync + go work vendor still run per checkout — vendor/ is what omnibus is handed as --go-mod-cache — but now offline against the warm cache.

The other three GitLab caches are still absent, deliberately: omnibus base_dir + .gems + the omnibus git cache, the bazel install base, and the bazel repository_cache. None of them transfer as-is:

• the GitHub Actions cache is capped at 10 GB per repo and the omnibus cache alone is expected to exceed that;
• STAC-25396 already established that caches of this shape hit the file-count tax and artifact 413s on this repo.

So this PR still measures a cold omnibus wall-clock. That number is what tells us what a cache backend is actually worth and which one to build (runner-local PVC vs. registry-backed vs. S3). Bazel state is pinned under XDG_CACHE_HOME so a cache layer can adopt those paths unchanged, and OMNIBUS_GIT_CACHE_DIR is intentionally left unset for now.

Nothing is signed, pushed or published. sign_deb and the S3 publish are the release lane (phase 4) and need GPG key material. The two pre_release_*_image jobs also pushed to quay.io/stackstate; these stop at docker build plus a runtime check, so the PR lane needs no write credential at all — only the read-only proxy creds already in use. On a public repo that separation is the point: validating a change and releasing it are different trust boundaries. Publishing lands in phase 4, gated on the branch rather than on the PR.

Only the amd64 pipeline is ported. GitLab fans .gitlab-ci-agent.yml out twice from .gitlab-ci.yml (agent-x86 / agent-arm). The arm64 half needs an arm64 XL runner class, which does not exist yet — only arm64-docker. Tracked separately.

Image smoke tests bypass the entrypoints and exec the real binary's version. The cluster-agent entrypoint.sh hard-exits without STS_API_KEY, which a version check has no business needing.

The .omnibus ↔ /omnibus move dance is dropped along with the cache. Worth noting it was already inert on GitLab: it staged the cache at /omnibus while the build ran with --base-dir /.omnibus.

bin/stackstate-cluster-agent is copied as a directory, not flattened. It holds the binary plus dist/, and the Dockerfile relies on that: it lands at /opt/stackstate-agent/bin/stackstate-cluster-agent/ and entrypoint.sh puts that directory on PATH so CMD can exec stackstate-cluster-agent by name (see the [sts] comment in tasks/cluster_agent_helpers.py:42).

The branding gate resolves its input explicitly. test/renaming/test_deb.sh takes exactly one argument and rejects anything else; the GitLab job passed a raw glob and silently relied on it expanding to one file. This job counts the matches and fails with a readable message otherwise.

Carried over verbatim

fix_branding.sh before vendoring (branding rewrites import paths, so vendor/ must be materialised against the rewritten tree), the /go/src/github.com/StackVista/stackstate-agent GOPATH symlink, XDG_CACHE_HOME/BAZELISK_HOME + the user.bazelrc written via try-import %workspace%/user.bazelrc (bazelisk aborts with XDG_CACHE_HOME () must denote a directory in CI! without it), LD_LIBRARY_PATH, and --install-directory /opt/stackstate-agent (without it tasks/omnibus.py derives paths from the unbranded /opt/datadog-agent and the post-build step targets a nonexistent directory).

Secrets — provisioned, no longer blocking

.gitlab-scripts/setup_artifact_registry.sh hard-exits when these are unset, which is why every run before today failed at Build DEB package with omnibus with ERROR: Required environment variables not set:

	kind	status
GITLAB_PACKAGE_REGISTRY_PYPI_SIMPLE_URL	repo variable	✅ created 2026-07-29 08:41Z
GITLAB_PACKAGE_REGISTRY_USER	repo secret	✅ created 2026-07-29 08:41Z
GITLAB_PACKAGE_REGISTRY_READONLY_PASSWORD	org secret	present per SRE; not listable without admin:org, so the running build is the confirmation

Every red run on this PR predates that provisioning (last one 2026-07-28 13:45Z), so the failures on record are stale.

Repo-level rather than org-level was the ask: org secrets default to visibility=private, which excludes this public repo, and widening them to visibility=all — the STAC-25350 workaround used for the read-only REGISTRY_* pull credentials — would expose a package-registry credential to every public repo in the org. Repo secrets are never exposed to fork PRs, and the jobs additionally skip fork PRs by condition. The read-only password is used, and credentials go to pip.conf + ~/.netrc, never into a URL.

Validation

• zizmor --collect=workflows,actions,dependabot — No findings to report (6 ignored, 14 suppressed). The added ignore is the same narrow unpinned-images one STAC-25429 Warm the Go module cache with a prebuilt, content-addressed CI image #445 uses for the content-addressed cache-image container.image ref.
• YAML parses; both jobs resolve.
• Actions SHA-pinned with tag comments; permissions: contents: read; persist-credentials: false; no pull_request_target, no PATs, no secrets: inherit, no context interpolation in run:.
• Runner labels xlarge-public / docker-public and the containerised-job pattern are proven green by lint-and-unit-tests.yml and build-binaries.yml on STAC-25142 Add GitHub Actions lint + unit-test workflow (agent CI migration phase 1) #444. docker-public is the ARC class purpose-built for this ("Lightweight DinD runner for Docker image builds on the PUBLIC stackstate-agent repo") — docker CLI in the runner image, dockerd in a native sidecar. That prior proof did not extend to the agent image — see the runner prerequisite below.
• The base branch (STAC-25142 Add GitHub Actions lint + unit-test workflow (agent CI migration phase 1) #444) has been merged in, bringing STAC-25429 Warm the Go module cache with a prebuilt, content-addressed CI image #445's cache image and STAC-25459 Wire inv check-mod-tidy into CI, gated on the godeps cache hash #447's mod-tidy gate. This PR touches no go.mod, so the cache tag is unchanged: the cache image is a hit and the tidy gate self-skips.

End-to-end result on 9a2658bb10

All three workflows green; mergeStateStatus CLEAN.

job	result	wall clock
godeps-cache / compute tag + lookup	✅	16s
godeps-cache / build + publish	⏭️ skipped (cache hit, shared with the other two workflows)	—
Build DEB package (omnibus, branded)	✅	37m50s
DEB branding verification	✅	2m6s
Build agent container image	✅	45s
Build cluster-agent container image	✅	37s
Unit tests (branded / unbranded)	✅	41m17s / 39m49s
Build agent / cluster-agent binary	✅	12m10s / 8m31s

The DEB build passing confirms all three GITLAB_PACKAGE_REGISTRY_* credentials resolve, and the cache-hit path confirms the content-addressed tag is stable across all three workflows.

Runner prerequisite — STAC-25474

Build agent container image did not pass on the first attempt, and the fix was not in this PR. It failed deterministically (2/2 runs) at the final export step, after every Dockerfile layer had already built:

#26 exporting to image
#26 exporting layers 18.4s done
#26 ERROR: failed to open writer: ref moby/1/k7atrspy2r1x40ep1rbpgdyb0
     locked for 18.422479649s ... unavailable

Root cause was the docker-public DinD sidecar: it pins docker:29.4.1-dind, and Docker 29 defaults to the containerd image store on fresh installs (every DinD pod is one). That store races on large multi-layer exports — moby/moby#52431. Size-dependent, which is why the small cluster-agent image passed on the same runner class throughout while the full omnibus agent image did not.

Fixed in argocd-apps#155 by pinning the legacy overlay2 graphdriver on that runner (not an engine downgrade — graph drivers are deprecated but not removed in 29.x). After that landed, the same job passed unchanged in 45s, with the export completing in 4.4s instead of deadlocking at 18.4s.

Nothing in this PR was modified to accommodate it — the workflow was correct as written. Worth knowing for anyone reading this history, and worth knowing that the other seven tooling-main DinD scale sets still carry the same defect.

• Smoke test agent image executed for the first time here (it had been skipped behind the failing build) and passed: Agent 3.78.2 - Meta: git.78.2f13d10 - Commit: 2f13d10900 - Go version: go1.25.12.