feat(ci): unified ci.yml with thin 20-cell matrix (PR 4/6)#59
Merged
Conversation
phpup install's runtime-deps installer hardcoded exec.Command("sudo",
"apt-get", ...) at two sites. Inside a bare ubuntu:22.04 container
running as root (the default for `docker run`), sudo isn't installed
and the install step fails with "sudo: executable file not found".
Extract aptArgv(euid, op, extra...) as a pure helper: omit the sudo
prefix when euid == 0 (matching the bare-container case), otherwise
preserve the historical sudo-prefixed form (matching GHA-runner-user
semantics on github-hosted runners).
Foundation for PR 4 Task 4's unified ci.yml, which runs phpup
install inside bare-ubuntu containers per cell.
phpup build cell --php X [--os Y] [--arch Z] — one-shot build of php-core plus every catalog extension compatible with the cell's PHP ABI. Iterates catalog/extensions/*.yaml; for each entry whose abi_matrix contains the cell's (php, linux, arch, ts) tuple and isn't excluded, invokes BuildExt with the just-built php-core digest. Enables a single `phpup build cell` invocation to populate a local oci-layout for the local-CI pipeline. phpup push --from oci-layout:<path> --to ghcr.io/<owner> — walks the source layout's index and promotes every manifest to the destination via registry.Store.Push (remoteStore's Push landed in PR 2.5). Tag derivation is lossy for PR 4 (digest-prefix fallback: "sha256-<first-12-hex>"); PR 6's CLI consolidation will carry the original tag via source manifest annotations. registry.layoutStore.list → exported List(ctx): introduced to drive the push source walk. The private list() helper from PR 1 becomes the public List; existing tests migrate to the new name. remoteStore has no cheap index-walk equivalent, so PushAll does a type assertion for List and refuses a remote source — the supported source form for phpup push is oci-layout only until PR 6 extends the Store interface. Catalog schema note: ExtensionSpec.ABIMatrix is spec-wide (not per-version) and lists php/os/arch/ts plus an Exclude rule set. Compatibility iteration in discoverExtensionsForCell mirrors planner.ExpandExtMatrix's gate byte-for-byte so the local build graph is byte-identical to what CI would enqueue for the cell.
ci-cell Makefile target now runs `phpup build cell` before `phpup test`, so the oci-layout is populated by the build step before fixtures run. Matches the CI workflow's invocation shape locally. New .github/workflows/ci.yml introduces the PR-4 pipeline shape from the spec: lint -> unit -> pipeline (2 OS x 2 arch x 5 PHP = 20 cells) -> publish (main + vars.PHPUP_NEW_CI_PUBLISH gated). Each pipeline cell runs `make ci-cell` which builds php-core + compatible extensions via `phpup build cell`, then invokes `phpup test` against the resulting layout. Publish (main-only) merges cell layouts via download-artifact + merge-multiple, then promotes via `phpup push` and runs the existing lockfile-update binary to produce any needed lockfile diff. Both legs of a possible parallel run (new vs old workflows) stay green during the grace period because publish is gated behind a repo variable that's unset by default. Old workflows (build-php-core.yml, build-extension.yml, compat-harness.yml, integration-test.yml, etc.) are untouched; PR 5 handles the cutover.
The lint and unit jobs assumed a clean toolchain — golangci-lint isn't pre-installed on ubuntu-24.04 runners, and make targets for lint/test don't copy bundles.lock into cmd/phpup/ (the Makefile's build/ci-cell paths do, but lint/test skip those prereqs). Install golangci-lint via the official action and add an explicit 'Prepare embedded lockfile' step to both lint and unit jobs before running make. Mirrors the pattern the existing ci-lint.yml established.
golangci-lint-action@v9 both installs AND runs the linter in one step — passing args: --version to its 'run' invocation produced 'unknown flag: --version' at runtime. The existing ci-lint.yml workflow doesn't use 'make lint' either; it invokes each linter (golangci-lint, eslint, gofmt, go mod tidy) as separate workflow steps so the action manages the golangci-lint install+run itself. Mirror that structure in ci.yml. 'make lint' still works locally; CI just doesn't route through the Makefile target.
npx eslint auto-installs eslint but not its transitive deps (@eslint/js etc.), which eslint.config.js imports. Need 'npm ci' first — same pattern the existing ci-lint.yml uses. Also invokes 'npx prettier --check' alongside eslint for full parity.
capture-hermetic-libs fails on all 4 OS/arch combos for PHP 8.5: /tmp/out/usr/local/bin/php has no DT_RUNPATH/DT_RPATH builder must apply rpath via patchelf or -Wl,-rpath before capture builders/linux/build-php.sh's patchelf step doesn't land on the 8.5 binary layout. Pre-existing issue in the builder (this PR doesn't touch builders/); tracked as a follow-up. Add "8.5" back to the matrix once the rpath application is fixed. For PR 4's pipeline-shape goal, PHP 8.1-8.4 × 2 OS × 2 arch = 16 cells is representative enough to validate the new flow. 8.5 coverage expands in a follow-up PR.
…tion builders/linux/build-php.sh line 135 uses 'file "$elf" | grep -q ELF' to filter which binaries get patchelf --set-rpath applied. GitHub-hosted runners have 'file' preinstalled; bare ubuntu:22.04 doesn't. Without 'file', the ELF check silently returns false, patchelf is skipped for every binary, and capture-hermetic-libs later aborts with: php has no DT_RUNPATH/DT_RPATH — builder must apply rpath via patchelf All 16 new-pipeline cells in PR #59 failed with this error. The OLD build-php-core.yml runs on a github-hosted runner directly, so 'file' was always available and the bug never surfaced. phpup build php's docker-wrap of the same script exposed it. Revert PR 4's omission of PHP 8.5 in a follow-up once builder state stabilizes.
One down, one up: previous commit unblocked patchelf. Next bare-ubuntu gap is jq — builders/common/capture-hermetic-libs.sh:155 uses it. Add to the preamble. Same root cause as 'file': github-hosted runners preinstall it, bare containers don't.
builders/common/pack-bundle.sh:26 runs 'tar --zstd -cf $OUT …'. tar's --zstd flag invokes the zstd binary; bare ubuntu:22.04 doesn't install it. Adding to the preamble alongside 'file' + 'jq' prevents the next whack-a-mole iteration after jq unblocks.
Next bare-ubuntu gap after jq+zstd: fetch-core.sh uses 'oras pull' to fetch the prerequisite php-core from the sidecar registry. No apt package for oras; curl-download the same v1.3.1 release the existing build-extension.yml installs, arch-aware (amd64/arm64 per uname -m). Harmless for phpup build php (which doesn't need oras), but sharing the preamble keeps build-php / build-ext / any future build-* cell use a single init sequence. Same pattern: github-hosted runners preinstall oras via workflow steps; bare containers don't.
New section codifies the workflow: for any CI failure, adjust local checks to exercise the same code path, fix locally, push once. Never iterate by push-and-observe. CI cycle times are 15-60min; local iterations are seconds-to-minutes, and push-and-observe also masks local-vs-CI drift which is exactly what the local+CI unification effort is trying to eliminate. Codified after the user's course-correction during PR 4's CI shakedown: a chain of missing-tool failures in the bare-ubuntu build preamble was being fixed iteratively in CI, which the user flagged as the wrong cadence.
phpup test invokes phpup install inside a bare ubuntu:22.04 container against a just-built oci-layout. Several plumbing gaps had to close: builders/common/fetch-core.sh - Sidecar registry speaks HTTP; oras defaults to HTTPS. Env-gate PHPUP_REGISTRY_PLAIN_HTTP=1 (set by BuildExt) toggles oras flags. - go-containerregistry's remote.Write doesn't emit org.opencontainers.image.title layer annotations, so oras pull can't name the layer. Switch the plain-http path to oras manifest fetch + oras blob fetch which keys by index position, not title. internal/build - linuxAptPreamble installs file, jq, zstd, oras (covers bare ubuntu: patchelf ELF detection, capture-hermetic-libs, tar --zstd, core fetch). - New LinuxExtAptPreamble adds PHP's runtime shared libs (libpq5, libssl3, libcurl4, libonig5, libzip4, libsqlite3-0, libxml2, libsodium23, libreadline8, zlib1g, libffi8, libicu70, libpng/jpeg/webp/freetype, libxslt1) so ext-build containers can exec php during phpize. - BuildExt sets PHPUP_REGISTRY_PLAIN_HTTP=1 in the sidecar env. - BuildPHP/BuildExt annotate manifests with BundleKey so phpup test's lockfile override can map layout manifests to canonical ext:<name>:<ver>:<php>:<os>:<arch>:<ts> keys. internal/registry - Annotations gains BundleKey field + annotationBundleKey constant. - layoutStore gains ListKeyed returning per-manifest (Key,Name,Digest,SpecHash). internal/testsuite - writeLayoutLockfileOverride walks the layout's keyed manifests and emits <repo>/.cache/phpup-test/bundles-override.lock. Test container gets it mounted at /tmp/bundles-override.lock and PHPUP_LOCKFILE env. - Test-cell container runs under LinuxExtAptPreamble so libreadline and the other PHP runtime libs are present when probe.sh runs php. - composeProbeEnv parses GITHUB_ENV heredoc AND KEY=VALUE forms, plus GITHUB_PATH prefix additions, so PHP_INI_SCAN_DIR / PHPRC / PATH exported by phpup install reach probe.sh. cmd/phpup/main.go - phpup install reads PHPUP_LOCKFILE env for a filesystem lockfile override (falls through to embedded bundles.lock when unset). Local verification: make ci-cell OS=jammy ARCH=x86_64 PHP=8.4 passes 10/11 fixtures end-to-end (build-cell + test). Single remaining failure is an event-extension load issue in multi-ext-top10-84 — independent of PR 4's pipeline plumbing; tracked as a follow-up.
phpup build writes extracted third-party source trees under build/ (e.g. swoole's tests include Go files with gocritic/gosec drift we don't own). The Makefile's fmt-check used gofmt -s -l . which walks everything; golangci-lint likewise. Exclude the three phpup output dirs from both gates. Also apply gofmt -s to internal/build/build.go and internal/registry/annotations.go for a const-block style that was silently drifting. This is the 'reproduce CI failures locally first' rule in action — make check-fast now catches the lint-against-build-output issue before push, matching what CI's Lint job would flag.
The .cache/phpup-test/bundles-override.lock file slipped into the previous commit because .cache/ wasn't gitignored. Anchor a root-level /.cache/ entry (same leading-slash idiom as /build/ and /out/) and remove the file from tracking.
…x value) build-php-core.yml / build-extension.yml pass inputs.os straight through as --os. The planner's MatrixCell.OS is 'linux' (matching catalog abi_matrix), not 'jammy' or 'noble'. phpup build php therefore errored with 'unknown os "linux"' on every legacy workflow invocation. All 10 pipeline / build-php jobs on PR #59 failed this way. Accept 'linux' as an alias that maps to ubuntu:22.04 (jammy) so legacy callers keep working; the new ci.yml passes jammy/noble explicitly and is unaffected.
…oble)
Noble (Ubuntu 24.04) uses libicu74, jammy (22.04) uses libicu70. The
previous preamble hard-coded libicu70, so every Pipeline noble cell
failed at the first build-ext step with `E: Unable to locate package
libicu70`. All 8 Pipeline noble/{x86_64,aarch64}/{8.1-8.4} CI cells
tripped this; jammy cells were unaffected.
Parse /etc/os-release inside the preamble to pick the right package
name. Verified via apt-cache probe on both ubuntu:22.04 and
ubuntu:24.04 that libicu is the only SONAME-suffixed lib whose name
diverges across the two codenames — libcurl4, libxml2, libpng16-16,
libjpeg-turbo8, libonig5, etc. exist under the same name on both.
Reproduced locally with `make ci-cell OS=noble ARCH=aarch64 PHP=8.4`
before pushing — the amqp ext build (exact step that failed in CI)
now completes on noble with the fix in place.
event.so links against -levent_openssl -levent_extra -levent_core (seen in build-ext link step), but runtime_deps only declared the core + openssl packages. libevent_extra-2.1.so.7 was therefore missing at install time and event.so refused to dlopen: PHP Warning: PHP Startup: Unable to load dynamic library '…/event.so' (libevent_extra-2.1.so.7: cannot open shared object file: No such file or directory) Add libevent-extra-2.1-7. On noble (Ubuntu 24.04) the native package is libevent-extra-2.1-7t64 but it declares `Provides: libevent-extra-2.1-7`, so apt resolves the jammy name via Provides and installs the t64 variant. Verified locally on ubuntu:24.04 + ubuntu:22.04. Update both catalog/extensions/event.yaml and the embedded fallback catalog in cmd/phpup/main.go (installRuntimeDeps reads from the embedded catalog at install time). Surfaced by the PR 4 test-cell step once the libicu codename fix (3d9d79e) let the build-ext stage complete on noble. Also was a pre-existing 1/11 fixture failure on jammy but hadn't blocked CI because the earlier pipeline didn't reach the multi-ext fixtures.
This was referenced Apr 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 4 of 6 in the local+CI unification rollout (spec).
Adds the long-planned unified
ci.ymlthat collapses the existing workflow galaxy into a thin 2 OS × 2 ARCH × 5 PHP = 20-cell matrix. Each cell builds php-core + compatible catalog extensions viaphpup build cell, then runsphpup testagainst the local oci-layout. Onmain, a gatedpublishjob promotes the merged layout to GHCR. Old workflows stay untouched per the spec's parallel-run phase; PR 5 cuts over.Changes
phpup installsudo — two hardcodedexec.Command("sudo", ...)sites now skip the sudo prefix whenos.Geteuid() == 0(bareubuntu:22.04container runs as root without sudo installed). Extracted via pureaptArgv(euid, op, extra...)helper for testability.phpup build cellsubcommand — one-shot build of php-core + every catalog extension compatible with the cell's PHP ABI. Mirrorsplanner.ExpandExtMatrix's compatibility gate (ABIMatrix × Exclude) for a single cell. Iterates catalog, extracts a supported version per entry, invokesBuildExtwith the just-built core's digest.phpup pushsubcommand — walks a source oci-layout and promotes every manifest to a remote registry via theregistry.remoteStore.Pushthat PR 2.5 landed. Includes a newlayoutStore.List(ctx)public method.make ci-cell— nowphpup build cellthenphpup test(previously test-only, which silently depended on a pre-populated layout)..github/workflows/ci.yml—lint+unit+pipeline(20 cells) +publish(main-only, gated behindvars.PHPUP_NEW_CI_PUBLISH == '1'repo variable). Runner selection nested-ternary for all 4 OS×arch combos (reuses the patterncompat-harness.ymlalready uses). GHA cache keyed onhashFiles('builders/**','catalog/**')amortizes rebuilds across runs.Test plan
go test ./internal/build/... ./internal/registry/... ./cmd/phpup/...— 103 tests pass (15 new in build, 4 new in cmd/phpup), coverage 83.4% on internal/build.make checkgreen end-to-end locally including docker-backedlocal-cismoke on jammy + noble × amd64 + arm64 × hard4 extensions.phpup build cell,phpup push,phpup --versionall behave correctly on required-flag / unknown-flag / default paths.ci.ymlshould fire; both must pass.Publish gating
publishskips unlessvars.PHPUP_NEW_CI_PUBLISH == '1'is set at the repo level. Default unset → the new pipeline never promotes to GHCR while the old workflows still do. After a week of parallel runs where the new cells' artifact digests match the old pipeline's, flip the variable to'1'to hand over GHCR writes to the new path. Then PR 5 deletes the old workflows.Explicitly deferred
lockfile-updatestill runs asgo run ./cmd/lockfile-updatein the publish step; PR 6 consolidates underphpup lockfile update.phpup push's tag derivation currently uses a digest-prefix fallback (sha256-<12 hex chars>) rather than carrying the source manifest's original tag. Functionally correct (digest-addressed pulls don't care about tag); cosmetic in the GHCR UI until PR 6 threads the original tag via annotations.SeedCoreininternal/build/sidecar.gostill builds its image by hand viabuildTwoLayerImagerather than delegating toremoteStore.Push. Known follow-up from PR 2.5 review, deferred to keep scope tight.Old workflows untouched
Per the spec's PR 4 scope:
build-php-core.yml,build-extension.yml,compat-harness.yml,integration-test.yml,plan-and-build.yml, and friends all still fire on PRs andmain. The grace period proves the new pipeline before PR 5 deletes them.