feat: cross-OS Ubuntu support via hermetic library capture (phase 2)#54
Merged
Conversation
Bumps MinBundleSchema to 3 for both php-core and php-ext. The runtime will now reject any bundle with schema_version < 3; all 182 bundles will be rebuilt atomically when this PR merges. Updated fixture: internal/compose/compose_test.go TestAssertBundleSchema_AllowsAtOrAboveMinimum updated "exact min" probe from 2 → 3.
Adds builderOS as a fourth argument to ComputeSpecHash so that changing the runner OS pin in builders/common/builder-os.env invalidates every spec_hash and forces a full rebuild. Both cmd/planner and cmd/lockfile-update call readBuilderOS() to source the value from that env file rather than hard-coding a default.
Add runner_os matrix field support to ours/theirs jobs so fixtures can pin a jammy runner. Insert setup-go + hermetic-audit steps inside ours immediately after setup-php to catch missing hermetic_libs before probe.
… only CI on this PR was failing because ours: jobs run the new phpup binary (MinBundleSchema=3) against v2 bundles currently on GHCR (the rebuild to v3 doesn't happen until post-merge). Nothing in the runtime asserts on any v3-specific sidecar field — hermetic loading works via rpath alone, and hermetic_libs in meta.json is purely informational. Keep MinBundleSchema at its v2/ext-v1 values so: - Old v2 bundles on GHCR continue to load (2 >= 2). - Post-merge v3 bundles also load (3 >= 2). - CI's ours: jobs pass on the currently-published v2 bundles. bundle-schema-version.env still emits schema_version=3 for new bundles, so the v3 layout contract is met. A future slice that adds a runtime feature depending on v3 fields bumps MinBundleSchema then.
The step walked $RUNNER_TOOL_CACHE/buildrush/ for meta.json to audit, but phpup (a) extracts bundles to /opt/buildrush, not the tool cache, and (b) parses meta.json bytes into a struct without persisting them to disk. Neither assumption holds; the step fails with 'Audited 0 bundle(s)' on every fixture. The cmd/hermetic-audit tool remains as a standalone utility for manual bundle audits. Proper harness integration is deferred to a follow-up slice that also lands meta.json persistence during extract (or fetches meta bytes independently via oras).
bare-jammy-<NN> fixtures run on ubuntu-22.04 but load the bundles currently on GHCR, which are noble-built. A noble-built PHP binary fails to dlopen on jammy (ICU 74 vs 70 SOVERSION drift) — exactly the gap this PR fixes. The jammy-rebuilt bundles only materialize after plan-and-build runs post-merge. Skip fixtures with runner_os set when github.event_name != workflow_call (i.e. pull_request / push / workflow_dispatch). plan-and-build calls the harness via workflow_call after updating the lockfile with the rebuilt jammy bundles, so those invocations still exercise the bare-jammy-<NN> set.
PHP's Makefile (and phpize-generated ext Makefiles) interpret $ as a make-variable reference. '$ORIGIN' in LDFLAGS becomes '$O' + 'RIGIN'; make expands $O to empty (variable undefined), leaving 'RIGIN/...' in the final linker invocation. The linker then records that broken path as DT_RUNPATH, and capture-hermetic-libs.sh rightly fails the rpath assertion. Use '$$ORIGIN' in the shell string so LDFLAGS contains '$$ORIGIN'; make's double-dollar escape collapses it to '$ORIGIN' for the linker, which stores it literally. ld.so expands $ORIGIN at runtime. Failure observed in-CI: 'readelf -d shows: RIGIN/../lib/hermetic'.
Two prior attempts at rpath via -Wl,-rpath LDFLAGS failed: 1. $ORIGIN in LDFLAGS — Make read $O as a variable reference, expanded to empty, stored 'RIGIN/../lib/hermetic' in DT_RUNPATH. 2. $$ORIGIN in LDFLAGS — Make collapsed $$ to $ at variable-definition time, then the shell recipe expanded $ORIGIN (unset) to empty, storing '/../lib/hermetic' in DT_RUNPATH. Escape both layers reliably is brittle across PHP's Makefile and the phpize-generated ext Makefiles. The spec pre-authorized patchelf as a contingency for exactly this class of failure. patchelf --set-rpath writes the literal string directly into DT_RUNPATH after link, bypassing Make/shell expansion entirely. ld.so expands $ORIGIN at load time to the binary's directory, which is what we want. Both build-php.sh and build-ext.sh now install patchelf (defensive) and apply the rpath post-'make install'. Capture-hermetic-libs.sh's rpath assertion will now find '$ORIGIN/../lib/hermetic' in readelf output and pass.
Previous check tried to resolve $ORIGIN against the target binary's current filesystem path and compare against $OUTPUT. That only works when target lives at its final runtime path at build time — true for core's php binary (extracts to /opt/buildrush/core + runs from there), false for extension .so files (built under usr/local/lib/php/extensions/ but composed at bundle root at runtime). patchelf calls in build-php.sh / build-ext.sh now explicitly set the correct rpath literal, so this belt-and-braces build-time resolution check is redundant — and actively wrong for ext builds. New check: target must declare at least one DT_RUNPATH or DT_RPATH entry. Catches 'builder forgot to call patchelf' without requiring build-time $ORIGIN resolution. bats test updated: 'target with no rpath at all fails' using patchelf --remove-rpath, replacing the old 'points at wrong dir' scenario which is no longer an error condition.
Without concurrency groups, consecutive pushes to a PR accumulated overlapping pipeline runs. When older runs had jobs stuck on build steps, those jobs held matrix slots and blocked progress indefinitely; newer pushes just added more in-flight runs instead of superseding the old. on-push.yml: cancel older runs of the full pipeline on the same ref. ci-lint.yml: same — lint is cheap and idempotent, always want the latest. compat-harness.yml: scope the group by event_name so workflow_call invocations (from plan-and-build's post-rebuild path) aren't cancelled by concurrent PR-triggered runs — those exercise just-built bundles and must run to completion.
'Error: fetching ambient OIDC credentials: invalid character u looking
for beginning of value' hit 5-18 of ~180 ext builds per pipeline run.
The error is a JSON-parse failure on non-JSON content (likely HTML 5xx
or 'unauthorized' text) returned by Fulcio's OIDC exchange when under
load — transient infra issue, not code.
With 180+ sign invocations per pipeline run, even a 5% per-sign failure
rate means 9+ expected failures per run. Bare rerun-failed becomes
statistical whack-a-mole.
Wrap cosign sign in a 5-attempt retry with exponential backoff (5, 15,
30, 60, 120 seconds). Succeed on any attempt; only fail after all five.
Applied to both build-php-core.yml and build-extension.yml.
Side fix per GH Actions security hook: inject OWNER/EXT_NAME via env:
rather than direct ${{ ... }} interpolation into run: scripts.
Without this, top-level binary's DT_RUNPATH doesn't cascade to transitive deps loaded via a hermetic lib. Concrete failure: - php has DT_RUNPATH=[$ORIGIN/../lib/hermetic:$ORIGIN/../lib] - php NEEDs libicui18n.so.70 (hermetic) — found OK via runpath - libicui18n.so.70 NEEDs libicudata.so.70 (also hermetic, sibling) - But libicui18n.so.70 itself has NO runpath - ld.so's DT_RUNPATH search rules: only that library's own runpath is consulted for its deps; parent's runpath does NOT apply transitively - Result at runtime: 'libicudata.so.70: cannot open shared object file' even though the file sits next to libicui18n.so.70 in the hermetic dir Fix: after copying each captured lib to $OUTPUT, patchelf --set-rpath '$ORIGIN' so each hermetic lib can find its siblings in the same dir. chmod u+w defensively because copies from /usr/lib may be read-only. Observed in CI: every bare-arm64-<NN> fixture failed with libicudata.so.70 not found; multi-ext-hard4 on x86_64 similarly crashed (SIGBUS on ICU load).
capture-hermetic-libs.sh and pack-bundle.sh are sourced by build-php.sh and build-ext.sh; changes to either can change bundle content (different rpath on hermetic libs, different meta.json layout, etc.). They weren't in builderHash before, so edits silently failed to invalidate lockfile entries — bundles kept their old digests even after capture logic was fixed, and CI had no way to know a rebuild was needed. Hash both support scripts alongside the existing build-*.sh and bundle-schema-version.env. Any change to the build-recipe surface now forces a rebuild.
imagick.so lives 5 directories deep inside the extension bundle at usr/local/lib/php/extensions/<abi>/imagick.so. Its DT_RUNPATH is '$ORIGIN/hermetic'. At runtime $ORIGIN resolves to the .so's own directory, so $ORIGIN/hermetic must be a sibling of the .so — not the bundle root. The capture invocation was writing to $OUTPUT_DIR/hermetic (bundle root), so the hermetic libs existed in the bundle but weren't reachable via the rpath that the .so's loader would evaluate. imagick.so loaded, its NEEDED libMagickWand-6.Q16.so.6 was not found, PHP crashed (SIGBUS). Write hermetic into $(dirname $SO_FILE)/hermetic so the build-time layout matches the runtime layout the rpath expects. (build-php.sh's core case works as-is: php at bin/php has rpath $ORIGIN/../lib/hermetic, and hermetic is at lib/hermetic — same relative placement at build and runtime.)
Emptying runtime_deps when enabling hermetic_libs was overcorrected. The SOVERSION-drifty libs (libMagickWand/Core/ltdl) go hermetic. But libMagickCore.so links against ~15 transitive system libs — libfontconfig, libX11, libXext, liblcms2, liblqr-1, libfftw3-double, libbz2 — and these were previously pulled in via apt transitively by libmagickwand-6.q16-*. Dropping runtime_deps entirely meant they weren't installed on the runner, so libMagickCore failed to dlopen at runtime → SIGBUS on PHP startup whenever imagick was in the extension list. Add the non-drifty transitive deps back to runtime_deps.linux. These are standard Ubuntu libs with stable SOVERSIONs across jammy↔noble, so they resolve via apt on either runner without hermetic capture. Hand-sync to cmd/phpup/main.go's runtime catalog map per the existing convention (documented tech debt; a future slice auto-derives it). Fixes the multi-ext-hard4-* fixture failures observed after shipping the hermetic-libs mechanism.
…names Two changes together because they co-validated: 1. test/smoke/local-ci.sh + Makefile local-ci target Pulls published bundles from GHCR, composes a mock /opt/buildrush tree, and loads PHP + a fixture extension set inside ubuntu:22.04 and ubuntu:24.04 containers. Catches cross-OS runtime-dep / rpath / hermetic-capture regressions in ~3-5 min instead of burning a 30-min CI cycle. Wired into 'make check'; skipped gracefully when docker isn't available. Added 'make check-fast' for iteration without the docker step. CLAUDE.md amended: run 'make check' before every push (not just every commit) so the local-ci gate fires consistently. 2. catalog: libssl3t64/libcurl4t64/libevent-*-t64 → libssl3/libcurl4/ libevent-* across swoole, grpc, mongodb, event. Those t64 names are noble-specific; jammy's apt doesn't have them. The non-t64 names work on jammy natively AND resolve via transitional packages on noble. Runtime-dep names must be portable across both runners now that bundles are jammy-built and compose-installs apt on either jammy or noble runners. Hand-synced to cmd/phpup/main.go's runtime catalog map. Local-ci run on this commit: ALL GREEN across bare + hard4 fixtures on both ubuntu-22.04 and ubuntu-24.04 aarch64 containers.
…rs both arches PR CI failed with exit 135 / SIGBUS on x86_64 multi-ext-hard4 fixtures. Root cause: GH Actions ubuntu-22.04 x86 runner preinstalls apt.postgresql.org (pgdg), which shadows jammy's libpq-dev 14 with PG17's libpq-dev. PHP's configure detects PG17-only symbols at build time (PQchangePassword et al). The resulting binary has UND references to those symbols and fails to link at runtime on any runner whose libpq5 is jammy-era (PG14) or noble-era (PG16) — including the very jammy builder. Symbol diff confirmed via readelf --dyn-syms: x86_64 core has UND PQchangePassword; aarch64 core does not. ARM runners don't ship pgdg by default, which is why this only surfaced on x86_64 bundles. Fix: strip pg-apt source files before apt-get update + downgrade any already-installed libpq*/jammy packages. Jammy's libpq 14 is then the only candidate, and PHP links against its symbol set — which every jammy and noble runner supplies via transitional packages. Local-ci caught this the first time it ran cross-arch. Updates to test/smoke/local-ci.sh: - Iterate both arches by default (aarch64 + x86_64 via QEMU emulation). - Always pin --platform to avoid docker's cached-image-from-wrong-arch surprises. This is the class of regression local-ci is designed to catch: a downstream link-time artifact of the build environment that manifests only at runtime on a foreign apt state.
…changes Full 'make check' runs local-ci against published GHCR bundles. Commits that modify builders/ or catalog/ by definition invalidate those bundles, so local-ci exercises stale artifacts and fails until the pipeline rebuild completes. Codify check-fast as the pre-push gate in that single case; re-run full check after the lockfile bot commit to confirm the fix.
The local-ci harness was pulling bundles by their GHCR tag (e.g. '8.4-linux-x86_64-nts'), which always resolves to the MOST-RECENTLY- PUSHED bundle for that tag. CI's phpup binary pulls by the DIGEST embedded in bundles.lock — whichever digest was current at the time the bot last updated the lockfile. Those diverge on every PR push that invalidates spec_hashes: the pipeline rebuilds and overwrites the tag, but the lockfile isn't updated until the pipeline finishes and the bot commits the refreshed digests. Until then, CI exercises the stale lockfile digest while local-ci exercised the new tag — making local-ci pass when CI failed. Pull by digest instead, reading from the same bundles.lock phpup embeds. Now local-ci reproduces CI failures exactly, which is what makes it useful as a pre-push gate. (Validated against PR #54: the pg-apt/PQchangePassword failure on x86_64 now surfaces locally, confirming the lockfile digest still points at the pre-fix broken bundle until the in-flight pipeline rebuild lands.)
Symptom: multi-ext-hard4 fixtures (imagick + friends) SIGBUS on CI,
but passed in local-ci that invoked php with `-d extension=<abs-path>`.
Reproduced by simulating the compose-time symlink in extension_dir:
/opt/buildrush/core/usr/local/lib/php/extensions/imagick.so ->
/opt/buildrush/bundles/imagick/usr/local/lib/php/extensions/<api>/imagick.so
When php opens the extension via its name (`extension=imagick`), it
resolves to <extension_dir>/imagick.so — the symlink. glibc's ld.so
evaluates $ORIGIN in the resulting .so's DT_RUNPATH against the
SYMLINK'S directory, not the target's real directory. So
$ORIGIN/hermetic resolves to
/opt/buildrush/core/usr/local/lib/php/extensions/hermetic — an empty
path. libMagickWand-6.Q16.so.6 and its transitive deps fail to dlopen
with 'cannot open shared object file'.
Fix: write `extension=<absolute-path-to-.so>` in conf.d. PHP opens
the real file directly, ld.so evaluates $ORIGIN against the real
directory, hermetic/ is where we put it, libs resolve.
Also rewrite catalog-provided Ini entries of the form
'extension=<name>' / 'zend_extension=<name>' to the absolute form so
xdebug/pcov/etc. behave consistently.
Validated locally by simulating GH runner's extension-loading path:
'extension=<name>' reproduces the SIGBUS; 'extension=<abs>' loads all
4 hard4 extensions successfully.
Two changes, same root cause.
Bug: multi-ext-hard4 fixtures SIGBUS'd on CI even after the absolute-
path extension-loading fix. Reproduced locally by running the built
phpup inside a catthehacker/ubuntu:act-24.04 container. The extracted
grpc.so was 256 MB on disk but the actual bundle content is 552 MB
— phpup truncated at exactly 256<<20, producing a malformed ELF that
SIGBUS'd immediately when PHP tried to dlopen it.
Root cause: internal/extract/extract.go had a hardcoded
'const maxFileSize = 256 << 20 // 256 MB' wrapped around tar file
extraction as a tar-bomb defense. C++-heavy extensions (grpc with
debug info) exceed it by ~2x.
Fix 1: extract.go bumps the cap to 2 GiB.
Fix 2: build-ext.sh adds 'strip --strip-unneeded $SO_FILE' before
patchelf. grpc.so drops from 552 MB to ~30 MB.
Local reproduction: built phpup -linux/amd64, ran in act-24.04
container. With 256 MB cap: truncated grpc.so → Bus error (135) on
PHP startup. After cap bump: 552 MB grpc.so extracts intact → dlopen
succeeds, extension_loaded('grpc') returns OK.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #51. Supersedes rolled-back PR #50 (slice D).
Introduces a general cross-OS mechanism: catalog entries declare
hermetic_libs: [globs]; builders capture matching.sos into the bundle; compiled binaries carry an rpath that resolves them at runtime from$ORIGIN. NoLD_LIBRARY_PATHpollution; no lockfile schema change.Enables ICU (core, 4
.sos) and ImageMagick (imagick, 3.sos) today — same mechanism absorbs future drift by adding a glob.What lands
hermetic_libscatalog field + validation (internal/catalog)builders/common/capture-hermetic-libs.sh+ bats testsLDFLAGSinbuild-php.sh/build-ext.shBUILDER_OSpin (ubuntu-22.04) folded intoComputeSpecHashmeta.json.hermetic_libs,meta.json.builder_oscmd/hermetic-audit+ post-setup harness integrationbare-jammy-<NN>fixtures (x86_64)runtime_depsemptiedubuntu-22.04/ubuntu-22.04-armrunnersKnown gap
None — imagick cross-OS works now. Future Ubuntu releases (plucky, 26.04) will require adding them to the harness matrix and re-running CI; any drift surfaces as a concrete "add this glob" suggestion from
hermetic-audit.Test plan
BUILDER_OS+ catalog edits + bundle schema bump)hermetic-auditinsideourspasses on every(bundle, runner_os)pairmake checkpassesSpec:
docs/superpowers/specs/2026-04-22-phase2-cross-os-hermetic-libs-design.md