From 700a3c5937c343ad1cf23195b828e35b807b85d4 Mon Sep 17 00:00:00 2001 From: "Jonathan D.A. Jewell" <6759885+hyperpolymath@users.noreply.github.com> Date: Tue, 21 Jul 2026 16:22:11 +0100 Subject: [PATCH] fix(ci): staleness gate fails on named defects, not on the calendar MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MEASURED 2026-07-21: 294 of 350 consumers were red on the single pin d7c22711 (2026-06-26) — every one of them green a fortnight earlier, with nothing changed in any consumer. aspasia is the clean natural experiment: same pin, 2026-07-09 run at 32 commits / 12d = passing notice; 2026-07-21 at 63 commits / 24d = hard error. standards moves ~2.6 commits/day, so a consumer exhausts both window budgets ~14 days after any propagation. Holding the fleet green under that rule means re-pinning ~300 repos every fortnight (order 7,800 PRs/year). A gate that fails everyone on a timer is not a guard: it trains the estate to ignore it, and it buries the findings that matter — the same sweep shows 148 real Workflow security linter failures and 74 anti-pattern failures that read as noise once the fleet is uniformly red. 1. Age outside the window is now a ::notice, not an error. The window is still computed and still reported, so propagate-workflow-pins.sh and the Hypatia sha_bump_propagation rule keep their signal. 2. A named deny-list (KNOWN_BAD_BEFORE) replaces age as the hard failure. Each entry is :; a pin that is a strict ancestor of the fix carries that defect and is rejected at any age. This is what the window was only ever a proxy for, and it is strictly better in both directions: a RECENT pin carrying the defect is now caught, and an old pin carrying none is no longer punished by the calendar. First entry is e9c8888769a7 (#441). Before it, hypatia-scan-reusable and the validate-hypatia-baseline job cached the built Hypatia scanner under a keyless key while the build steps were guarded by `if [ ! -d ]`, so the first scanner build ever cached was reused forever and scanner fixes never took effect. A pin older than this reports a FALSE GREEN. All 294 consumers on d7c22711 predate it by one day and stay red — now for a true and actionable reason. 3. Integrity no longer rests on the runner's clone. The UNKNOWN / "may be forged" verdict was firing on legitimate pins: awesome-haskell pins governance-reusable@5a93d9d5 and was accused on four consecutive runs over 17 days, while that commit verifies as a true ancestor of main locally in both treeless and --depth 200 clones and via the compare API (behind=0, ahead=90). The mechanism was never reproduced off-runner. A hard FORGED verdict now requires either a COMPLETE local clone (not shallow, not partial) or confirmation from GET /repos/{nwo}/compare/{pin}...{branch}. Where neither is available the gate warns and passes: "cannot verify" is not "compromised". Zero API calls on the happy path — the server is consulted only when about to accuse. The deny-list gets the same fallback, so a degraded runner cannot silently skip it; "could not check" is reported as SKIPPED, never as passed. Tests: the hermetic fixture suite goes 12 -> 16 cases. Out-of-window now asserts exit 0 AND the advisory notice (a new run_case_out helper — an exit code alone cannot distinguish "passed silently" from "passed with the notice the propagation path depends on"). New coverage: deny-list rejects at any age, rejects even when in-window, does not reject the fixing commit itself, and is scoped per reusable. 16/16 pass, and the suite stays hermetic because a complete fixture clone resolves the forged case without the network. Also verified against live standards history and in degraded-clone mode; both give identical verdicts for fresh / denied / old-but-clean / forged. Co-Authored-By: Claude Opus 4.8 --- ...ADR-003-workflow-pin-staleness-window.adoc | 61 ++++ scripts/check-workflow-staleness.sh | 262 ++++++++++++++++-- .../tests/check-workflow-staleness-test.sh | 63 ++++- 3 files changed, 366 insertions(+), 20 deletions(-) diff --git a/docs/decisions/ADR-003-workflow-pin-staleness-window.adoc b/docs/decisions/ADR-003-workflow-pin-staleness-window.adoc index 003fa733..0f4ad6dc 100644 --- a/docs/decisions/ADR-003-workflow-pin-staleness-window.adoc +++ b/docs/decisions/ADR-003-workflow-pin-staleness-window.adoc @@ -145,6 +145,10 @@ minimal data) instead of `--depth 1`, with a `--depth 200` fallback. `scripts/tests/check-workflow-staleness-test.sh` (hermetic fixture history) covers fresh / ahead / in-window-by-commits / in-window-by-days / out-of-window / forged, plus the retired-enforcer and SARIF structural rules. +Extended by the 2026-07-21 amendment below to 16 cases: out-of-window now +asserts exit 0 *and* the advisory notice, plus deny-list coverage (rejected at +any age, rejected even when in-window, the fixing commit itself not rejected, +and entries scoped per reusable). `scripts/tests/propagate-workflow-pins-test.sh` covers audit/fix/idempotence and branch staging. End-to-end against real standards history, a stapeln-shaped consumer pinned 3 commits behind HEAD passes the relaxed gate @@ -162,3 +166,60 @@ stapeln-shaped consumer pinned 3 commits behind HEAD passes the relaxed gate rules out mutable-tag pinning (option C). * `scripts/propagate-gitignore-67-68.sh` — the audit-first propagation pattern this helper follows. + +== Amendment (2026-07-21): age is advisory; defects are named + +The window narrowed the treadmill but did not remove it. `standards` moves +~2.6 commits/day, so a consumer exhausts both budgets roughly a fortnight +after any propagation. + +Measured on 2026-07-21: *294 of 350* consumers were red on the single pin +`d7c22711` (2026-06-26) — every one of them green a fortnight earlier, with +nothing changed in any consumer. `aspasia` is the clean natural experiment: +same pin, 2026-07-09 run at 32 commits / 12d = passing notice; 2026-07-21 at +63 commits / 24d = hard error. Keeping the fleet green under that rule means +re-pinning ~300 repos every fortnight — of the order of 7,800 PRs/year. + +A gate that fails everyone on a timer is not a guard. It trains the estate to +ignore it and it buries the findings that matter: the same run reports 148 +real `Workflow security linter` failures and 74 anti-pattern failures, all of +which read as noise once the fleet is uniformly red. + +Two changes, both in `scripts/check-workflow-staleness.sh`: + +. *Age outside the window is now a `::notice`, not an error.* The window still + exists and is still reported, so `propagate-workflow-pins.sh` and the Hypatia + `sha_bump_propagation` rule keep their signal. +. *A named deny-list replaces age as the hard failure* (`KNOWN_BAD_BEFORE`). + Each entry is `:`; a pin that is a strict ancestor of the + fix carries that defect and is rejected at any age. This is what the window + was only ever a proxy for, and it is strictly better in both directions — a + *recent* pin carrying the defect is now caught, and an old pin carrying none + is no longer punished by the calendar. + +The first entry is `e9c8888769a7` (#441): before it, `hypatia-scan-reusable` +and the `validate-hypatia-baseline` job cached the built Hypatia scanner under +a keyless key while the build steps were guarded by `if [ ! -d ]`, so the first +scanner build ever cached was reused forever and scanner fixes never took +effect. A pin older than this reports a *false green* — which is why it must +be refreshed and cannot be waited out. All 294 consumers on `d7c22711` predate +it by one day, so they remain red, now for a true and actionable reason. + +=== Integrity no longer rests on the runner's clone + +The `UNKNOWN` verdict ("may be forged") was firing on legitimate pins. +`hyperpolymath/awesome-haskell` pins `governance-reusable@5a93d9d5` and was +accused on four consecutive runs across 17 days, while that commit verifies as +a true ancestor of `main` locally in both treeless and `--depth 200` clones and +via the compare API (`behind=0, ahead=90`). The mechanism was never reproduced +off-runner. + +A supply-chain accusation must not depend on the health of a clone the gate +does not control. A hard `FORGED` verdict now requires either a *complete* +local clone (not shallow, not partial) or confirmation from +`GET /repos/{nwo}/compare/{pin}...{branch}`. Where neither is available the +gate emits a `::warning` and passes: *cannot verify* is not *compromised*. +Cost is zero API calls on the happy path — the server is consulted only when +the gate is about to accuse. The deny-list gets the same fallback, so a +degraded runner cannot silently skip it: "could not check" is reported as +SKIPPED, never as passed. diff --git a/scripts/check-workflow-staleness.sh b/scripts/check-workflow-staleness.sh index 98615852..9d065157 100755 --- a/scripts/check-workflow-staleness.sh +++ b/scripts/check-workflow-staleness.sh @@ -21,17 +21,34 @@ set -eo pipefail # escalates): a pin PASSES if it is a genuine ancestor of standards HEAD that # is *within a recency window* — at most STALENESS_WINDOW_COMMITS commits # behind HEAD, OR at most STALENESS_WINDOW_DAYS days old (union; either -# qualifies). A pin still FAILS if it is older than the window (so pins cannot -# rot forever and the known-bad historical pins — pre-cache-fix Hypatia, the -# SARIF-publishing Scorecard — remain blocked), or if it is not a real -# standards commit at all (supply-chain integrity: forged / unrelated SHAs are -# rejected). +# qualifies). +# +# ── Why age alone no longer FAILS (2026-07-21) ────────────────────────────── +# The window was still a clock, just a slower one. standards moves ~2.6 +# commits/day, so a consumer exhausts both budgets ~14 days after any +# propagation — measured: 294 of 350 consumers red on the single pin d7c22711, +# every one of them green a fortnight earlier, with nothing changed in any +# consumer. Keeping the fleet green would have meant re-pinning ~300 repos +# every fortnight (~7,800 PRs/year). A gate that fails everyone on a timer is +# not a guard; it trains the estate to ignore it, and it buries the findings +# that matter. +# +# So the gate now fails on facts about the pin, never on its age: +# * KNOWN_BAD_BEFORE — the pin predates the fix for a *named* defect. This is +# the honest expression of what the window was proxying for, and it is +# strictly better in both directions: a recent pin carrying the defect is +# still caught, and an old pin carrying none is not punished by the +# calendar. The pre-cache-fix Hypatia scanner (#441) is the first entry. +# * FORGED — the pin is not a commit of this repository reachable from the +# default branch, confirmed server-side (supply-chain integrity). +# * the two structural rules (retired scorecard-enforcer; Scorecard SARIF). +# Age outside the window is reported as a ::notice for the propagation path to +# act on. # # Deliberately jumping a consumer to HEAD (e.g. to pick up a fix early) is the # job of the propagation path — `scripts/propagate-workflow-pins.sh` plus the # Hypatia `sha_bump_propagation` rule + gitbot-fleet — which opens an -# audit-first bump PR. This gate no longer *forces* that bump; it only blocks -# pins that have fallen out of the window. See +# audit-first bump PR. See # docs/decisions/ADR-003-workflow-pin-staleness-window.adoc. # # Environment knobs (all optional): @@ -43,7 +60,17 @@ set -eo pipefail # is available for ancestry/age math). # STALENESS_WINDOW_COMMITS — max commits-behind-HEAD a pin may be (default 50). # STALENESS_WINDOW_DAYS — max age in days a pin may be (default 14, matching -# HYP-S006 stale_after_days). +# HYP-S006 stale_after_days). Advisory since +# 2026-07-21: exceeding it emits a notice, not an +# error. +# STALENESS_STANDARDS_NWO — owner/repo a pin must belong to, used for the +# server-side integrity check (default +# hyperpolymath/standards). +# STALENESS_STANDARDS_BRANCH— branch a pin must be reachable from (default main). +# STALENESS_KNOWN_BAD_BEFORE— override the known-bad deny-list, as space- or +# comma-separated : entries. +# "-" disables it. For the hermetic fixture tests +# and downstream forks. REPO_ROOT="${1:-.}" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -51,6 +78,44 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" WINDOW_COMMITS="${STALENESS_WINDOW_COMMITS:-50}" WINDOW_DAYS="${STALENESS_WINDOW_DAYS:-14}" +# Where to ask when the local checkout cannot be trusted (see verify-by-API +# below). Overridable so a fork can point the gate at its own standards. +STANDARDS_NWO="${STALENESS_STANDARDS_NWO:-hyperpolymath/standards}" +STANDARDS_BRANCH="${STALENESS_STANDARDS_BRANCH:-main}" + +# ── Known-bad pins: hard fail regardless of age ───────────────────────────── +# A recency *window* is only a proxy for "this pin might contain a known +# defect". Where the defect is actually known, say so directly: name the commit +# that fixed it and reject anything strictly older. This is both stricter (a +# recent pin carrying the defect is still caught) and kinder (an old pin +# without it is not punished for the calendar). +# +# Each entry is :. +# +# e9c8888769a7 (2026-06-27, #441) — before this commit, hypatia-scan-reusable +# and the validate-hypatia-baseline job in governance-reusable both cached the +# built Hypatia scanner under the keyless key +# `hypatia-scanner-v2-${{ runner.os }}-build`, while the clone/build steps +# were guarded by `if [ ! -d ]` / `if [ ! -x ]`. The FIRST scanner build ever +# cached was therefore restored and reused forever and scanner fixes never +# took effect in CI. A pin older than this runs a frozen scanner and reports +# a FALSE GREEN — the one failure mode this estate cannot tolerate. Such a +# pin must be refreshed; it cannot be waited out. +KNOWN_BAD_BEFORE=( + "hypatia-scan-reusable.yml:e9c8888769a703924cc3c0d717900960d78aea00" + "governance-reusable.yml:e9c8888769a703924cc3c0d717900960d78aea00" +) +# Overridable (space- or comma-separated :) so the hermetic +# regression fixture and downstream forks can exercise this list without +# depending on real standards SHAs. Set to a single "-" to disable. +if [ -n "${STALENESS_KNOWN_BAD_BEFORE:-}" ]; then + if [ "$STALENESS_KNOWN_BAD_BEFORE" = "-" ]; then + KNOWN_BAD_BEFORE=() + else + IFS=', ' read -r -a KNOWN_BAD_BEFORE <<< "$STALENESS_KNOWN_BAD_BEFORE" + fi +fi + # The standards reusables a consumer pins by SHA. The set is intentionally the # trio named in the governance contract; extend here if a new reusable joins # the freshness gate. Commented-out `uses:` examples are skipped (see below). @@ -124,19 +189,152 @@ ensure_commit() { git -C "$STANDARDS_DIR" cat-file -e "${sha}^{commit}" 2>/dev/null } +# ── Verify a pin against the server, not the runner's clone ───────────────── +# MEASURED (2026-07-21): the local `merge-base --is-ancestor` path deterministic- +# ally mis-classified legitimate pins as forged in CI. hyperpolymath/awesome- +# haskell pins governance-reusable@5a93d9d57cc0 and was told "not a recognised +# ancestor ... may be forged" on four consecutive runs over 17 days, while that +# same commit verifies as a true ancestor of main both locally (treeless AND +# --depth 200 clones) and via the API (behind=0, ahead=90). The mechanism was +# never reproduced off-runner. +# +# Rather than depend on the clone being intact for a *security* verdict, ask +# GitHub. The compare endpoint is authoritative and immune to whatever degrades +# the runner's clone (partial-clone promisor failure, the --depth 200 fallback, +# runner git version). +# +# Cost: zero calls on the happy path. This is consulted only when we are about +# to accuse a pin of being forged — so the unauthenticated 60/hr limit is not a +# concern. (The staleness job is given no token, so auth is best-effort.) +# Is the local standards checkout complete enough for its *negative* answers to +# be trusted? A shallow or partial clone can fail to resolve or relate a commit +# for reasons that have nothing to do with that commit's legitimacy, and a +# hard failure must never rest on that. A complete clone's "no" is final. +clone_is_complete() { + [ "$HAVE_HISTORY" = true ] || return 1 + local gd + gd=$(git -C "$STANDARDS_DIR" rev-parse --git-dir 2>/dev/null) || return 1 + [ -e "$STANDARDS_DIR/$gd/shallow" ] || [ -e "$gd/shallow" ] && return 1 + [ -n "$(git -C "$STANDARDS_DIR" config --get remote.origin.partialclonefilter 2>/dev/null)" ] && return 1 + [ "$(git -C "$STANDARDS_DIR" config --get remote.origin.promisor 2>/dev/null)" = "true" ] && return 1 + return 0 +} + +# Decide a pin the local clone could not vouch for. Trust a complete clone's +# negative; otherwise get a second opinion from the server before accusing. +resolve_negative() { + local pin="$1" + if clone_is_complete; then + echo "FORGED" + else + classify_via_api "$pin" + fi +} + +# api_compare [head] (head defaults to the standards default branch) +api_compare() { + local pin="$1" head="${2:-$STANDARDS_BRANCH}" url body code st ahead behind + command -v curl >/dev/null 2>&1 || return 1 + url="https://api.github.com/repos/${STANDARDS_NWO}/compare/${pin}...${head}" + local -a auth=() + [ -n "${GITHUB_TOKEN:-}" ] && auth=(-H "Authorization: Bearer ${GITHUB_TOKEN}") + body=$(curl -sS --max-time 20 -w '\n%{http_code}' \ + -H 'Accept: application/vnd.github+json' \ + "${auth[@]}" "$url" 2>/dev/null) || return 1 + code=$(printf '%s\n' "$body" | tail -n1) + body=$(printf '%s\n' "$body" | sed '$d') + case "$code" in + 200) ;; + 404|422) echo "NOTFOUND 0 0"; return 0 ;; # SHA not in this repository + *) return 1 ;; # rate limited / offline / 5xx + esac + # `status` describes head (main) relative to base (the pin): + # identical | ahead (main ahead => pin IS an ancestor) | behind | diverged + st=$(printf '%s' "$body" | grep -o '"status"[[:space:]]*:[[:space:]]*"[a-z]*"' | head -n1 | sed 's/.*"\([a-z]*\)"$/\1/') + ahead=$(printf '%s' "$body" | grep -o '"ahead_by"[[:space:]]*:[[:space:]]*[0-9]*' | head -n1 | grep -o '[0-9]*$') + behind=$(printf '%s' "$body" | grep -o '"behind_by"[[:space:]]*:[[:space:]]*[0-9]*' | head -n1 | grep -o '[0-9]*$') + [ -n "$st" ] || return 1 + echo "$st ${ahead:-0} ${behind:-0}" +} + +# Second opinion for a pin the local clone could not vouch for. Echoes the same +# verdict vocabulary as classify_pin. FORGED is the only hard-fail outcome; +# "we could not check" is deliberately NOT an accusation. +classify_via_api() { + local pin="$1" out st ahead + out=$(api_compare "$pin") || { echo "UNVERIFIABLE"; return; } + # shellcheck disable=SC2086 + set -- $out; st="$1"; ahead="$2" + case "$st" in + NOTFOUND|diverged) echo "FORGED" ;; + identical) echo "FRESH" ;; + behind) echo "AHEAD" ;; + ahead) + # main is $ahead commits ahead of the pin => the pin is a real ancestor. + # Age is unavailable by this route; the window is advisory now, so '?' + # is honest rather than a fabricated number. + if [ "$ahead" -le "$WINDOW_COMMITS" ]; then echo "IN_WINDOW $ahead ?" + else echo "OUT_OF_WINDOW $ahead ?"; fi ;; + *) echo "UNVERIFIABLE" ;; + esac +} + +# Does this pin predate the fix for a known false-green defect in this +# reusable? Returns 0 (yes, reject) only when we can prove it; an unresolvable +# pin is never accused here. +pin_is_known_bad() { + local reusable="$1" pin="$2" entry fixsha pin_full fix_full out st + KNOWN_BAD_UNCHECKED="" + for entry in "${KNOWN_BAD_BEFORE[@]}"; do + [ "${entry%%:*}" = "$reusable" ] || continue + fixsha="${entry#*:}" + + # Preferred: decide locally, no network. + if [ "$HAVE_HISTORY" = true ] && ensure_commit "$pin" && ensure_commit "$fixsha"; then + pin_full=$(git -C "$STANDARDS_DIR" rev-parse "${pin}^{commit}" 2>/dev/null) || continue + fix_full=$(git -C "$STANDARDS_DIR" rev-parse "${fixsha}^{commit}" 2>/dev/null) || continue + [ "$pin_full" = "$fix_full" ] && continue # the fix itself is fine + if git -C "$STANDARDS_DIR" merge-base --is-ancestor "$pin_full" "$fix_full" 2>/dev/null; then + KNOWN_BAD_FIX="$fix_full" + return 0 + fi + continue + fi + + # Degraded clone: ask the server rather than skip. A check that quietly + # stops checking when the runner is unhealthy is a fake gate — the very + # class of defect this deny-list exists to catch. + if out=$(api_compare "$pin" "$fixsha"); then + # shellcheck disable=SC2086 + set -- $out; st="$1" + if [ "$st" = "ahead" ]; then # fix is ahead of pin + KNOWN_BAD_FIX="$fixsha" + return 0 + fi + continue + fi + + # Neither route available: say so out loud (handled by the caller). + KNOWN_BAD_UNCHECKED="$fixsha" + done + return 1 +} + # Classify a pin relative to CURRENT_SHA. Echoes one of: # FRESH | AHEAD | IN_WINDOW | OUT_OF_WINDOW | UNKNOWN classify_pin() { local pin="$1" - # Legacy/degraded mode: no standards history available -> exact match only. + # Degraded mode: no standards history available. Previously this meant + # "exact match or you are forged", which turned a broken clone into a + # supply-chain accusation. Ask the server instead. if [ "$HAVE_HISTORY" != true ]; then - [ "$pin" = "$CURRENT_SHA" ] && echo "FRESH" || echo "UNKNOWN" + [ "$pin" = "$CURRENT_SHA" ] && echo "FRESH" || classify_via_api "$pin" return fi if ! ensure_commit "$pin"; then - echo "UNKNOWN" + resolve_negative "$pin" return fi @@ -176,14 +374,29 @@ classify_pin() { return fi - # Diverged: a real commit, but not on the line between an ancestor and HEAD. - echo "UNKNOWN" + # The local clone says this pin is neither ancestor nor descendant of HEAD. + # That is the forgery signal — and precisely the verdict measured to fire on + # legitimate pins. Never accuse on a degraded clone's word alone. + resolve_negative "$pin" } # Apply the freshness verdict for one reusable pin found in one workflow file. check_reusable_pin() { local wf="$1" reusable="$2" pin="$3" local verdict behind age + + # A named defect outranks the calendar: check it first and independently of + # how fresh the pin looks. This is the only age-related hard failure. + KNOWN_BAD_FIX=""; KNOWN_BAD_UNCHECKED="" + if pin_is_known_bad "$reusable" "$pin"; then + echo "::error file=$wf::${reusable} pin ${pin:0:12} predates ${KNOWN_BAD_FIX:0:12} and carries the frozen-Hypatia-scanner-cache defect (#441): the first scanner build ever cached is restored forever, so scanner fixes never take effect and the scan reports a FALSE GREEN. Refresh this pin — waiting will not fix it (scripts/propagate-workflow-pins.sh)." + FAILED=1 + return + fi + if [ -n "$KNOWN_BAD_UNCHECKED" ]; then + echo "::warning file=$wf::${reusable} pin ${pin:0:12} could not be checked against the known-bad list (local standards history unavailable and the compare API unreachable). This check was SKIPPED, not passed." + fi + read -r verdict behind age < <(classify_pin "$pin") case "$verdict" in @@ -195,13 +408,26 @@ check_reusable_pin() { echo "::notice file=$wf::${reusable} pin ${pin:0:12} is ${behind} commit(s) / ${age}d behind standards HEAD — within the recency window (<=${WINDOW_COMMITS} commits or <=${WINDOW_DAYS}d). Bump deliberately with scripts/propagate-workflow-pins.sh when convenient." ;; OUT_OF_WINDOW) - echo "::error file=$wf::${reusable} pin ${pin:0:12} is ${behind} commit(s) / ${age}d behind standards HEAD — outside the recency window (>${WINDOW_COMMITS} commits AND >${WINDOW_DAYS}d). Refresh toward ${CURRENT_SHA:0:12} (e.g. scripts/propagate-workflow-pins.sh)." - FAILED=1 + # Advisory, NOT a failure. Age alone is not a defect, and making it one + # made the gate a clock: standards moves ~2.6 commits/day, so every + # consumer went red ~14 days after each propagation whether or not + # anything was wrong with it (measured 2026-07-21: 294 of 350 consumers + # red on one pin, all of them green a fortnight earlier). A gate that + # fails the whole fleet on a timer teaches everyone to ignore it. Real + # defects are named in KNOWN_BAD_BEFORE above and still hard-fail. + echo "::notice file=$wf::${reusable} pin ${pin:0:12} is ${behind} commit(s) / ${age}d behind standards HEAD — outside the recency window (>${WINDOW_COMMITS} commits AND >${WINDOW_DAYS}d). Advisory only: no known defect is attached to this pin. Refresh toward ${CURRENT_SHA:0:12} with scripts/propagate-workflow-pins.sh." ;; - UNKNOWN|*) - echo "::error file=$wf::${reusable} pin ${pin:0:12} is not a recognised ancestor of standards HEAD (${CURRENT_SHA:0:12}). It may be forged, from a fork, or too divergent to verify. Pin a published standards commit." + FORGED) + echo "::error file=$wf::${reusable} pin ${pin:0:12} is not a commit of ${STANDARDS_NWO} reachable from ${STANDARDS_BRANCH} (confirmed against the GitHub compare API, not just this runner's clone). It may be forged, from a fork, or from rewritten history. Pin a published standards commit." FAILED=1 ;; + UNVERIFIABLE|*) + # Could not reach a verdict: the clone is degraded AND the API was + # unreachable or rate-limited. "Cannot verify" is not "compromised" — + # warn loudly, do not fail. Failing here is what produced four + # consecutive false forgery accusations against awesome-haskell. + echo "::warning file=$wf::${reusable} pin ${pin:0:12} could not be verified against ${STANDARDS_NWO} (local clone degraded and the compare API was unreachable). Not treated as a failure; re-run to re-check." + ;; esac } @@ -233,7 +459,7 @@ for wf in "$REPO_ROOT"/.github/workflows/*.yml "$REPO_ROOT"/.github/workflows/*. done if [ $FAILED -ne 0 ]; then - echo "::error::Remove legacy scorecard-enforcer.yml, refresh out-of-window standards reusable pins toward a recent commit, and keep Scorecard out of GitHub Code Scanning unless it runs for every PR head commit." + echo "::error::Staleness gate failed. Each error above names a specific defect: a pin predating a known false-green fix (refresh it — waiting will not help), a pin that is not a published standards commit, a retired scorecard-enforcer.yml, or Scorecard uploading SARIF to Code Scanning. Pins that are merely old are reported as notices and do not fail." exit 1 fi diff --git a/scripts/tests/check-workflow-staleness-test.sh b/scripts/tests/check-workflow-staleness-test.sh index 145e7ba5..54c06f68 100755 --- a/scripts/tests/check-workflow-staleness-test.sh +++ b/scripts/tests/check-workflow-staleness-test.sh @@ -76,6 +76,29 @@ run_case() { fi } +# run_case_out desc expected_exit expected_output_regex repo [VAR=VAL ...] +# As run_case, but also asserts the gate said the right thing — an exit code +# alone cannot tell "passed silently" from "passed with the advisory notice +# the propagation path depends on". +run_case_out() { + local desc="$1" expected="$2" regex="$3" repo="$4"; shift 4 + TOTAL=$((TOTAL + 1)) + local out rc + set +e + out=$(env "$@" \ + GITHUB_REPOSITORY="hyperpolymath/test-repo" \ + STALENESS_STANDARDS_DIR="$FIX" \ + bash "$CHECK_SCRIPT" "$repo" 2>&1) + rc=$? + set -e + if [ "$rc" -eq "$expected" ] && printf '%s' "$out" | grep -qE "$regex"; then + echo "PASS: $desc" + PASS=$((PASS + 1)) + else + echo "FAIL: $desc (expected exit $expected + /$regex/, got exit $rc)" + fi +} + # ── 1. FRESH: pin == HEAD ─────────────────────────────────────────────────── R="$TEST_DIR/fresh"; mk_repo "$R" write_pin "$R/.github/workflows/governance.yml" "governance-reusable.yml" "$C5" @@ -98,12 +121,48 @@ write_pin "$R/.github/workflows/scorecard.yml" "scorecard-reusable.yml" "$C1" run_case "300d/4-behind pin passes on commits axis (window 5 commits / 1 day)" 0 "$R" \ STALENESS_WINDOW_COMMITS=5 STALENESS_WINDOW_DAYS=1 -# ── 5. OUT_OF_WINDOW: outside both axes -> fail ───────────────────────────── +# ── 5. OUT_OF_WINDOW: outside both axes -> ADVISORY, not a failure ────────── +# Contract change (2026-07-21): age alone no longer fails. Making it fail turned +# the gate into a clock that reddened the whole fleet ~14 days after every +# propagation (294 of 350 consumers, one pin, nothing wrong with any of them). +# The pin must still be *reported* so the propagation path has something to act +# on — hence the output assertion, not just the exit code. R="$TEST_DIR/stale"; mk_repo "$R" write_pin "$R/.github/workflows/governance.yml" "governance-reusable.yml" "$C0" -run_case "ancient pin (400d, 5 behind) fails (window 2 commits / 14 days)" 1 "$R" \ +run_case_out "ancient pin (400d, 5 behind) passes with an advisory notice" 0 \ + '::notice.*outside the recency window' "$R" \ STALENESS_WINDOW_COMMITS=2 STALENESS_WINDOW_DAYS=14 +# ── 5b. KNOWN-BAD: pin predates a named defect fix -> hard fail at any age ─── +# The deny-list is what the window was only ever proxying for. Strictly better +# in both directions: it catches this, and it does not punish 5 above. +R="$TEST_DIR/knownbad"; mk_repo "$R" +write_pin "$R/.github/workflows/governance.yml" "governance-reusable.yml" "$C1" +run_case_out "pin predating a known-bad fix fails regardless of window" 1 \ + 'predates.*carries the' "$R" \ + STALENESS_WINDOW_COMMITS=9999 STALENESS_WINDOW_DAYS=9999 \ + STALENESS_KNOWN_BAD_BEFORE="governance-reusable.yml:$C3" + +# ── 5c. KNOWN-BAD outranks freshness: an IN-WINDOW pin is still rejected ───── +R="$TEST_DIR/knownbad-fresh"; mk_repo "$R" +write_pin "$R/.github/workflows/governance.yml" "governance-reusable.yml" "$C4" +run_case "in-window pin carrying a known defect still fails" 1 "$R" \ + STALENESS_KNOWN_BAD_BEFORE="governance-reusable.yml:$C5" + +# ── 5d. The fix commit itself is NOT denied (boundary is strict) ───────────── +R="$TEST_DIR/knownbad-boundary"; mk_repo "$R" +write_pin "$R/.github/workflows/governance.yml" "governance-reusable.yml" "$C3" +run_case "the fixing commit itself is not denied" 0 "$R" \ + STALENESS_WINDOW_COMMITS=9999 STALENESS_WINDOW_DAYS=9999 \ + STALENESS_KNOWN_BAD_BEFORE="governance-reusable.yml:$C3" + +# ── 5e. The deny-list is scoped per reusable, not applied globally ─────────── +R="$TEST_DIR/knownbad-scope"; mk_repo "$R" +write_pin "$R/.github/workflows/scorecard.yml" "scorecard-reusable.yml" "$C1" +run_case "a deny-list entry for another reusable does not fail this one" 0 "$R" \ + STALENESS_WINDOW_COMMITS=9999 STALENESS_WINDOW_DAYS=9999 \ + STALENESS_KNOWN_BAD_BEFORE="governance-reusable.yml:$C5" + # ── 6. UNKNOWN / forged SHA -> fail ───────────────────────────────────────── R="$TEST_DIR/forged"; mk_repo "$R" write_pin "$R/.github/workflows/governance.yml" "governance-reusable.yml" "$FORGED"