ci: fix nuttx builds failing with 'no space left on device'#4999
Open
matthargett wants to merge 1 commit into
Open
ci: fix nuttx builds failing with 'no space left on device'#4999matthargett wants to merge 1 commit into
matthargett wants to merge 1 commit into
Conversation
matthargett
requested review from
TianlongLiang,
lum1n0us and
yamt
as code owners
July 17, 2026 19:30
build_iwasm_on_nuttx ran as a `container:` job on the large apache-nuttx-ci-linux image and built google/bloaty (protobuf/abseil submodules) from source on every matrix leg for the size report. A container job pulls its image at job init, before any step can free space, and the runner's default free space plus that from-source build didn't fit, so legs intermittently failed with "no space left on device" (both at image pull and during the bloaty compile). This is flaky on main. - Drop `container:`; run on ubuntu-latest, remove the unused preinstalled toolchains (~25 GB) first, then run the NuttX configure + build inside the apache image via `docker run`. The image is pulled after disk is freed and everything writes to the mounted host workspace, so it fits (verified ~119 GB free on the runner). - Build bloaty once in a `build_bloaty` job and cache it, keyed on BLOATY_REF (bump the ref -> cache miss -> rebuild + refresh). The matrix restores the prebuilt binary instead of every leg recompiling bloaty and protobuf; the `bloaty -d compileunits --source-filter` output is unchanged. - Drop a duplicated wamr_config_option matrix entry (it was listed twice). Verified on a fork run: 119 GB free after cleanup, image pulls, all board matrix legs build and link, bloaty is built once + cached and the matrix restores it, and the size report prints - no "no space left on device".
matthargett
force-pushed
the
fix/nuttx-enospc-rework
branch
from
July 17, 2026 20:10
468ec5a to
82a6d39
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
compilation on nuttxis intermittently red onmainand across PRs — everybuild_iwasm_on_nuttxleg fails withno space left on device, in two places: the container image pull (Docker pull failed, the dominant mode) and during the build while compiling the from-sourcebloaty(protobuf/abseil).Root cause
It's a
container:job on the largeapache-nuttx-ci-linuximage, and it also buildsgoogle/bloaty(recursive protobuf/abseil submodules) from source on every matrix leg for the size report. A container job pulls its image at job init, before any step runs, so no step can free disk first — and the runner's default free space plus that from-source build doesn't fit.Fix
container:. Run onubuntu-latest, remove the unused preinstalled toolchains (~25 GB) as step 1, then run the NuttX configure+build inside the apache image viadocker runwith the workspace mounted. The image is pulled after disk is freed (verified ~119 GB free), so both modes are fixed.build_bloatyjob compiles the pinned bloaty in the image and caches the binary, keyed onBLOATY_REF— bump the ref and the cache key changes, so it rebuilds and refreshes automatically. The matrixneeds: build_bloatyand restores the prebuilt binary, so 63 protobuf builds collapse to one. The size report (bloaty -d compileunits --source-filter wamr/libc-builtin/libc-wasi) is byte-for-byte unchanged — the MCU binary-size regression signal is fully preserved.wamr_config_optionmatrix entry (it was listed twice), removing a redundant leg.Verification (on a fork)
/dev/root 145G 26G 119G 18% /; the apache image pulls; all board matrix legs build and link, no ENOSPC.build_bloatybuilds once andCache saved with key: bloaty-<ref>-ubuntu-latest; the matrix legsCache restored from key: bloaty-<ref>-...(restore, not rebuild) and the size report prints.CI cost
Measured on the fork runs: building bloaty (protobuf/abseil/…) from source adds ~3 min to every matrix leg (~8–9.5 min per leg vs ~5.7 min with a cache restore), and it was being done on all 56 legs. Doing it once (~7.6 min, and ~free on a warm cache) instead saves roughly 56 × 3 ≈ 170 Actions-minutes per run; dropping the duplicated matrix leg saves ~7 × 5.7 ≈ 40 more — about ~200 Actions-minutes per nuttx run.
At the current ~18 nuttx runs/week on this repo that's roughly ~3,600 Actions-minutes (~60 hours) per week (warm cache; a bloaty-ref bump adds one ~8-min rebuild that run). This is on top of no longer wasting minutes on the ENOSPC-failed runs.