Python coverage silently lost when test runs under an instrumented py_binary --run_under wrapper

[maxbirkner]

🐞 bug report

Affected Rule

The issue is caused by coverage collection shared by py_test and py_binary (python/private/stage2_bootstrap_template.py, _maybe_collect_coverage).

Is this a regression?

Not a clean regression — the fixed pylcov.dat filename has been present for a long time. The bug only surfaces when two instrumented Python processes share one COVERAGE_DIR, which most setups never hit.

Description

When bazel coverage runs a Python test under a --run_under wrapper that is itself a py_binary, both the wrapper and the test it spawns are instrumented by rules_python (configure_coverage_tool = True). Both processes inherit the same COVERAGE_DIR and both write their lcov output to the same fixed filename:

# python/private/stage2_bootstrap_template.py (_maybe_collect_coverage)
lcov_path = os.path.join(coverage_dir, "pylcov.dat")
...
cov.lcov_report(outfile=lcov_path, ignore_errors=True)

If the wrapper runs the real test via subprocess.run(...) (rather than os.execv) so that it can post-process the child's output, the wrapper's own coverage is finalized after the child exits. The wrapper executes no instrumented test code, so its report is empty (0 lines hit). Because cov.lcov_report() opens the output file with open(path, "w") (truncating), the wrapper overwrites the child's real pylcov.dat with an all-zero report.

Net effect: the test's real Python line coverage is silently replaced by zeros. The build still passes; the coverage numbers are just wrong (every line shows as not hit).

Root cause: pylcov.dat is a fixed filename inside a shared COVERAGE_DIR. Whenever more than one instrumented Python process writes to that directory, the last writer wins and clobbers the others. A py_binary used as a --run_under wrapper that subprocess.runs the test is the most natural way to hit this, but any instrumented Python process that spawns another instrumented Python process (and shares COVERAGE_DIR) is affected.

Proposed fix: Write each process's lcov to a unique filename, e.g.:

lcov_path = os.path.join(coverage_dir, "pylcov_{}.dat".format(unique_id))

where unique_id is the per-process UUID already generated a few lines above for the coverage rcfile. Bazel's lcov_merger already merges all .dat files in COVERAGE_DIR and unions them, so the wrapper's empty report and the child's real report combine correctly and the real coverage is preserved. This is a minimal, general fix that does not depend on the wrapper scenario.

Related:

• Coverage now fails if there are no covered Python files even if test succeeds #2762 — Coverage now fails if there are no covered Python files even if test succeeds. (Distinct bug — that is the NoDataError crash; this report is about the silent clobber. The wrapper scenario tends to trigger both.)
• Option for coverage lcov to ignore empty database coveragepy/coveragepy#2054 — Option for coverage lcov to ignore empty database.

🔬 Minimal Reproduction

A self-contained, runnable reproduction is available here:
https://github.com/maxbirkner/rules_python-coverage-run-under-repro

It is a standalone bzlmod project (rules_python 1.7.0, Python 3.13, stdlib only) with a py_library, a py_test, and a py_binary wrapper used as --run_under. Clone it, build, and observe the clobber, then switch to the fix branch (which applies the unique-filename patch via single_version_override) and rerun to see the real coverage restored:

git clone https://github.com/maxbirkner/rules_python-coverage-run-under-repro.git
cd rules_python-coverage-run-under-repro

# 1) GOOD — no wrapper: the test's real line coverage is recorded.
bazel coverage //:lib_test --nocache_test_results --test_output=all
awk '/^SF:/{f=$0} /^L[FH]:/{print f"  "$0}' bazel-testlogs/lib_test/coverage.dat | grep lib.py
# => SF:lib.py  LH:12   (real coverage)

# 2) BUG — same test under a py_binary --run_under wrapper: coverage is clobbered.
bazel coverage --config=run_under_wrapper //:lib_test --nocache_test_results --test_output=all
awk '/^SF:/{f=$0} /^L[FH]:/{print f"  "$0}' bazel-testlogs/lib_test/coverage.dat | grep lib.py
# => SF:lib.py  LH:0    (clobbered, but the test still PASSES)

# 3) FIX — the fix branch patches rules_python to use a unique pylcov_<uuid>.dat.
git checkout fix
bazel coverage --config=run_under_wrapper //:lib_test --nocache_test_results --test_output=all
awk '/^SF:/{f=$0} /^L[FH]:/{print f"  "$0}' bazel-testlogs/lib_test/coverage.dat | grep lib.py
# => SF:lib.py  LH:12   (real coverage restored, even with the wrapper)

The essential ingredients are:

1. A py_binary wrapper that runs its arguments as a subprocess and stays alive afterwards (so it cannot exec), e.g. to inspect the child's output:

   import subprocess, sys
   def main(argv):
       result = subprocess.run(argv[1:])
       # ... post-process here (this is why we can't exec) ...
       raise SystemExit(result.returncode)
   if __name__ == "__main__":
       main(sys.argv)

2. A normal py_test with real, covered source.

3. Running with the wrapper as --run_under and coverage enabled:

   bazel coverage --run_under=//path:wrapper //path:my_test
   

Observed: my_test shows LH:0 for its source files.
Expected: my_test shows its real line hits.

Removing --run_under (or making the wrapper a non-Python target) restores correct coverage.

🔥 Exception or Error

No exception or crash — the failure is silent. The test PASSES and `bazel coverage`
succeeds, but the generated lcov shows every line of the test's source as not hit:

  SF:lib.py
  ...
  LH:0
  LF:11
  end_of_record

Expected the real line hits (e.g. LH:12) instead of LH:0.

🌍 Your Environment

Operating System:

  
  Linux
  

Output of bazel version:

  
  Bazelisk version: v1.20.0
  

Rules_python version:

  
  1.7.0
  

Anything else relevant?

• coverage.py: 7.6.12
• Python: 3.13
• We patch stage2_bootstrap_template.py downstream as a workaround to use a unique
  pylcov_<uuid>.dat filename (plus, separately, tolerate NoDataError per Coverage now fails if there are no covered Python files even if test succeeds #2762 and
  strip invalid BRDA:0, records that lcov >= 2.0 rejects).

[maxbirkner]

Just noticed this was fixed in

• fix(coverage): handle nested coverage collection #3823

two days ago 😆 🥳 .