Cross-tool oracle comparison (Infer#/CodeQL) + related-work positioning#21
Merged
Conversation
Validate the leak check against mature C# analysers (Infer#, CodeQL) on the same code, and state honestly where Own.NET sits among them. - scripts/oracle_compare.py: diff Own.NET vs Infer#/CodeQL leak findings into agree / own-only / oracle-only buckets (plus Own.NET-only defect classes and oracle out-of-scope context). Reuses the miner's own-check parser; one SARIF parser for both oracles; basename+line matching (--line-tol); --json output; --selftest (9/9). - .github/workflows/oracle.yml: workflow_dispatch running own-check + CodeQL (build-mode none) + Infer# (built binaries) over a target, diffing to the run summary + an artifact. Each oracle step is continue-on-error -- the oracles need the target to build, our extractor does not. - docs/notes/oracle.md: methodology, the three buckets, how to read it, gaps. - README: "Related work / позиционирование" -- Infer#/CodeQL/IDisposableAnalyzers/ CA2000 are the recall bar and the oracle; our difference is borrow + region + double-dispose in one model. Closest same-idea-different-language is C++/Rust lifetime work (Polonius). - ci.yml: run the miner + oracle selftests on every push, not just on dispatch. Verified locally: ruff clean (whole tree), oracle selftest 9/9, miner 7/7, run_tests.py exit 0 (analysis 125/125, fuzz 3000, gallery/corpus/wpf/loops/spec/ ownir all green). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rg8kSk1YT14x7A1vo5zgED
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughAdds a cross-tool oracle comparison pipeline to Own.NET. A new ChangesOracle Cross-Tool Comparison Pipeline
Sequence Diagram(s)sequenceDiagram
participant Trigger as workflow_dispatch
participant Selftest as oracle_compare.py --selftest
participant OwnCheck as scripts/own-check.sh
participant CodeQL as github/codeql-action
participant InferSharp as microsoft/infersharpaction
participant Comparator as oracle_compare.py
participant Summary as GITHUB_STEP_SUMMARY
Trigger->>Selftest: validate comparator fixtures
Selftest-->>Trigger: pass/fail
Trigger->>OwnCheck: scan .cs sources → own.txt
Trigger->>CodeQL: init + analyze C# (build-mode: none) → codeql.sarif
Trigger->>InferSharp: dotnet build → _bin, run → infer.sarif
OwnCheck-->>Comparator: own.txt
CodeQL-->>Comparator: codeql.sarif (if present)
InferSharp-->>Comparator: infer.sarif (if present)
Comparator->>Summary: publish report.md + upload oracle-report artifact
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://cdn.github.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/oracle.yml:
- Around line 26-27: The git clone operation using the `--branch` flag does not
accept raw commit SHAs, which conflicts with the documented input specification
that claims to accept branch, tag, or sha. Modify the git clone logic to handle
all three ref types properly by replacing the `git clone --branch "$REF"`
approach with a method that supports commit SHAs, such as cloning without the
branch flag and then using `git checkout` with the ref value, or using `git
fetch` with the ref and then checking it out. Ensure the updated logic works for
branches, tags, and commit SHAs as originally documented in the input
description.
In `@scripts/oracle_compare.py`:
- Around line 131-136: The `parse(text)` call on line 131 returns a tuple of
(findings, unparsed) but the unparsed portion is being discarded with an
underscore assignment, causing silent data loss if the Own.NET parser encounters
unrecognized line formats. Instead of ignoring the unparsed return value,
capture it in a variable and add a check that raises an exception or logs a
warning if any unparsed findings exist, surfacing parser drift issues rather
than silently dropping findings and skewing the comparison results.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 6340d09e-be7b-4d5c-bcb3-6eb2a74e3d2d
📒 Files selected for processing (5)
.github/workflows/ci.yml.github/workflows/oracle.ymlREADME.mddocs/notes/oracle.mdscripts/oracle_compare.py
- oracle.yml: clone then fetch + checkout the ref, so a raw commit SHA works as the `ref` input documents (was `git clone --branch`, which only takes a branch/tag and would fail a SHA dispatch). - oracle_compare.py: surface own-check parser drift instead of dropping it. build_own now returns the unparsed-line count; main warns on stderr and the report header flags it. Chose a non-fatal note over a hard raise: this is eval tooling run over third-party repos, so one stray stdout line shouldn't sink the whole comparison (the miner surfaces its unparsed count the same way). Selftest is now 12/12 and covers the drift path. Verified: ruff clean (whole tree), oracle selftest 12/12, run_tests.py exit 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rg8kSk1YT14x7A1vo5zgED
This was referenced Jun 17, 2026
This was referenced Jun 24, 2026
oracle: fix dotted-test-dir filter + dispose-optional StringWriter/StringReader (+ field notes)
#147
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things: where we sit, and how we check ourselves against the neighbours
Follow-up to the Dapper mining work (#19/#20). We found zero real leaks in
Dapper — a precision win — but a clean run on its own can't tell "correctly found
nothing" from "silently skipped everything". This PR adds an external reference
for exactly that, and writes down honestly where Own.NET stands.
(a) README — "Related work / позиционирование"
A grounded section: we are not the first or best leak detector for C#. Infer#,
CodeQL (
cs/local-not-disposed), IDisposableAnalyzers, CA2000/CA2213 are therecall bar — interprocedural, battle-tested, no "honest skip". Our difference is
model, not coverage: borrow + region + double-dispose / use-after-dispose
in one ownership model, which the leak-only queries don't express. Closest
same-idea-different-language is C++/Rust lifetime work (Polonius,
C++ Lifetime profile, Clang's experimental lifetime-safety) — and their Datalog
fact vocabulary is essentially our OwnIR.
(b) Cross-tool oracle harness
Run Own.NET, Infer#, and CodeQL over the same repo and diff the leak-class
findings:
Plus Own.NET-only defect classes (OWN002/OWN003 — no oracle leak-query
equivalent) and oracle out-of-scope hits (null-deref, taint…) as context.
scripts/oracle_compare.py— the diff core. Reuses the miner'sown-checkparser; one SARIF parser for both oracles (Infer# and CodeQL both emit SARIF);
basename + line-window matching (robust to cross-tool path prefixes);
--json;--selftest(9/9, embedded fixtures)..github/workflows/oracle.yml—workflow_dispatchrunning own-check +CodeQL (
build-mode: none) + Infer# (built binaries) over a target, diffing tothe run summary + an artifact. Each oracle step is
continue-on-error: theoracles need the target to build, our extractor doesn't — that asymmetry is
the point.
docs/notes/oracle.md— methodology, the three buckets, how to read it, gaps.ci.yml— runs the miner + oracle selftests on every push (not just on dispatch).Honesty
The comparator is unit-tested and runs anywhere on pre-produced outputs. The
tri-tool orchestration is validated on dispatch, same as
mine.yml(no localInfer#/CodeQL here). Agreement raises confidence; it doesn't prove soundness
(that's the Boogie/Dafny backend, still roadmap).
Verified locally
ruffclean (whole tree) · oracle selftest 9/9 · miner selftest 7/7 ·run_tests.pyexit 0 (analysis 125/125, fuzz 3000, gallery/corpus/wpf/loops/spec/ownir all green).
🤖 Generated with Claude Code
https://claude.ai/code/session_01Rg8kSk1YT14x7A1vo5zgED
Generated by Claude Code
Summary by CodeRabbit