feat: read evals from usage artifact instead of dedicated evals artifact#45797
Conversation
evals.jsonl is now included in the usage artifact (copied there by the conclusion job). Update the client-side code accordingly: - ArtifactSetEvals now resolves to the usage artifact name so that --artifacts evals downloads usage (which contains evals.jsonl) - applyEvalsArtifact no longer appends evals when usage is already in the artifacts list, since usage already provides evals - Thread evalsOnly bool through concurrentRunDownloadParams so the cache bypass in tryLoadCachedRunResult uses the explicit flag rather than inspecting the artifact filter for a name that no longer matches - Update --evals flag descriptions in audit and logs commands to reflect that evals are read from the usage artifact - Add TestResolveArtifactFilter case asserting evals→usage resolution Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
🤖 PR Triage
Score breakdown: Impact 28/50 · Urgency 20/30 · Quality 17/20 Rationale: Fixes evals artifact resolution regression —
|
|
@copilot download evals if not found in usage artifact |
…tifact Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Done. Added The fallback is triggered by a new |
|
🧠 Matt Pocock Skills Reviewer is reviewing this pull request using Matt Pocock's engineering skills... |
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. No ADR enforcement needed: PR #45797 does not have the 'implementation' label and has ≤100 new lines of code in business logic directories (99 additions, threshold is 100). |
|
✅ Test Quality Sentinel completed test quality analysis. |
|
🔎 PR Code Quality Reviewer is reviewing code quality for this pull request... |
There was a problem hiding this comment.
Pull request overview
Updates eval retrieval to use the usage artifact and adds legacy fallback handling.
Changes:
- Maps the evals artifact set to usage.
- Propagates eval-request context through download orchestration.
- Updates CLI descriptions and tests.
Show a summary per file
| File | Description |
|---|---|
pkg/cli/logs_run_processor.go |
Adds eval request tracking and legacy fallback. |
pkg/cli/logs_parallel_test.go |
Updates download helper calls. |
pkg/cli/logs_orchestrator.go |
Passes eval context to downloads. |
pkg/cli/logs_orchestrator_test.go |
Updates orchestrator test calls. |
pkg/cli/logs_orchestrator_stdin.go |
Propagates stdin eval options. |
pkg/cli/logs_command.go |
Updates the eval flag description. |
pkg/cli/logs_artifact_set.go |
Maps evals to usage and deduplicates selection. |
pkg/cli/logs_artifact_set_test.go |
Tests evals-to-usage resolution. |
pkg/cli/audit.go |
Updates audit messaging and comments. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 9/9 changed files
- Comments generated: 2
- Review effort level: Medium
| ensureEvalsResultsFromBranch(ctx, summary.Run, runOutputDir, params.dlOwner, params.dlRepo, params.dlHost, params.verbose) | ||
| if evalsRequested && !hasEvals { | ||
| logsOrchestratorLog.Printf("Cache bypass for run %d: evals artifact requested but not present locally", run.DatabaseID) | ||
| if params.evalsOnly && !hasEvals { |
| // evals: evals results are now included in the usage artifact. | ||
| ArtifactSetEvals: {constants.UsageArtifactName}, |
![github-actions[bot]](https://cdn.github.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review: feat: read evals from usage artifact instead of dedicated evals artifact
The overall approach is correct — remapping ArtifactSetEvals to resolve to the usage artifact and introducing a fallback for older runs is well-designed. One blocking bug was found.
Blocking issue: In tryLoadCachedRunResult (logs_run_processor.go:271), the cache bypass condition was changed from the old evalsRequested (which fired for both --evals and --artifacts evals) to params.evalsOnly (only fires for --evals). The PR already introduces evalsArtifactRequested that covers both cases and uses it correctly in processSingleRunDownload. The same field should be used in tryLoadCachedRunResult for consistency and correctness.
🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · 40.3 AIC · ⌖ 6.87 AIC · ⊞ 5K
| ensureEvalsResultsFromBranch(ctx, summary.Run, runOutputDir, params.dlOwner, params.dlRepo, params.dlHost, params.verbose) | ||
| if evalsRequested && !hasEvals { | ||
| logsOrchestratorLog.Printf("Cache bypass for run %d: evals artifact requested but not present locally", run.DatabaseID) | ||
| if params.evalsOnly && !hasEvals { |
There was a problem hiding this comment.
Bug: cache bypass uses evalsOnly instead of evalsArtifactRequested
The old code used evalsRequested := slices.Contains(params.artifactFilter, constants.EvalsArtifactName) which was true for both --evals and --artifacts evals. The new code replaces this with params.evalsOnly, which is only true when --evals is passed.
As a result, when a user passes --artifacts evals (without --evals), the cache bypass no longer fires, and a stale cache without evals.jsonl will be returned instead of triggering a fresh download.
The PR already introduces evalsArtifactRequested (which covers both cases) and uses it correctly in processSingleRunDownload line 218. The same field should be used here:
// before
if params.evalsOnly && !hasEvals {
// after
if params.evalsArtifactRequested && !hasEvals {@copilot please address this.
🧪 Test Quality Sentinel Report✅ Test Quality Score: 90/100 — Excellent
📊 Metrics (14 tests)
i️ Test Inflation Analysislogs_orchestrator_test.go: 14 lines added / 1 line in logs_orchestrator.go = 14:1 ratio (exceeds 2:1 threshold) Justification: All 13 new integration tests cover concurrent download behavior with careful attention to:
This is justified complexity: concurrency bugs are silent and hard to reproduce, requiring comprehensive testing of race conditions. Verdict
Strengths:
Notes:
Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
evals.jsonlis now included in the usage artifact (conclusion job copies it from the evals job). The client-side--evalsflag was still requesting a dedicatedevalsartifact download and the cache bypass was keyed on the artifact name, which broke silently after the mapping change.Changes
ArtifactSetEvals→ resolves tousage(logs_artifact_set.go):--artifacts evalsnow downloads the usage artifact.runHasEvalsalready probesusage/evals.jsonl; this aligns the download side with detection.applyEvalsArtifactskips whenusagealready present (logs_artifact_set.go): Thelogsdefault is--artifacts usage; previously--evalsappended a redundantevalsentry. Now it short-circuits whenusageis already in the list.Thread
evalsOnlythrough the download pipeline (logs_run_processor.go,logs_orchestrator.go,logs_orchestrator_stdin.go): The cache bypass intryLoadCachedRunResultwas checkingslices.Contains(artifactFilter, "evals")— always false after the mapping change. Replaced with an explicitparams.evalsOnlyfield propagated from the callers.Flag descriptions and comments updated in
audit.goandlogs_command.goto reflect the usage artifact as the evals source.Test: Added
ResolveArtifactFilter(["evals"]) == ["usage"]assertion; updated alldownloadRunArtifactsConcurrentcall sites with the newevalsOnlyparameter.