ci: run evals weekly and record results on main

[angeloashmore]

Resolves:

Description

Runs the eval suite in CI: weekly on Monday 06:00 UTC, and on demand via workflow dispatch. Results land on main through a bot-opened PR the job squash-merges immediately, since main's ruleset requires changes to arrive by PR. The results PR body carries the run's report, so watching the repo's PRs doubles as a weekly report; run-over-run history is the git log of evals/results.json.

Eval failures do not fail the job; evals are measurements, not gates. The job fails only when evals/results.json was not rewritten (the commit is the gate), meaning the harness died before grading. The eval step never receives GH_TOKEN because the agent under test can read the step's environment.

Requires one new repository secret: ANTHROPIC_API_KEY (already set).

Checklist

• A comprehensive Linear ticket, providing sufficient context and details to facilitate the review of the PR, is linked to the PR.
• If my changes require tests, I added them.
• If my changes affect backward compatibility, it has been discussed.
• If my changes require an update to the CONTRIBUTING.md guide, I updated it.

Preview

How to QA ¹

After this merges: Actions → evals → Run workflow. The run takes roughly 10–15 minutes and about $31; confirm it opens and squash-merges a test(eval): record eval run at … PR whose body shows the per-eval report.

🤖 Generated with Claude Code

---

Note

Medium Risk
Grants CI write/PR permissions and uses production API and e2e secrets, but scope is limited to eval automation with deliberate token isolation from the agent step.

Overview
Adds a new evals GitHub Actions workflow that runs the existing npm run evals suite on a schedule (Mondays 06:00 UTC), on workflow_dispatch, and temporarily on pushes to aa/evals-workflow for pre-merge testing.

The job uses a concurrency group so runs don’t overlap on the shared e2e Prismic account, checks out with persist-credentials: false so the agent under test can’t use the checkout token to push, and keeps GH_TOKEN out of the eval step (only the record step gets it). Eval failures are non-blocking (|| true); recording on main only opens a bot PR that commits evals/results.json, embeds the evals:report output in the PR body, and squash-merges immediately to satisfy branch protection.

^{Reviewed by Cursor Bugbot for commit 68bb4ab. Bugbot is set up for automated code reviews on this repo. Configure here.}

Footnotes

1. Please use these labels when submitting a review:
   ❓ #ask: Ask a question.
   💡 #idea: Suggest an idea.
   ⚠️ #issue: Strongly suggest a change.
   🎉 #nice: Share a compliment. ↩

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1adbd64. Configure here.}