Add swarm dispatch scope and dispatch-mode benchmark harness - #199
Add swarm dispatch scope and dispatch-mode benchmark harness#199khaliqgant wants to merge 1 commit into
Conversation
Adds a new 'swarm' triage scope alongside single/team: a lead + workers share one checkout and a live relay channel (agent-relay MCP messaging, already fully plumbed via AgentSpec.channel -> SpawnInput.channel) instead of team's isolated parallel fan-out. Opt-in only via the agent:swarm label, never inferred heuristically. renderAgentTask now briefs the lead/workers with their coordination channel and each other's names. Adds benchmark/ machinery to actually measure whether team/swarm beat a single agent, and whether swarm's live collaboration beats team's isolated fan-out: a task corpus schema + loader, a resumable (task x mode x repeat) matrix builder, a real-dispatch runner (gh + factory dispatch + verify.sh scoring) behind a testable DispatchRunner interface, and a markdown report generator grouped by coordination-benefit difficulty tier. Pure logic is unit-tested; the real-IO adapter is proven by running it, not by mocking child_process. Includes a SWE-bench Verified adapter script (tested live against 2 real instances) for an externally-comparable subset, with loud caveats about repointing to a controlled fork before dispatching. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1jnbLuMdTR2hTWv9EUnkq
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
Warning Review limit reached
Next review available in: 47 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (32)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://cdn.github.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 42b5a86ba0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const channel = swarmChannel(issue) | ||
| return swarmMemberSlugs(maxImplementers).map((slug) => ({ | ||
| name: agentNameForRole(issue, 'impl', { repo: route.repo, discriminator: slug }), | ||
| role: 'implementer' as const, |
There was a problem hiding this comment.
Every swarm member is registered as an ordinary implementer on the same repository, checkout, and lifecycle branch. The existing exit path publishes a PR for whichever implementer finishes first, while the publication cache and #allImplementersHaveCompletionPr checks are repository/branch-based; consequently, an early worker exit can publish the partially edited shared checkout and make every swarm member appear complete, triggering the babysitter or releasing the still-working lead. Workers need to be non-publishing/non-terminal participants, or publication must wait for the lead after all workers have reported done or blocked.
Useful? React with 👍 / 👎.
| '--title', task.title, | ||
| '--body', task.issueBody, | ||
| '--label', 'factory', | ||
| '--label', label, |
There was a problem hiding this comment.
For each single-repository benchmark task, this creates an issue with only factory and agent:team labels. GitHub mirror routing supplies one repository route, and labelDerivedDispatchDecision creates team implementers by mapping those routes, so the team cell spawns exactly one implementer—the same cardinality as single. The resulting team-versus-single measurements therefore do not test multi-agent fan-out; the runner must provide multiple assignments or use a team mode that splits one repository task among multiple implementers.
Useful? React with 👍 / 👎.
| } | ||
| log(`created ${task.targetRepo}#${issueNumber} (${label})`) | ||
|
|
||
| await exec('node', ['bin/factory.mjs', 'dispatch', issueNumber, '--config', options.factoryConfigPath]) |
There was a problem hiding this comment.
The runner never uses task.baseRef before dispatching, so Factory creates its lifecycle branch from the configured checkout's default base instead. This is especially corrupting for SWE-bench tasks, where the adapter records the instance's exact base_commit: agents solve and tests score whatever revision happens to be current in the fork rather than the benchmark instance. Prepare/reset the dispatch checkout at baseRef and ensure the PR uses that base for every cell.
Useful? React with 👍 / 👎.
| } | ||
| log(`created ${task.targetRepo}#${issueNumber} (${label})`) | ||
|
|
||
| await exec('node', ['bin/factory.mjs', 'dispatch', issueNumber, '--config', options.factoryConfigPath]) |
There was a problem hiding this comment.
The created issue belongs to task.targetRepo, but dispatch receives only its numeric ID. When the benchmark config maps multiple target repositories—as required for a multi-repository SWE-bench subset—a bare number is rejected as ambiguous when it exists in multiple configured repos, or is resolved through repos.default to a different repository. Pass the mounted repository-qualified issue path rather than discarding targetRepo here.
Useful? React with 👍 / 👎.
| } | ||
|
|
||
| const allResults = await loadResults(RESULTS_PATH) | ||
| const rows = buildReportRows(tasks, allResults) |
There was a problem hiding this comment.
buildReportRows is always called without costSamples, and no preceding benchmark code records duration or cost, so every generated report necessarily prints n/a for both advertised metrics. This prevents the harness from measuring whether multi-agent modes improve latency or justify their extra spend; capture these values per run and pass them into the report aggregation.
Useful? React with 👍 / 👎.
| '', | ||
| `You are a SWARM WORKER collaborating with a lead (${others}) live in this SAME checkout.`, | ||
| `Watch the shared relay channel #${swarm.channel} for direction from the lead. Announce what you're starting, ask there if scope is ambiguous, and post when a subtask is done or blocked.`, | ||
| 'Do not open your own pull request or push to origin yourself — the lead integrates everyone\'s work and finishes it.', |
There was a problem hiding this comment.
This tells a swarm worker not to push, but renderAgentTask immediately appends the common instructions Commit the implementation and tests and Push the branch to origin. Because all swarm members share the same branch and checkout, workers following the later instruction can race the lead's commits or expose an incomplete branch. Render role-specific commit/push instructions so only the lead is asked to push.
Useful? React with 👍 / 👎.
2 participants
Change
Adds a new
swarmtriage scope alongsidesingle/team: a lead + workers share one checkout and a live relay channel (agent-relay MCP messaging — already fully plumbed viaAgentSpec.channel->SpawnInput.channel, just previously unpopulated) instead ofteam's isolated parallel fan-out. Opt-in only via theagent:swarmlabel, never inferred heuristically, so existingsingle/teambehavior is unchanged.renderAgentTasknow briefs the lead/workers with their coordination channel and each other's names.Also adds
benchmark/machinery to actually measure whetherteam/swarmbeat a single agent, and whether swarm's live collaboration beats team's isolated fan-out: a task corpus schema + loader, a resumable(task x mode x repeat)matrix builder, a real-dispatch runner (gh+factory dispatch+verify.shscoring) behind a testableDispatchRunnerinterface, and a markdown report generator grouped by coordination-benefit difficulty tier. Pure logic is unit-tested; the real-IO adapter is proven by running it (not by mockingchild_process) — seebenchmark/README.mdfor what's still required operationally before a real run (a disposable sandbox repo, authored tasks, a live workspace). Includes a SWE-bench Verified adapter script, tested live against 2 real instances, for an externally-comparable subset, with loud caveats about repointing to a controlled fork before dispatching.Verification
packagecheck passed on the current head SHA.factory-e2e-<head-sha>attestation artifact was inspected.Verification tier: Tier 1 (package boundary) — this PR adds new opt-in surface area (a new label-gated scope, a standalone
benchmark/tooling package) with no change to defaultsingle/teamdispatch behavior; no live/load evidence applies.Head SHA tested: 42b5a86
Adapter/preview evidence: N/A — no adapter or preview surface touched.
Live canary issue / PR / run: N/A —
swarmscope requires an explicitagent:swarmlabel to activate, so there is no live canary path until a real workspace opts in;benchmark/run.tsis the intended live-canary vehicle for a future run against a real sandbox repo, not run in this PR.Failure-injection or load evidence: N/A — no runtime/infra behavior change for existing scopes.
Cleanup receipt: N/A — no disposable resources created;
npm run verify:e2e's packed-consumer install/tarball are cleaned up by the script itself.npx tsc -p tsconfig.build.json --noEmit,npx tsc -p benchmark/tsconfig.json --noEmit, andnpx vitest runall pass (1422/1428, the 6 failures are pre-existing/environmental — git-sandbox remote-parsing issues and a missingdist/fixture — unrelated to this diff and reproduce onmain).Generated by Claude Code
Summary by cubic
Adds a new
swarmdispatch scope (lead + workers share one checkout and a live relay channel) gated by theagent:swarmlabel, and introduces a benchmark harness to measuresinglevsteamvsswarmwith real tasks and reports.New Features
agent:swarm; never inferred.renderAgentTaskbriefs roles and names;AgentSpec.channelandswarmRoleset.triage.maxImplementers.benchmark/):gh+factory dispatchand verifies via taskverify.sh(benchmark/run.ts).npm run benchmark.Migration
single/team/workflowbehavior.agent:swarmto an issue and ensuretriage.maxImplementers > 1.benchmark/tasks/targeting a disposable sandbox repo, then runnpm run benchmark -- --config ./factory.config.json(use--only-task,--only-mode,--repeatsto narrow).Written for commit 42b5a86. Summary will update on new commits.