Evals - #170
Draft
davidabram wants to merge 27 commits into
Draft
Conversation
Replace the broad agent prompt set with a focused /next-task workflow that separates readiness review, implementation approval, task execution, and durable context reconciliation. Add structured contracts, strict plan argument handling, and explicit blocked and continuation guidance across workflow handoffs. Co-authored-by: SCE <sce@crocoder.dev>
Split durable context loading from plan authoring, add structured contracts for both phases, and strengthen post-task context reconciliation. Keep the implementation gate explicit while ensuring /change-to-plan stops after producing an implementation-ready plan. Co-authored-by: SCE <sce@crocoder.dev>
Introduce /validate as a report-only final verification phase and add a separate plan context synchronization skill that runs only after validation passes. Structured result and report references keep failure, blocking, and success handoffs unambiguous.
Replace the shared-content catalog with canonical packages for /change-to-plan, /next-task, and /validate. Generate self-contained Pi, OpenCode, and Claude skills from workflow-owned Pkl, remove the automated and legacy workflow trees, and enforce the new inventory in parity checks. Plan: rebuild-pkl-workflow-markdown (T01, T02, T03, T04, T05, T06, T07, T08, T09) Co-authored-by: SCE <sce@crocoder.dev>
Stop compiling against committed generated trees. Evaluate canonical Pkl into Cargo OUT_DIR for repository builds, provide a deterministic packaging fallback for published and sandboxed builds, and align Nix, crates.io, and Flatpak paths with the ephemeral payload. Plan: generate-cli-assets-in-cargo-out-dir (T01, T02, T03) Co-authored-by: SCE <sce@crocoder.dev>
Materialize the canonical workflow packages at the repository root for local OpenCode and Claude use while removing superseded agents, commands, and skills from those trees.
Define durable rules for context loading, planning, implementation approval, task synchronization, and final validation. Link the new contract from the context map and align existing workflow guidance with it. Plan: canonicalize-context-workflow-rules (T01)
Generate a fresh payload before repository Cargo commands and teach Nix and build.rs to consume an explicit generated-input directory. Add wrapper tests and update build contracts so Crane no longer evaluates Pkl independently in each Cargo derivation. Plan: generate-cli-assets-in-cargo-out-dir (T04, T05, T06) Co-authored-by: SCE <sce@crocoder.dev>
Reduce task sync output to the durable changes, unresolved gaps, and next workflow command that users need. Update the canonical Pkl template and all rendered skill references to share the same concise contract. Plan: simplify-task-context-sync-report (T01) Co-authored-by: SCE <sce@crocoder.dev>
Reintroduce /commit as a fourth canonical workflow with explicit ownership: the command handles staging and the single commit, while sce-atomic-commit analyzes staged changes and proposes repository-style messages. Render the self-contained package for Pi, OpenCode, and Claude and enforce the expanded workflow inventory. Plan: commit-workflow (T01, T02, T03) Co-authored-by: SCE <sce@crocoder.dev>
Add --bootstrap-context for context-only initialization and ensure every successful setup path creates missing durable context files without replacing existing content. Document the setup contract and cover the new command surface and filesystem behavior. Plan: setup-bootstrap-context (T01, T02) Co-authored-by: SCE <sce@crocoder.dev>
Render structured Claude phase results as human-facing Markdown, then fold machine-facing contract details directly into each SKILL.md so package-local references remain presentation-only. Align command handoffs, parser behavior, metadata coverage, generated skills, and durable architecture guidance with the single-reference package model. Plan: claude-markdown-workflow-results (T01) Plan: skill-references-human-presentation-only (T01, T02) Co-authored-by: SCE <sce@crocoder.dev>
Collapse Claude workflow orchestration into four self-contained skills with one output reference each, while preserving the decomposed Pi and OpenCode packages. Co-authored-by: SCE <sce@crocoder.dev>
Workflow composition lived in `claude-workflow-results.pkl`, which imported Claude metadata directly and hardcoded `compatibility: claude` and the `allowed-tools:` frontmatter line into its skill and command renderers. Pi and OpenCode cannot reuse it in that shape, so collapsing them to workflow-level packages would mean duplicating the phase internalization tables and output layouts per target and guaranteeing drift between texts meant to be identical. Rename the module to `workflow-composite.pkl` and make it target-neutral: `ClaudeWorkflow` becomes `CompositeWorkflow`, the command-to-workflow-skill slug mapping moves here from `claude-metadata.pkl`, and `renderSkill`, `renderCommand`, and `skillDocuments` each take a newline-terminated `extraFrontmatterLines` string carrying only the frontmatter a target supports. Claude now passes `compatibility: claude` and its `allowed-tools` line at the call site and re-exports the shared slug mapping, so its generated bytes are unchanged while Pi and OpenCode gain a consumable entrypoint. Co-authored-by: SCE <sce@crocoder.dev>
Pi consumed the canonical phase packages directly, emitting four prompts carrying full command bodies plus eight self-contained phase-skill packages. That left Pi on a different composition model than Claude, even though neither target adds target-specific frontmatter, so the composite renderer was duplicated work waiting to happen. pi-content.pkl now renders through the shared workflow-composite.pkl, passing an empty extra-frontmatter string for both prompts and skills. Pi emits four thin prompts, each routing to exactly one workflow skill, and four packages containing only SKILL.md and references/output.md. The coverage check drops its Claude-specific naming: the expected skill mapping and document paths become target-neutral, assertClaudeCommandRoute becomes assertCommandRoute(target, slug, document), and four Pi route checks are added so Pi prompts are held to the same one-skill routing rule as Claude commands. OpenCode keeps the canonical phase packages and byte-identical output. Co-authored-by: SCE <sce@crocoder.dev>
OpenCode was the last target still emitting the eight canonical phase packages and per-command phase chains, so its commands carried multi-step `skills` lists and its agents allowlisted every phase skill individually. That kept two divergent shapes alive across targets and made the phase packages a generated output surface rather than an authoring input. Route opencode-content.pkl through the shared workflow-composite module: each command now derives a one-entry `entry-skill`/`skills` pair from `workflowSkillSlugByCommand`, and skill documents render as four two-file packages with OpenCode metadata only on the entrypoint. The agent permission allowlists narrow to the four workflow skills. The coverage check switches OpenCode to `expectedWorkflowSkillDocumentPaths`, adds per-command route assertions, and drops the now-unused `expectedCanonicalSkillDocumentPaths` set. Co-authored-by: SCE <sce@crocoder.dev>
Durable context still described Pi and OpenCode as consuming eight self-contained phase-skill packages with YAML phase-result contracts, and scoped the composed single-skill model to Claude alone. That stopped being true once all three targets began rendering through the shared workflow composite renderer, leaving the decision records, overview, architecture, glossary, context map, and the five context/sce workflow documents contradicting the generated output. Add a decision record establishing four thin entrypoints plus four two-file workflow packages for every target, one shared composition path, and no phase package or phase-result transport anywhere. Mark the superseded scope inline in the 2026-07-27 and 2026-07-29 Claude records and in the plan/code agent-separation record, and rewrite the durable files so the eight phase names denote canonical authoring modules and internal phases rather than an installed surface. Also narrow the described OpenCode routing-agent skill allowlists to the four workflow slugs. Co-authored-by: SCE <sce@crocoder.dev>
…ackages Co-authored-by: SCE <sce@crocoder.dev>
Add a typed workflow catalog for shared identity and routing metadata. Enforce the exact 46-artifact inventory, reject stale workflow references, and cover failure cases with negative fixtures. Remove the stray rendered evaluation artifact and correct documentation for the shared four-package target model without changing generated output. Co-authored-by: SCE <sce@crocoder.dev>
Replace prose-based workflow internalization with structured rendering and named semantic sections. Centralize deterministic Pkl generation, input discovery, inventories, and handoff publication behind one producer contract. Co-authored-by: SCE <sce@crocoder.dev>
Prevent agents from inventing, skipping, reordering, or merging workflow steps, and from wrapping defined output layouts with extra prose. Apply the shared preamble rules across all generated target skill packages. Co-authored-by: SCE <sce@crocoder.dev>
…tom documents Co-authored-by: SCE <sce@crocoder.dev>
Add a standalone cross-target sce-decision package without exposing a new command or prompt. Allow next-task and validate to invoke it only for qualifying system-wide decisions after successful execution or validation. Enforce immutable accepted ADRs, propagate decision blockers, report written or reused ADR paths, and extend generation contracts to cover the 52-file artifact inventory and narrow sibling-skill exception. Co-authored-by: SCE <sce@crocoder.dev>
…flow Add a dual-mode /handover workflow — writer mode captures the current session and repository state as a durable handover document under context/handovers/; loader mode reads one back for continuation in another session. This restores the surface removed by the 2026-07-27 workflow-oriented Pkl generation cutover, on top of the current thin-command/self-contained-skill architecture. Co-authored-by: SCE <sce@crocoder.dev>
config/pkl/base/workflow-*.pkl left a blank source line before several packageOnlyBlock/compositeOnlyBlock interpolations. Under composite render mode, the block renders empty and the blank source line plus its own "\n\n" separator produced 2-3 blank lines instead of one across generated .claude/.opencode/.pi SKILL.md files. Remove the redundant blank lines at each call site. generation-contract-check.pkl had no guard against a stray multi-blank-line join or an inlined layout duplicating a sibling references/output.md section regressing back in. Add no-blank-line-runs and output-dedup contract checks with checked-in negative fixtures (blank-line-run-check.pkl, output-dedup-check.pkl), wire them into check-generated.sh, and document both checks in context/architecture.md and context/patterns.md. Co-authored-by: SCE <sce@crocoder.dev>
The repository generates SCE payloads for Claude Code, Pi, and OpenCode but carried no guidance for writing Effect code, leaving Effect v4 API usage and production defaults to be re-derived per task. Add an `effect` skill with a branch chooser that routes to eight topic references — schema, services and layers, config, scheduling, caching, streams, HTTP clients, and testing — so only the relevant material is read. The skill defers to project-local conventions and pins its examples to the Effect version documented in this repository. The skill is installed identically under `.claude/skills/`, `.pi/skills/`, and `.opencode/skills/`, keeping the three targets in parity.
Generated workflow prose is currently checked only structurally: `pkl-check-generated` asserts artifact paths and metadata, and the generation contract check asserts rendered-Markdown shape. Nothing verifies that an agent reading those documents actually honors the workflow's gates. Plan an `evals/` Effect v4 package that runs a workflow case against Claude Code, Pi, and OpenCode in a disposable fixture repository under `agent-run`, normalizes each run into one decoded result record, and grades it deterministically in two layers: structural conformance sourced from a single checked-in contract profile, and per-case required/forbidden facts. An LLM-judge quality score is an explicit non-goal. Scope is one workflow, `/change-to-plan`, with three cases including a negative case that passes only on refusal-with-questions. The runner stays extensible to the remaining canonical workflows. Co-authored-by: SCE <sce@crocoder.dev>
davidabram
marked this pull request as draft
July 30, 2026 21:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.