From 3c0de598421255e365ce1ca7f2b22a075fd758ba Mon Sep 17 00:00:00 2001 From: Carr1005 Date: Sun, 26 Jul 2026 17:41:20 +0800 Subject: [PATCH] Extract git-repo ingestion into a standalone /ingest-course-repo skill PR #2 bolted an optional repo-ingestion step onto /generate-spec. The capability is useful; the placement was not. It forced generate-spec to carve out an exception in its own "Don't write to materials/" rule, and made a single-purpose skill do two jobs. Every other skill in the loop is single-purpose and hands off. This extracts it rather than reverting it: the script moves with its history (git mv), the prose moves largely intact, and the shortcut keeps working behind its own front door. /generate-spec is materials -> spec.md again, with a flat write invariant; /ingest-course-repo is the sole writer into materials/. Two bugs fixed on the way: - Silent clobber. The prescribed --out was byte-identical to the name of a hand-downloaded context dump (-context.md), and the script overwrites without asking. Output now lands in materials/notebooks/from-repo/, so the two coexist. - The target repo was un-ingestible. deeplearningai-eng/courses is a 3.4 GB monorepo holding ~260 courses with no path scoping in the script, so ingesting it would have cloned everything and merged every course into one context file. ingest_repo.py gains an optional --subdir that scopes discovery and switches the clone to --filter=blob:none --sparse with GIT_LFS_SKIP_SMUDGE=1. Pulling course_10 now transfers 8.7 MB in ~8s. Omitted, behaviour is byte-identical to before, so standalone course repos are unaffected. The skill parses a GitHub tree URL into repo + --ref + --subdir, and refuses the bare monorepo root with a pointer to browse for the course folder. Framing throughout is corrected from PR #2's "skip the manual notebook download" to what it actually is: an optional supplement that adds the course's real code and helper.py on top of the site download, which remains the main path. Verified: backward compatibility against the pre-change script on a local fixture (byte-identical); subdir scoping; sparse clone size; bad-subdir and transcripts-path guards; no clobber of a hand-downloaded file; tree-URL parse producing the same output as hand-typed flags; both refusal paths firing without invoking git. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/skills/generate-spec/SKILL.md | 38 ++----- .claude/skills/ingest-course-repo/SKILL.md | 100 ++++++++++++++++++ .../scripts/ingest_repo.py | 65 +++++++++--- .claude/skills/new-course/SKILL.md | 10 +- AGENTS.md | 18 +++- CLAUDE.md | 18 +++- README.md | 77 +++++++++----- 7 files changed, 244 insertions(+), 82 deletions(-) create mode 100644 .claude/skills/ingest-course-repo/SKILL.md rename .claude/skills/{generate-spec => ingest-course-repo}/scripts/ingest_repo.py (84%) diff --git a/.claude/skills/generate-spec/SKILL.md b/.claude/skills/generate-spec/SKILL.md index 0c7d719..ab90557 100644 --- a/.claude/skills/generate-spec/SKILL.md +++ b/.claude/skills/generate-spec/SKILL.md @@ -9,15 +9,11 @@ description: > from this course", "turn the materials into a spec", "build the spec out of materials/", "regenerate spec.md", "run the spec generation", "I've added the notebooks, please create the spec", or "/generate-spec". - Optionally accepts a git repo (SSH URL or local path) as an argument to - auto-ingest the notebooks + helper.py into materials/notebooks/ before - generating; without an argument it uses whatever is already in materials/. Reads every file under materials/notebooks/ and materials/transcripts/ and follows references/spec-generation-guide.md. Writes spec.md in the current working directory. Must run from inside a course folder (courses//). Do NOT use this skill for editing an existing spec.md or for writing product specs unrelated to a course. -argument-hint: "[repo-ssh-url]" allowed-tools: [Read, Write, Glob, Grep, Bash] --- @@ -32,36 +28,21 @@ another engineer (or coding agent) can implement end-to-end. cwd: `materials/notebooks/`, `materials/transcripts/`, `builds/`, `evals/`. If any are missing, refuse with: "Run /generate-spec from inside a course folder — `cd courses/` first." Stop. -2. **(Optional) Ingest notebooks + helper.py from a git repo.** If the user - supplied a repo argument — an `ssh://…`, `git@…:…`, `https://…`, or a local - path / `file://` dir — run the deterministic ingest script from the course - folder BEFORE reading materials: - ```bash - python3 "$CLAUDE_PROJECT_DIR/.claude/skills/generate-spec/scripts/ingest_repo.py" \ - --out "materials/notebooks/-context.md" [--ref ] - ``` - Use the course-folder name (cwd basename) for ``. The script clones - over the user's SSH key, renders every `.ipynb` cell-by-cell plus a - de-duplicated `## Helper Module Context` section, and writes ONLY into - `materials/notebooks/`. Relay its one-line summary. If it exits non-zero - (clone/SSH failure, nothing to ingest, or a bad `--out`), surface its error - and Stop. **If no repo argument was given, skip this step entirely and use - whatever is already in `materials/`.** Transcripts are NEVER produced here — - they remain a manual drop under `materials/transcripts/`. -3. **Read the guide.** Read +2. **Read the guide.** Read `.claude/skills/generate-spec/references/spec-generation-guide.md` relative to `$CLAUDE_PROJECT_DIR` (anchor on the repo root, not cwd). Follow it fully — it is the prescriptive contract for the spec's content, structure, and quality. Do not restate or summarize its rules here; defer to the guide so the two files can't drift. -4. **Read every material.** Glob `materials/notebooks/**/*` and +3. **Read every material.** Glob `materials/notebooks/**/*` and `materials/transcripts/**/*`, then read each non-empty file. If both directories are empty (only `.gitkeep`), refuse with: "No materials found. Add notebooks to materials/notebooks/ and transcripts to - materials/transcripts/, then re-run." Stop. (After a Step 2 ingest, the - freshly written `-context.md` is picked up here.) -5. **Write spec.md.** Produce a self-contained build-ready spec at + materials/transcripts/, then re-run — or optionally run + /ingest-course-repo to add the course repo's + notebooks and helper.py on top." Stop. +4. **Write spec.md.** Produce a self-contained build-ready spec at `./spec.md` in cwd (overwrite if it exists — this is regeneration, not editing). What the spec must contain and how it handles learner input are governed entirely by the guide — don't duplicate those rules here. @@ -72,15 +53,12 @@ another engineer (or coding agent) can implement end-to-end. conversation. It's a UX hint, not a system requirement (the slice can also be controlled with `--until=""`, and falls back to end-of-transcript if no closing match is found). -6. **Report.** Tell the user the path written and that they can review, +5. **Report.** Tell the user the path written and that they can review, then run /prepare-build when ready. ## Don't -- Don't write to `builds/`, `evals/`, or `materials/` yourself. The Step 2 - ingest script is the ONLY permitted writer into `materials/`, and only into - `materials/notebooks/`. Never hand-write or edit anything under `materials/`, - and never touch `materials/transcripts/`. +- Don't write to `builds/`, `evals/`, or `materials/`. - Don't pull in external knowledge to fill gaps in the materials — faithfulness to the course IS what /eval-materials-vs-build measures later. diff --git a/.claude/skills/ingest-course-repo/SKILL.md b/.claude/skills/ingest-course-repo/SKILL.md new file mode 100644 index 0000000..4480855 --- /dev/null +++ b/.claude/skills/ingest-course-repo/SKILL.md @@ -0,0 +1,100 @@ +--- +name: ingest-course-repo +description: > + Adds a course repo's notebooks and helper modules to a course's + materials/ in the spec-build-lab, rendered into + materials/notebooks/from-repo/-context.md. Use this skill ANY + time the user wants to pull course code out of a git repo — including + phrasings like "ingest the course repo", "pull the notebooks from this + repo", "here's the course repo", "add the course code to materials", + "get the notebooks from git", or "/ingest-course-repo ". Takes a + course-scoped repo URL: a GitHub tree URL pointing at one course + folder, a plain SSH/HTTPS repo URL, or a local path. It is an OPTIONAL + supplement to the manual material download, never a replacement — it + adds the course's real code and helper.py on top, and NEVER produces + transcripts. Must run from inside a course folder (courses//). + Do NOT use this skill to write spec.md — that's /generate-spec. +argument-hint: +allowed-tools: [Bash, Read, Glob] +--- + +# ingest-course-repo + +Render a course repo's `.ipynb` files and helper module(s) into +`materials/notebooks/from-repo/-context.md`. + +This is an **extra**, not a shortcut. The main way materials arrive is still a +manual download from https://course-context-lab.vercel.app — both the notebooks +`.md` and the transcripts. What this skill adds on top is the course's real, +runnable code and its `helper.py` definitions, which the site dump doesn't +carry. Transcripts are never produced here. + +## Steps + +1. **Verify cwd is a course folder.** Confirm all of these exist relative to + cwd: `materials/notebooks/`, `materials/transcripts/`, `builds/`, `evals/`. + If any are missing, refuse with: "Run /ingest-course-repo from inside a + course folder — `cd courses/` first." Stop. +2. **Require the repo URL.** If `$ARGUMENTS` is empty, say: "Which course repo? + If it's a DeepLearning.AI course, find it at + https://github.com/deeplearningai-eng/courses and paste the URL of the + course's folder — e.g. `.../courses/tree/main/course_10`." Stop until they + answer. +3. **Parse the URL.** Strip any `?query` or `#fragment` first, then: + + | Input | Becomes | + |---|---| + | `https://github.com///tree//` | repo `https://github.com//`, `--ref `, `--subdir ` | + | `https://github.com//` | repo as given, no `--subdir` | + | `git@…:…`, `ssh://…`, local path, `file://…` | repo as given, no `--subdir` | + + Strip only the `/tree//` tail; don't add or remove a `.git` + suffix. The repo string lands in the generated file's provenance line, so it + should read back as what the user actually pasted. + + A `/blob/` URL points at a file — ask for the folder URL instead. Splitting + `tree//` assumes the ref is a single segment; if the branch name + contains a `/`, that guess is wrong, so let the user correct it and pass + `--ref` / `--subdir` explicitly. +4. **Refuse the courses index.** If the input resolves to + `deeplearningai-eng/courses` with no subdirectory, refuse with: "That's the + courses index — 262 courses, 3.4 GB. Browse to your course and paste its + folder URL, e.g. `.../courses/tree/main/course_10`." Stop. Any other repo + root is fine and proceeds without `--subdir`. +5. **Run the ingest script** from the course folder: + ```bash + python3 "$CLAUDE_PROJECT_DIR/.claude/skills/ingest-course-repo/scripts/ingest_repo.py" \ + --out "materials/notebooks/from-repo/-context.md" \ + [--ref ] [--subdir ] + ``` + Use the course-folder name (cwd basename) for ``. With `--subdir` + the clone is sparse and blobless, so only that course's files are fetched. + Other flags, rarely needed: `--title` (document H1), `--helper-name` + (default `helper.py`), `--keep-clone` (debugging). If the script exits + non-zero — clone/auth failure, a subdir that doesn't exist, nothing to + ingest, or a bad `--out` — surface its error verbatim and Stop. +6. **Report.** Echo the script's one-line summary (notebooks, cells, helper + files, unique defs, conflicts). Then `Glob materials/notebooks/*.md`: if a + hand-downloaded `*-context.md` sits at the top level, add a note that both + are now read by `/generate-spec` — the site dump is authoritative for lesson + numbering and titles, this file carries the real code and `helper.py`. State + it and move on; don't ask the user to choose. +7. **Hand off.** Remind them the site download is still the main source: if + `materials/transcripts/` is empty, transcripts must come from + https://course-context-lab.vercel.app. Once materials are in place, run + `/generate-spec`. + +## Don't + +- Don't write `spec.md` — that's `/generate-spec`. +- Don't write anywhere under `materials/` except + `materials/notebooks/from-repo/`, and never into `materials/transcripts/`. + The script enforces this too, but don't rely on it. +- Don't overwrite or edit a hand-downloaded `*-context.md`. The `from-repo/` + subfolder exists precisely because the two share a filename. +- Don't WebFetch course content, or the courses index — the user browses it and + pastes a URL. The agent only clones the repo it was handed. +- Don't clone a monorepo root hoping to filter afterwards. Scope it with + `--subdir` so the fetch stays small. +- Don't run from the repo root or from inside a build folder. Materials are + off-limits during a build, and a `PreToolUse` hook enforces that. diff --git a/.claude/skills/generate-spec/scripts/ingest_repo.py b/.claude/skills/ingest-course-repo/scripts/ingest_repo.py similarity index 84% rename from .claude/skills/generate-spec/scripts/ingest_repo.py rename to .claude/skills/ingest-course-repo/scripts/ingest_repo.py index ac3e5cd..c13df18 100644 --- a/.claude/skills/generate-spec/scripts/ingest_repo.py +++ b/.claude/skills/ingest-course-repo/scripts/ingest_repo.py @@ -3,7 +3,8 @@ Usage: python3 ingest_repo.py --out [--ref ] - [--title ] [--helper-name helper.py] [--keep-clone] + [--subdir ] [--title ] + [--helper-name helper.py] [--keep-clone] Clones (SSH URL, https, or a local path / file:// for testing) over the caller's git credentials, then renders every notebook (.ipynb) cell-by-cell into @@ -11,6 +12,11 @@ de-duplicated "Helper Module Context" section built from the repo's helper.py module(s). +--subdir scopes everything to one repo-relative directory and switches the +clone to partial + sparse, so a single course can be pulled out of a +multi-gigabyte monorepo without fetching the rest. Omitted, the whole repo is +discovered exactly as before. + The generated markdown is written to --out, which MUST live under a course's materials/notebooks/ directory. transcripts/ are never touched — they remain a manual drop. @@ -32,6 +38,7 @@ import ast import hashlib import json +import os import re import shutil import subprocess @@ -52,11 +59,17 @@ def is_local_repo(repo: str) -> Path | None: return p if p.is_dir() else None -def clone_repo(repo: str, ref: str | None, dest: Path) -> Path: +def clone_repo(repo: str, ref: str | None, dest: Path, + subdir: str | None = None) -> Path: """Clone `repo` into `dest`. A local dir is copied, not cloned. Shallow-clones by default. When `ref` looks like a commit SHA we do a full clone then checkout, since --depth 1 can't resolve an arbitrary SHA. + + When `subdir` is given the clone is partial (--filter=blob:none) and + sparse, so blobs are fetched only for that subtree. That's what makes + pulling one course out of a multi-gigabyte monorepo tractable; without it + the behaviour is byte-for-byte what it was before. """ local = is_local_repo(repo) if local is not None: @@ -67,20 +80,29 @@ def clone_repo(repo: str, ref: str | None, dest: Path) -> Path: return dest ref_is_sha = bool(ref and _SHA_RE.match(ref)) + sparse = ["--filter=blob:none", "--sparse"] if subdir else [] if ref and not ref_is_sha: - cmd = ["git", "clone", "--depth", "1", "--branch", ref, repo, str(dest)] + cmd = ["git", "clone", "--depth", "1", *sparse, "--branch", ref, + repo, str(dest)] elif not ref: - cmd = ["git", "clone", "--depth", "1", repo, str(dest)] + cmd = ["git", "clone", "--depth", "1", *sparse, repo, str(dest)] else: - cmd = ["git", "clone", repo, str(dest)] + # An arbitrary SHA can't be shallow-cloned. With --filter the full + # clone stays metadata-only, so this is still cheap on a big repo. + cmd = ["git", "clone", *sparse, repo, str(dest)] _run_git(cmd, repo) + if subdir: + _run_git(["git", "-C", str(dest), "sparse-checkout", "set", subdir], repo) if ref_is_sha: _run_git(["git", "-C", str(dest), "checkout", ref], repo) return dest def _run_git(cmd: list[str], repo: str) -> None: - proc = subprocess.run(cmd, capture_output=True, text=True) + # Ingestion reads notebooks and helper modules, never LFS payloads + # (datasets, weights). Skipping the smudge filter keeps checkouts small. + env = {**os.environ, "GIT_LFS_SKIP_SMUDGE": "1"} + proc = subprocess.run(cmd, capture_output=True, text=True, env=env) if proc.returncode != 0: tail = (proc.stderr or proc.stdout or "").strip().splitlines()[-4:] detail = "\n ".join(tail) @@ -275,14 +297,22 @@ def render_helpers(defs: list[Def], conflicts: set[str], labels: list[str], # ── Document assembly ──────────────────────────────────────────────────── _MAP_NOTE = ( - "> Generated from a git repo by ingest_repo.py. Lesson numbers are inferred " + "> Generated from {source} by ingest_repo.py. Lesson numbers are inferred " "from notebook order and are a convenience label only — the transcripts under " "materials/transcripts/ are authoritative for lesson numbering and titles." ) +def source_label(repo: str, ref: str | None, subdir: str | None) -> str: + """Provenance string for the document header — repo, subdir, ref.""" + label = f"`{repo}`" + if subdir: + label += f" (subdirectory `{subdir}`)" + return label + f" @ {ref or 'default branch'}" + + def build_markdown(title: str, notebooks: list[tuple[str, str, int, list[dict]]], - helper_section: str) -> str: + helper_section: str, source: str) -> str: """Assemble title → Lesson Map → notebook sections → helper section.""" parts = [f"# {title}", "", "---", "", "## Lesson Map", ""] if notebooks: @@ -290,7 +320,7 @@ def build_markdown(title: str, notebooks: list[tuple[str, str, int, list[dict]]] parts.append(f"- {rel} → Lesson {no}: {ntitle}") else: parts.append("- (no .ipynb files found)") - parts += ["", _MAP_NOTE, ""] + parts += ["", _MAP_NOTE.format(source=source), ""] for rel, ntitle, no, cells in notebooks: parts.append("---") parts.append("") @@ -330,6 +360,9 @@ def parse_args(argv: list[str]) -> argparse.Namespace: p.add_argument("--out", "-o", required=True, type=Path, help="destination .md under a course's materials/notebooks/") p.add_argument("--ref", default=None, help="branch/tag/commit (default: repo default)") + p.add_argument("--subdir", default=None, + help="repo-relative dir to ingest; enables a sparse, " + "blobless clone (use for monorepos)") p.add_argument("--title", default=None, help="document H1 (default: from --out name)") p.add_argument("--helper-name", default="helper.py", help="filename treated as a helper module (default: helper.py)") @@ -346,7 +379,13 @@ def main(argv: list[str]) -> int: tmp = Path(tempfile.mkdtemp(prefix="ingest-repo-")) try: - root = clone_repo(args.repo, args.ref, tmp / "repo") + clone_root = clone_repo(args.repo, args.ref, tmp / "repo", args.subdir) + root = clone_root / args.subdir if args.subdir else clone_root + if not root.is_dir(): + sys.exit( + f"error: subdir {args.subdir!r} not found in {args.repo}" + f"@{args.ref or 'default'}" + ) nb_paths = find_notebooks(root) notebooks: list[tuple[str, str, int, list[dict]]] = [] @@ -378,7 +417,8 @@ def main(argv: list[str]) -> int: title = args.title or derive_title(out) title_note = "" if args.title else " (heuristic)" - document = build_markdown(title, notebooks, helper_section) + source = source_label(args.repo, args.ref, args.subdir) + document = build_markdown(title, notebooks, helper_section, source) out.write_text(document, encoding="utf-8") finally: if not args.keep_clone: @@ -389,7 +429,8 @@ def main(argv: list[str]) -> int: f"{len(helper_paths)} helper file(s) → {len(kept)} unique def(s) " f"({len(conflicts)} name-conflict(s), {symlinks_collapsed} symlink(s) " f"collapsed), title {title!r}{title_note}, " - f"from {args.repo}@{args.ref or 'default'}." + f"from {args.repo}{f' [{args.subdir}]' if args.subdir else ''}" + f"@{args.ref or 'default'}." ) return 0 diff --git a/.claude/skills/new-course/SKILL.md b/.claude/skills/new-course/SKILL.md index cda8bc7..4069cda 100644 --- a/.claude/skills/new-course/SKILL.md +++ b/.claude/skills/new-course/SKILL.md @@ -52,10 +52,12 @@ the template; it does not read materials and does not generate `spec.md`. having me traverse your filesystem." 4. "Then `cd courses/$ARGUMENTS` and run /generate-spec when the materials are in place — say 'go ahead' when you're ready." - 5. "Shortcut for notebooks: if the course code lives in a private GitHub - repo you can reach over SSH, you can skip the manual notebook download — - run `/generate-spec ` and it will ingest the `.ipynb` - files and `helper.py` into materials/notebooks/ for you. Transcripts + 5. "Optional extra: if you also want the course's real code, find the + course at https://github.com/deeplearningai-eng/courses and run + `/ingest-course-repo ` before /generate-spec — it + adds the `.ipynb` files and `helper.py` under + courses/$ARGUMENTS/materials/notebooks/from-repo/. This is an addition + to step 2, not a replacement: the notebooks `.md` and the transcripts still come from the site." ## Don't diff --git a/AGENTS.md b/AGENTS.md index 1262d71..b1f1abb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,10 +14,10 @@ Launch the session once at the repo root and stay there. Within this one session your working directory moves: - repo root → `/new-course`. -- `courses//` → `/generate-spec`, `/prepare-build`, - `/extract-build-log`, `/eval-spec-vs-build`, `/eval-materials-vs-build`. `cd` - here before running them; they act on the current course. Relative paths below - are from this folder. +- `courses//` → `/ingest-course-repo` (optional), `/generate-spec`, + `/prepare-build`, `/extract-build-log`, `/eval-spec-vs-build`, + `/eval-materials-vs-build`. `cd` here before running them; they act on the + current course. Relative paths below are from this folder. - `courses//builds/run-NN/` → the actual app development (install deps, run services, optional app-level `git init`). `cd` here to build. @@ -33,6 +33,16 @@ supported automatically — see "Building" below. transcripts manually, and remind them that dragging files from Finder into the terminal pastes the absolute paths so the agent can `cp` them in cheaply. **The agent does NOT WebFetch from that URL** — it's a manual user step. +- `/ingest-course-repo ` — run from a course folder. **Optional extra**, + never a replacement for the manual download: it renders a course repo's + `.ipynb` files and `helper.py` into + `materials/notebooks/from-repo/-context.md`, adding the course's real + code on top of the site dump. Takes a course-scoped repo URL — for + DeepLearning.AI courses that's a folder URL under + https://github.com/deeplearningai-eng/courses, a ~3.4 GB monorepo where a + bare-root URL is refused. **The agent does NOT WebFetch that index** — the + user browses it and pastes a URL; the agent only clones what it's handed. + Writes nothing under `materials/transcripts/`, which stays a manual download. - `/generate-spec` — reads `materials/notebooks/` and `materials/transcripts/`, follows `.Codex/skills/generate-spec/references/spec-generation-guide.md`, and writes `spec.md`. diff --git a/CLAUDE.md b/CLAUDE.md index 3ce3c92..330a7dc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,10 +14,10 @@ Launch the session once at the repo root and stay there. Within this one session your working directory moves: - repo root → `/new-course`. -- `courses//` → `/generate-spec`, `/prepare-build`, - `/extract-build-log`, `/eval-spec-vs-build`, `/eval-materials-vs-build`. `cd` - here before running them; they act on the current course. Relative paths below - are from this folder. +- `courses//` → `/ingest-course-repo` (optional), `/generate-spec`, + `/prepare-build`, `/extract-build-log`, `/eval-spec-vs-build`, + `/eval-materials-vs-build`. `cd` here before running them; they act on the + current course. Relative paths below are from this folder. - `courses//builds/run-NN/` → the actual app development (install deps, run services, optional app-level `git init`). `cd` here to build. @@ -33,6 +33,16 @@ supported automatically — see "Building" below. transcripts manually, and remind them that dragging files from Finder into the terminal pastes the absolute paths so the agent can `cp` them in cheaply. **The agent does NOT WebFetch from that URL** — it's a manual user step. +- `/ingest-course-repo ` — run from a course folder. **Optional extra**, + never a replacement for the manual download: it renders a course repo's + `.ipynb` files and `helper.py` into + `materials/notebooks/from-repo/-context.md`, adding the course's real + code on top of the site dump. Takes a course-scoped repo URL — for + DeepLearning.AI courses that's a folder URL under + https://github.com/deeplearningai-eng/courses, a ~3.4 GB monorepo where a + bare-root URL is refused. **The agent does NOT WebFetch that index** — the + user browses it and pastes a URL; the agent only clones what it's handed. + Writes nothing under `materials/transcripts/`, which stays a manual download. - `/generate-spec` — reads `materials/notebooks/` and `materials/transcripts/`, follows `.claude/skills/generate-spec/references/spec-generation-guide.md`, and writes `spec.md`. diff --git a/README.md b/README.md index df2cb21..04dda12 100644 --- a/README.md +++ b/README.md @@ -26,10 +26,11 @@ spec-build-lab/ ├── .claude/ │ └── skills/ │ ├── new-course/ # scaffold courses// from the template +│ ├── ingest-course-repo/ # optional extra: course repo → materials/notebooks/from-repo/ +│ │ └── scripts/ingest_repo.py │ ├── generate-spec/ # materials → spec.md (uses the guide below) -│ │ ├── references/ -│ │ │ └── spec-generation-guide.md -│ │ └── scripts/ingest_repo.py # optional: git repo → materials/notebooks/ context dump +│ │ └── references/ +│ │ └── spec-generation-guide.md │ ├── prepare-build/ # allocate builds/run-NN/ + drop .session breadcrumb (auto-fires on build intent) │ ├── extract-build-log/ # slice the conversation from the session transcript │ │ └── scripts/extract.py @@ -61,7 +62,7 @@ the repo root. | You're working in… | What happens there | What you run | |---|---|---| | repo root | manage courses | `/new-course` | -| `courses//` | spec + extract + evaluation | `/generate-spec`, `/prepare-build`, `/extract-build-log`, `/eval-spec-vs-build`, `/eval-materials-vs-build` | +| `courses//` | spec + extract + evaluation | `/ingest-course-repo` (optional), `/generate-spec`, `/prepare-build`, `/extract-build-log`, `/eval-spec-vs-build`, `/eval-materials-vs-build` | | `courses//builds/run-NN/` | actual app development | the build itself — install deps, run the dev server, optional app-level `git init` | So `cd courses//` to work on a course; once a build starts, you and the @@ -76,6 +77,7 @@ which is what `/extract-build-log` slices. flowchart TD NC["/new-course
scaffold courses/NAME/"] MAT(["add materials:
notebooks + transcripts"]) + ICR["/ingest-course-repo
optional extra: course repo →
materials/notebooks/from-repo/"] GS["/generate-spec
materials → spec.md"] PB["/prepare-build
allocate builds/run-NN/,
snapshot spec → evals/run-NN/spec.md,
drop .session breadcrumb"] OP(["you + coding agent
build the app in builds/run-NN/"]) @@ -85,6 +87,8 @@ flowchart TD IT(["iterate"]) NC --> MAT --> GS --> PB --> OP --> EBL + MAT -. optional .-> ICR + ICR -. adds to materials .-> GS OP --> ESB & EMB ESB --> IT EMB --> IT @@ -108,17 +112,15 @@ evaluated run: # your filesystem. # notebooks -> courses/langchain-rag/materials/notebooks/ # transcripts -> courses/langchain-rag/materials/transcripts/ -# -# Notebook shortcut: if the course code lives in a git repo, skip the -# manual notebook download — pass the repo to /generate-spec in step 3 -# and it ingests the .ipynb files + helper.py for you (see "Ingesting -# notebooks from a git repo" below). Transcripts still come from the site. # 3 · move into the course and generate the spec cd courses/langchain-rag + +# optional extra — adds the course's real code + helper.py on top of +# the site download (see "Ingesting a course repo" below): +/ingest-course-repo https://github.com/deeplearningai-eng/courses/tree/main/course_10 + /generate-spec # -> courses/langchain-rag/spec.md -# …or, with the notebook shortcut: -/generate-spec git@github.com:org/course-repo.git # ingest, then generate # 4 · start a build (call it explicitly) /prepare-build # -> builds/run-01/ (the app lives here) @@ -145,26 +147,40 @@ cd ../.. # back up to courses/langchain-rag /prepare-build # -> builds/run-02/, evals/run-02/spec.md, ... ``` -## Ingesting notebooks from a git repo +## Ingesting a course repo -`/generate-spec` optionally takes a git repo as its argument. When given one, it -runs a deterministic ingest step **before** reading materials: +`/ingest-course-repo` is an **optional extra**, not a shortcut. Materials still +come from https://course-context-lab.vercel.app — both the notebooks `.md` and +the transcripts. What this adds on top is the course's real, runnable code and +its `helper.py` definitions, which the site dump doesn't carry. ```text -/generate-spec # e.g. git@github.com:org/course-repo.git +cd courses/ +/ingest-course-repo https://github.com/deeplearningai-eng/courses/tree/main/course_10 ``` -What it does, exactly (`generate-spec/scripts/ingest_repo.py`, stdlib-only): +For DeepLearning.AI courses, browse https://github.com/deeplearningai-eng/courses +to find yours and paste the URL of its **folder**. That repo is a ~3.4 GB +monorepo holding every course, so a course-scoped URL is required — passing the +bare repo root is refused. -1. **Clones the repo** shallowly using your own git credentials — so a private - repo works as long as *you* can reach it. Accepted forms: +What it does, exactly (`ingest-course-repo/scripts/ingest_repo.py`, stdlib-only): + +1. **Clones the repo** using your own git credentials — so a private repo works + as long as *you* can reach it. Accepted forms: + - **GitHub tree URL** — `…/tree//`, parsed into repo + `--ref` + + `--subdir`. This is the form to use for the monorepo. - **SSH** — `git@github.com:org/repo.git` or `ssh://…` (uses your SSH key) - **HTTPS** — `https://github.com/org/repo.git` - **Local path** — an existing directory or `file://` URL is copied, not cloned (handy for testing or repos you already have on disk) - A `--ref ` can pin what gets ingested. -2. **Renders every `.ipynb`** in the repo cell-by-cell into a single markdown - file at `materials/notebooks/-context.md` — the same shape as a + + With a `--subdir` the clone is **sparse and blobless**, so only that one + course's files are fetched — pulling `course_10` out of the 3.4 GB monorepo + transfers about 9 MB and takes a few seconds. LFS payloads are skipped; + ingestion reads notebooks and helper modules, never datasets. +2. **Renders every `.ipynb`** in scope cell-by-cell into a single markdown file + at `materials/notebooks/from-repo/-context.md` — the same shape as a hand-downloaded course context dump. 3. **Appends a de-duplicated "Helper Module Context" section** from the repo's `helper.py` module(s): each unique top-level def/class appears once, @@ -172,13 +188,18 @@ What it does, exactly (`generate-spec/scripts/ingest_repo.py`, stdlib-only): different-body collision keeps both variants flagged inline — nothing is silently dropped. -Scope guarantees: the script writes **only** into `materials/notebooks/`; -`materials/transcripts/` is never touched and remains a manual download from -the course site. Lesson numbering in the output is inferred from sorted -notebook order — a convenience label, not authentic platform numbering (the -transcripts stay authoritative for that). The ingest is idempotent: re-running -overwrites the context file cleanly. Without a repo argument, `/generate-spec` -behaves as before and uses whatever is already in `materials/`. +The `from-repo/` subfolder matters: a hand-downloaded context dump is named +`-context.md` too, so a flat write would silently overwrite it. Both +files coexist and `/generate-spec` reads both — the site dump stays +authoritative for lesson numbering and titles, the ingested file carries the +code. + +Scope guarantees: the script writes **only** into +`materials/notebooks/from-repo/`; `materials/transcripts/` is never touched and +remains a manual download from the course site. Lesson numbering in the output +is inferred from sorted notebook order — a convenience label, not authentic +platform numbering. The ingest is idempotent: re-running overwrites its own +output file cleanly and leaves everything else alone. `/prepare-build` is the expected, explicit step — it will also fire on its own if you simply start building, but the documented flow is to call it. The spec eval