Extract git-repo ingestion into a standalone /ingest-course-repo skill

.claude/skills/generate-spec/SKILL.md

@@ -9,15 +9,11 @@ description: >
   from this course", "turn the materials into a spec", "build the spec
   out of materials/", "regenerate spec.md", "run the spec generation",
   "I've added the notebooks, please create the spec", or "/generate-spec".
-  Optionally accepts a git repo (SSH URL or local path) as an argument to
-  auto-ingest the notebooks + helper.py into materials/notebooks/ before
-  generating; without an argument it uses whatever is already in materials/.
   Reads every file under materials/notebooks/ and materials/transcripts/
   and follows references/spec-generation-guide.md. Writes spec.md in the
   current working directory. Must run from inside a course folder
   (courses/<name>/). Do NOT use this skill for editing an existing
   spec.md or for writing product specs unrelated to a course.
-argument-hint: "[repo-ssh-url]"
 allowed-tools: [Read, Write, Glob, Grep, Bash]
 ---
 
@@ -32,36 +28,21 @@ another engineer (or coding agent) can implement end-to-end.
    cwd: `materials/notebooks/`, `materials/transcripts/`, `builds/`,
    `evals/`. If any are missing, refuse with: "Run /generate-spec from
    inside a course folder — `cd courses/<name>` first." Stop.
-2. **(Optional) Ingest notebooks + helper.py from a git repo.** If the user
-   supplied a repo argument — an `ssh://…`, `git@…:…`, `https://…`, or a local
-   path / `file://` dir — run the deterministic ingest script from the course
-   folder BEFORE reading materials:
-   ```bash
-   python3 "$CLAUDE_PROJECT_DIR/.claude/skills/generate-spec/scripts/ingest_repo.py" \
-     <repo> --out "materials/notebooks/<course>-context.md" [--ref <branch>]
-   ```
-   Use the course-folder name (cwd basename) for `<course>`. The script clones
-   over the user's SSH key, renders every `.ipynb` cell-by-cell plus a
-   de-duplicated `## Helper Module Context` section, and writes ONLY into
-   `materials/notebooks/`. Relay its one-line summary. If it exits non-zero
-   (clone/SSH failure, nothing to ingest, or a bad `--out`), surface its error
-   and Stop. **If no repo argument was given, skip this step entirely and use
-   whatever is already in `materials/`.** Transcripts are NEVER produced here —
-   they remain a manual drop under `materials/transcripts/`.
-3. **Read the guide.** Read
+2. **Read the guide.** Read
    `.claude/skills/generate-spec/references/spec-generation-guide.md`
    relative to `$CLAUDE_PROJECT_DIR` (anchor on the repo root, not cwd).
 
    Follow it fully — it is the prescriptive contract for the spec's
    content, structure, and quality. Do not restate or summarize its rules
    here; defer to the guide so the two files can't drift.
-4. **Read every material.** Glob `materials/notebooks/**/*` and
+3. **Read every material.** Glob `materials/notebooks/**/*` and
    `materials/transcripts/**/*`, then read each non-empty file. If both
    directories are empty (only `.gitkeep`), refuse with: "No materials
    found. Add notebooks to materials/notebooks/ and transcripts to
-   materials/transcripts/, then re-run." Stop. (After a Step 2 ingest, the
-   freshly written `<course>-context.md` is picked up here.)
-5. **Write spec.md.** Produce a self-contained build-ready spec at
+   materials/transcripts/, then re-run — or optionally run
+   /ingest-course-repo <course-folder-URL> to add the course repo's
+   notebooks and helper.py on top." Stop.
+4. **Write spec.md.** Produce a self-contained build-ready spec at
    `./spec.md` in cwd (overwrite if it exists — this is regeneration, not
    editing). What the spec must contain and how it handles learner input
    are governed entirely by the guide — don't duplicate those rules here.
@@ -72,15 +53,12 @@ another engineer (or coding agent) can implement end-to-end.
    conversation. It's a UX hint, not a system requirement (the slice can
    also be controlled with `--until="<phrase>"`, and falls back to
    end-of-transcript if no closing match is found).
-6. **Report.** Tell the user the path written and that they can review,
+5. **Report.** Tell the user the path written and that they can review,
    then run /prepare-build when ready.
 
 ## Don't
 
-- Don't write to `builds/`, `evals/`, or `materials/` yourself. The Step 2
-  ingest script is the ONLY permitted writer into `materials/`, and only into
-  `materials/notebooks/`. Never hand-write or edit anything under `materials/`,
-  and never touch `materials/transcripts/`.
+- Don't write to `builds/`, `evals/`, or `materials/`.
 - Don't pull in external knowledge to fill gaps in the materials —
   faithfulness to the course IS what /eval-materials-vs-build measures
   later.

.claude/skills/ingest-course-repo/SKILL.md

@@ -0,0 +1,100 @@
+---
+name: ingest-course-repo
+description: >
+  Adds a course repo's notebooks and helper modules to a course's
+  materials/ in the spec-build-lab, rendered into
+  materials/notebooks/from-repo/<course>-context.md. Use this skill ANY
+  time the user wants to pull course code out of a git repo — including
+  phrasings like "ingest the course repo", "pull the notebooks from this
+  repo", "here's the course repo", "add the course code to materials",
+  "get the notebooks from git", or "/ingest-course-repo <url>". Takes a
+  course-scoped repo URL: a GitHub tree URL pointing at one course
+  folder, a plain SSH/HTTPS repo URL, or a local path. It is an OPTIONAL
+  supplement to the manual material download, never a replacement — it
+  adds the course's real code and helper.py on top, and NEVER produces
+  transcripts. Must run from inside a course folder (courses/<name>/).
+  Do NOT use this skill to write spec.md — that's /generate-spec.
+argument-hint: <course-repo-url>
+allowed-tools: [Bash, Read, Glob]
+---
+
+# ingest-course-repo
+
+Render a course repo's `.ipynb` files and helper module(s) into
+`materials/notebooks/from-repo/<course>-context.md`.
+
+This is an **extra**, not a shortcut. The main way materials arrive is still a
+manual download from https://course-context-lab.vercel.app — both the notebooks
+`.md` and the transcripts. What this skill adds on top is the course's real,
+runnable code and its `helper.py` definitions, which the site dump doesn't
+carry. Transcripts are never produced here.
+
+## Steps
+
+1. **Verify cwd is a course folder.** Confirm all of these exist relative to
+   cwd: `materials/notebooks/`, `materials/transcripts/`, `builds/`, `evals/`.
+   If any are missing, refuse with: "Run /ingest-course-repo from inside a
+   course folder — `cd courses/<name>` first." Stop.
+2. **Require the repo URL.** If `$ARGUMENTS` is empty, say: "Which course repo?
+   If it's a DeepLearning.AI course, find it at
+   https://github.com/deeplearningai-eng/courses and paste the URL of the
+   course's folder — e.g. `.../courses/tree/main/course_10`." Stop until they
+   answer.
+3. **Parse the URL.** Strip any `?query` or `#fragment` first, then:
+
+   | Input | Becomes |
+   |---|---|
+   | `https://github.com/<org>/<repo>/tree/<ref>/<path…>` | repo `https://github.com/<org>/<repo>`, `--ref <ref>`, `--subdir <path…>` |
+   | `https://github.com/<org>/<repo>` | repo as given, no `--subdir` |
+   | `git@…:…`, `ssh://…`, local path, `file://…` | repo as given, no `--subdir` |
+
+   Strip only the `/tree/<ref>/<path…>` tail; don't add or remove a `.git`
+   suffix. The repo string lands in the generated file's provenance line, so it
+   should read back as what the user actually pasted.
+
+   A `/blob/` URL points at a file — ask for the folder URL instead. Splitting
+   `tree/<ref>/<path>` assumes the ref is a single segment; if the branch name
+   contains a `/`, that guess is wrong, so let the user correct it and pass
+   `--ref` / `--subdir` explicitly.
+4. **Refuse the courses index.** If the input resolves to
+   `deeplearningai-eng/courses` with no subdirectory, refuse with: "That's the
+   courses index — 262 courses, 3.4 GB. Browse to your course and paste its
+   folder URL, e.g. `.../courses/tree/main/course_10`." Stop. Any other repo
+   root is fine and proceeds without `--subdir`.
+5. **Run the ingest script** from the course folder:
+   ```bash
+   python3 "$CLAUDE_PROJECT_DIR/.claude/skills/ingest-course-repo/scripts/ingest_repo.py" \
+     <repo> --out "materials/notebooks/from-repo/<course>-context.md" \
+     [--ref <ref>] [--subdir <path>]
+   ```
+   Use the course-folder name (cwd basename) for `<course>`. With `--subdir`
+   the clone is sparse and blobless, so only that course's files are fetched.
+   Other flags, rarely needed: `--title` (document H1), `--helper-name`
+   (default `helper.py`), `--keep-clone` (debugging). If the script exits
+   non-zero — clone/auth failure, a subdir that doesn't exist, nothing to
+   ingest, or a bad `--out` — surface its error verbatim and Stop.
+6. **Report.** Echo the script's one-line summary (notebooks, cells, helper
+   files, unique defs, conflicts). Then `Glob materials/notebooks/*.md`: if a
+   hand-downloaded `*-context.md` sits at the top level, add a note that both
+   are now read by `/generate-spec` — the site dump is authoritative for lesson
+   numbering and titles, this file carries the real code and `helper.py`. State
+   it and move on; don't ask the user to choose.
+7. **Hand off.** Remind them the site download is still the main source: if
+   `materials/transcripts/` is empty, transcripts must come from
+   https://course-context-lab.vercel.app. Once materials are in place, run
+   `/generate-spec`.
+
+## Don't
+
+- Don't write `spec.md` — that's `/generate-spec`.
+- Don't write anywhere under `materials/` except
+  `materials/notebooks/from-repo/`, and never into `materials/transcripts/`.
+  The script enforces this too, but don't rely on it.
+- Don't overwrite or edit a hand-downloaded `*-context.md`. The `from-repo/`
+  subfolder exists precisely because the two share a filename.
+- Don't WebFetch course content, or the courses index — the user browses it and
+  pastes a URL. The agent only clones the repo it was handed.
+- Don't clone a monorepo root hoping to filter afterwards. Scope it with
+  `--subdir` so the fetch stays small.
+- Don't run from the repo root or from inside a build folder. Materials are
+  off-limits during a build, and a `PreToolUse` hook enforces that.

.claude/skills/generate-spec/scripts/ingest_repo.py → .claude/skills/ingest-course-repo/scripts/ingest_repo.py

@@ -3,14 +3,20 @@
 
 Usage:
     python3 ingest_repo.py <repo> --out <path.md> [--ref <branch/tag/sha>]
-                           [--title <str>] [--helper-name helper.py] [--keep-clone]
+                           [--subdir <path>] [--title <str>]
+                           [--helper-name helper.py] [--keep-clone]
 
 Clones <repo> (SSH URL, https, or a local path / file:// for testing) over the
 caller's git credentials, then renders every notebook (.ipynb) cell-by-cell into
 the same shape as a hand-downloaded course context dump, followed by a
 de-duplicated "Helper Module Context" section built from the repo's helper.py
 module(s).
 
+--subdir scopes everything to one repo-relative directory and switches the
+clone to partial + sparse, so a single course can be pulled out of a
+multi-gigabyte monorepo without fetching the rest. Omitted, the whole repo is
+discovered exactly as before.
+
 The generated markdown is written to --out, which MUST live under a course's
 materials/notebooks/ directory. transcripts/ are never touched — they remain a
 manual drop.
@@ -32,6 +38,7 @@
 import ast
 import hashlib
 import json
+import os
 import re
 import shutil
 import subprocess
@@ -52,11 +59,17 @@ def is_local_repo(repo: str) -> Path | None:
     return p if p.is_dir() else None
 
 
-def clone_repo(repo: str, ref: str | None, dest: Path) -> Path:
+def clone_repo(repo: str, ref: str | None, dest: Path,
+               subdir: str | None = None) -> Path:
     """Clone `repo` into `dest`. A local dir is copied, not cloned.
 
     Shallow-clones by default. When `ref` looks like a commit SHA we do a full
     clone then checkout, since --depth 1 can't resolve an arbitrary SHA.
+
+    When `subdir` is given the clone is partial (--filter=blob:none) and
+    sparse, so blobs are fetched only for that subtree. That's what makes
+    pulling one course out of a multi-gigabyte monorepo tractable; without it
+    the behaviour is byte-for-byte what it was before.
     """
     local = is_local_repo(repo)
     if local is not None:
@@ -67,20 +80,29 @@ def clone_repo(repo: str, ref: str | None, dest: Path) -> Path:
         return dest
 
     ref_is_sha = bool(ref and _SHA_RE.match(ref))
+    sparse = ["--filter=blob:none", "--sparse"] if subdir else []
     if ref and not ref_is_sha:
-        cmd = ["git", "clone", "--depth", "1", "--branch", ref, repo, str(dest)]
+        cmd = ["git", "clone", "--depth", "1", *sparse, "--branch", ref,
+               repo, str(dest)]
     elif not ref:
-        cmd = ["git", "clone", "--depth", "1", repo, str(dest)]
+        cmd = ["git", "clone", "--depth", "1", *sparse, repo, str(dest)]
     else:
-        cmd = ["git", "clone", repo, str(dest)]
+        # An arbitrary SHA can't be shallow-cloned. With --filter the full
+        # clone stays metadata-only, so this is still cheap on a big repo.
+        cmd = ["git", "clone", *sparse, repo, str(dest)]
     _run_git(cmd, repo)
+    if subdir:
+        _run_git(["git", "-C", str(dest), "sparse-checkout", "set", subdir], repo)
     if ref_is_sha:
         _run_git(["git", "-C", str(dest), "checkout", ref], repo)
     return dest
 
 
 def _run_git(cmd: list[str], repo: str) -> None:
-    proc = subprocess.run(cmd, capture_output=True, text=True)
+    # Ingestion reads notebooks and helper modules, never LFS payloads
+    # (datasets, weights). Skipping the smudge filter keeps checkouts small.
+    env = {**os.environ, "GIT_LFS_SKIP_SMUDGE": "1"}
+    proc = subprocess.run(cmd, capture_output=True, text=True, env=env)
     if proc.returncode != 0:
         tail = (proc.stderr or proc.stdout or "").strip().splitlines()[-4:]
         detail = "\n  ".join(tail)
@@ -275,22 +297,30 @@ def render_helpers(defs: list[Def], conflicts: set[str], labels: list[str],
 # ── Document assembly ────────────────────────────────────────────────────
 
 _MAP_NOTE = (
-    "> Generated from a git repo by ingest_repo.py. Lesson numbers are inferred "
+    "> Generated from {source} by ingest_repo.py. Lesson numbers are inferred "
     "from notebook order and are a convenience label only — the transcripts under "
     "materials/transcripts/ are authoritative for lesson numbering and titles."
 )
 
 
+def source_label(repo: str, ref: str | None, subdir: str | None) -> str:
+    """Provenance string for the document header — repo, subdir, ref."""
+    label = f"`{repo}`"
+    if subdir:
+        label += f" (subdirectory `{subdir}`)"
+    return label + f" @ {ref or 'default branch'}"
+
+
 def build_markdown(title: str, notebooks: list[tuple[str, str, int, list[dict]]],
-                   helper_section: str) -> str:
+                   helper_section: str, source: str) -> str:
     """Assemble title → Lesson Map → notebook sections → helper section."""
     parts = [f"# {title}", "", "---", "", "## Lesson Map", ""]
     if notebooks:
         for rel, ntitle, no, _cells in notebooks:
             parts.append(f"- {rel} → Lesson {no}: {ntitle}")
     else:
         parts.append("- (no .ipynb files found)")
-    parts += ["", _MAP_NOTE, ""]
+    parts += ["", _MAP_NOTE.format(source=source), ""]
     for rel, ntitle, no, cells in notebooks:
         parts.append("---")
         parts.append("")
@@ -330,6 +360,9 @@ def parse_args(argv: list[str]) -> argparse.Namespace:
     p.add_argument("--out", "-o", required=True, type=Path,
                    help="destination .md under a course's materials/notebooks/")
     p.add_argument("--ref", default=None, help="branch/tag/commit (default: repo default)")
+    p.add_argument("--subdir", default=None,
+                   help="repo-relative dir to ingest; enables a sparse, "
+                        "blobless clone (use for monorepos)")
     p.add_argument("--title", default=None, help="document H1 (default: from --out name)")
     p.add_argument("--helper-name", default="helper.py",
                    help="filename treated as a helper module (default: helper.py)")
@@ -346,7 +379,13 @@ def main(argv: list[str]) -> int:
 
     tmp = Path(tempfile.mkdtemp(prefix="ingest-repo-"))
     try:
-        root = clone_repo(args.repo, args.ref, tmp / "repo")
+        clone_root = clone_repo(args.repo, args.ref, tmp / "repo", args.subdir)
+        root = clone_root / args.subdir if args.subdir else clone_root
+        if not root.is_dir():
+            sys.exit(
+                f"error: subdir {args.subdir!r} not found in {args.repo}"
+                f"@{args.ref or 'default'}"
+            )
 
         nb_paths = find_notebooks(root)
         notebooks: list[tuple[str, str, int, list[dict]]] = []
@@ -378,7 +417,8 @@ def main(argv: list[str]) -> int:
 
         title = args.title or derive_title(out)
         title_note = "" if args.title else " (heuristic)"
-        document = build_markdown(title, notebooks, helper_section)
+        source = source_label(args.repo, args.ref, args.subdir)
+        document = build_markdown(title, notebooks, helper_section, source)
         out.write_text(document, encoding="utf-8")
     finally:
         if not args.keep_clone:
@@ -389,7 +429,8 @@ def main(argv: list[str]) -> int:
         f"{len(helper_paths)} helper file(s) → {len(kept)} unique def(s) "
         f"({len(conflicts)} name-conflict(s), {symlinks_collapsed} symlink(s) "
         f"collapsed), title {title!r}{title_note}, "
-        f"from {args.repo}@{args.ref or 'default'}."
+        f"from {args.repo}{f' [{args.subdir}]' if args.subdir else ''}"
+        f"@{args.ref or 'default'}."
     )
     return 0
 

.claude/skills/new-course/SKILL.md

@@ -52,10 +52,12 @@ the template; it does not read materials and does not generate `spec.md`.
       having me traverse your filesystem."
    4. "Then `cd courses/$ARGUMENTS` and run /generate-spec when the
       materials are in place — say 'go ahead' when you're ready."
-   5. "Shortcut for notebooks: if the course code lives in a private GitHub
-      repo you can reach over SSH, you can skip the manual notebook download —
-      run `/generate-spec <repo-ssh-url>` and it will ingest the `.ipynb`
-      files and `helper.py` into materials/notebooks/ for you. Transcripts
+   5. "Optional extra: if you also want the course's real code, find the
+      course at https://github.com/deeplearningai-eng/courses and run
+      `/ingest-course-repo <course-folder-URL>` before /generate-spec — it
+      adds the `.ipynb` files and `helper.py` under
+      courses/$ARGUMENTS/materials/notebooks/from-repo/. This is an addition
+      to step 2, not a replacement: the notebooks `.md` and the transcripts
       still come from the site."
 
 ## Don't