feat: add email digest ingestion for job-alert emails - #21
Open
creativereason wants to merge 1 commit into
Open
Conversation
Adds scripts/email-job-search.mjs — a provider-agnostic ingestion source for job-alert digest emails (Indeed, TheLadders, Lensa, etc.). It has no mail integration of its own: it takes pre-exported email JSON as input, so any mail source can feed it (Gmail export, IMAP, or an external scheduler like OpenClaw's gog-based Gmail triage, documented as one example wiring). Extraction uses a strict, conservative filtering prompt — reject a listing whenever title tier, salary, or location eligibility is unclear rather than including it on a hopeful match, since a false positive costs real review time and a missed listing costs nothing (it stays in the source digest). Also, while building this: - Extracted shared scripts/lib/github-contents.mjs (githubGet/githubPut) and scripts/lib/ai-provider.mjs (callAI), deduplicating identical logic that was previously copy-pasted across job-search.mjs and backfill-ai-summaries.mjs. - ai-provider.mjs adds native Anthropic Messages API support (AI_PROVIDER=anthropic, the default), so an Anthropic-shaped AI_API_KEY works instead of only OpenAI-compatible chat/completions endpoints. - Added a company+role dedup fallback (scripts/lib/job-key.mjs, mirroring lib/jobs.ts's jobKey) alongside the existing url-based dedup, since some platforms (TheLadders, Lensa) send tracking-redirect URLs that mint a fresh token per send — the same posting re-appearing in a later digest wouldn't match on url alone. - Applied the same strict-filtering and dedup-fallback changes to job-search.mjs's WebSearch pass for consistency. Verified against real Gmail digest content from Indeed, LinkedIn, TheLadders, and Lensa: correctly extracts and adds clear matches, correctly rejects weak/ambiguous listings (wrong discipline, under salary floor, unconfirmed remote), and correctly catches a same-company+role duplicate under a new tracking URL.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Problem This Solves
There's no way to turn job-alert emails you already receive (Indeed, TheLadders, Lensa, LinkedIn, etc.) into pending-queue entries — every existing ingestion source (WebSearch pass, Indeed MCP, career-page scraper) runs its own search instead of reading mail you already get.
Why This Change Was Made
Personal automation needed a script deckhandAI users can point any mail export at (Gmail, IMAP, mbox — no vendor lock-in), following the project's existing zero-infrastructure principle. Building it surfaced two real bugs in the existing
job-search.mjs/backfill-ai-summaries.mjsscripts (duplicated GitHub/AI-call logic, and an AI-provider mismatch that likely silently broke the weekly WebSearch pass for Anthropic-keyed setups), so this PR fixes those too.User Impact
scripts/email-job-search.mjs: takes a JSON array of already-exported emails (--input <file>or stdin), extracts qualifying postings per yourdata/config.jsonpreferences, dedupes, and writes to the pending queue — same contract as the existing scripts.scripts/job-search.mjsandscripts/backfill-ai-summaries.mjsnow sharescripts/lib/github-contents.mjsandscripts/lib/ai-provider.mjsinstead of duplicating that logic (net LOC reduction on both files).scripts/lib/ai-provider.mjsadds native Anthropic support (AI_PROVIDER=anthropic, the default) — previously these two scripts only spoke OpenAI-compatiblechat/completions, so an Anthropic-shapedAI_API_KEY(the common case if you already use Claude elsewhere) would 401.email-job-search.mjsandjob-search.mjsgot a stricter, more conservative filtering prompt (reject on ambiguous title tier/salary/location rather than including hopefully) and acompany+rolededup fallback (scripts/lib/job-key.mjs, mirrorslib/jobs.ts'sjobKey) for platforms like TheLadders/Lensa whose email links are per-send tracking redirects, not stable URLs.docs/scheduled-job-search.md, one-line README mention.Evidence
pnpm lintclean on all touched/new files.