Converge automation lines: adopt prod-only code, land 2 P0 fixes, fail loudly on GitHub auth - #48
Merged
Merged
Conversation
…m schema Port the content-automation line off the orphaned String-schema fork onto main (enum SkillStatus), fixing the P2032 drift that broke takoapi-daily-sync for ~1 week (the deployed sync image's Prisma client expected String; the prod DB column is a native enum). - scripts/daily-sync.ts: create() now sets status:"APPROVED" + source:"CURATED" (main defaults to PENDING/hidden). Built via Dockerfile.sync -> takoapi-sync image, whose Prisma client is now generated from main's enum schema. - Dockerfile.sync: install tsx at build so runtime `npx tsx` is offline-safe. - cloudbuild.sync.yaml: build the sync image (docker build -f Dockerfile.sync). - src/app/api/cron/check-github/route.ts: port the missing weekly GitHub dead-link checker, aligned to main's isAuthorizedCron (Bearer) convention. Verified: takoapi-daily-sync (4Gi) succeeds - 5385 skills, 10307 updated, 17 deduped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
scrape-skill-details.ts extracted githubUrl from github.com/openclaw/skills/tree/main/skills/<author>/<name> links on clawskills.sh pages — a repo that does not exist (HTTP 404). This had polluted 5028 of 5385 Skill.githubUrl rows in prod (now NULLed out in a separate one-time DB remediation). clawskills.sh exposes no per-skill source repo: the canonical per-skill link is the ClawHub page (already stored as clawHubUrl/clawSkillsUrl), and the only GitHub link on a page is the site-wide "Star" button to the aggregate list VoltAgent/awesome-openclaw-skills. So the scraper now leaves githubUrl untouched (NULL) for clawskills-sourced skills; genuine githubUrl values come only from source=GITHUB_SCRAPE skills. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes the content-automation convergence (after daily-sync + check-github). Ports the weekly-digest Cloud Scheduler endpoint onto the main line, which previously 404'd in prod because the route was never deployed. - add `resend` dependency (email delivery) + refresh package-lock - port src/lib/email.ts from rescue/content-automation-suite (Resend helper; sendWeeklyDigest builds new-skill/top-skill/total digest) - add src/app/api/cron/weekly-digest/route.ts, mirroring the import-hosted/check-github convention: force-dynamic, maxDuration 300, GET/POST = handle. Auth rewritten from the legacy `x-cron-secret` header to isAuthorizedCron(req) (Bearer <CRON_SECRET>). RESEND_API_KEY + CRON_SECRET are already wired into the Cloud Run `takoapi` service via Secret Manager. Scheduler header switch to `Authorization: Bearer` handled out-of-band. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
import-hosted upserted by slugify(card.name) with no guards on the update path, so a card in the community registry whose name collided with an existing entry silently overwrote that entry's endpoint/cardUrl while keeping its APPROVED status — a takeover vector via a single registry PR. Now guard: skip rows owned by another publisher or of kind PROJECT, and drop cross-origin endpoint changes back to PENDING for admin re-review. scrape-agents forced status=APPROVED on every refresh (and re-created deleted rows as APPROVED), so a scheduled run undid admin REJECT/DELETE — the root cause behind junk repos resurfacing at the top of /registry. Now pre-load REJECTED/DISABLED slugs and skip them, and no longer touch status on update; only newly discovered repos land APPROVED. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fetchOne used redirect: "follow" and only checked the initial URL string, so a public URL could 302 into 169.254.169.254 (cloud metadata) or a name could resolve to an internal IP unchecked. This path is reachable from the public /api/agents/submit endpoint, not just cron. Follow redirects manually (max 3), revalidating every hop, and add assertPublicHost() which resolves A/AAAA and rejects private / loopback / link-local / CGNAT / benchmarking (198.18/15) targets. Prod-gated to match the existing host guard so localhost cards still work in dev. Residual connect-level TOCTOU noted for socket-level follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All three cron responses now include durationMs so the scheduler log can
spot runs approaching the timeout (a truncated run leaves partial,
non-transactional writes). health gains maxDuration=120 and counts
writeErrors instead of swallowing them via .catch(()=>{}). Clarified that
maxDuration is not the effective ceiling on Cloud Run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
05: full review of every auto-ingestion path with confirmed evidence, pro/con debate, and verdict per finding. 06: authoritative cron job list (frequency, params, auth) with idempotent gcloud setup — the schedules were previously out-of-band knowledge with no config in the repo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… claude/merge-content-automation
…ew-739127' into claude/merge-content-automation
… into claude/merge-content-automation
…edence Follow-ups on top of the three merged automation branches: - scrape-agents: a 401 from GitHub search was logged and swallowed, so the cron answered 200 with 0 results while the catalog quietly stopped growing. That is exactly how an expired PAT went unnoticed for 19 days (no PROJECT row was written between 2026-07-13 and 2026-08-01). Now a 401 throws, and a run that finds zero repos across all 20 topic queries throws too — both surface as a 500 the scheduler records as a failed job. - check-github-status: `%` binds tighter than `+`, so the progress condition was `ok + 404 + (error % 100) === 0` and only ever fired when all counters were zero. Parenthesized the sum. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Converges the content-automation work that had drifted apart, and fixes the failure mode that silently killed the scraper for 19 days.
Why
Production was running
c5d33ea2— the tip of the unmerged branchclaude/brave-sammet-6bb73b(deployed 2026-07-04). So prod containedcheck-github+weekly-digestcron routes,src/lib/email.tsandscripts/daily-sync.tsthat did not exist on main: a normal deploy from main would have deleted two live cron routes and left their Cloud Scheduler jobs 404ing.What's in here
brave-sammet— brings the prod-only code into main (removes the deploy landmine).weekly-digestis inert until a subscriber signup flow exists; it is merged as-is because it is already what prod runs.automation-mechanism-review— two P0 fixes plus hardening:scrape-agentswrotestatus: "APPROVED"on every update, so an admin REJECT (or a delete) was undone on the next run — the root cause of junk at the top of the registry. Now tombstoned slugs are skipped and the update path never touchesstatus.competent-gagarin— stopsscrape-skill-details.tswriting a non-resolvingopenclaw/skillsgithubUrl.%-precedence bug in thecheck-github-statusprogress log.Verified
npm install·prisma generate·tsc --noEmit·eslint·npm run buildall clean. Build output contains all five cron routes (scrape-agents,import-hosted,health,check-github,weekly-digest).Separately, the expired
GITHUB_TOKENwas rotated (Secret Manager version 3) and the scraper verified live: 949 projects refreshed, 53 newly imported (1377 → 1430).🤖 Generated with Claude Code