feat: daily arXiv researcher — scan papers for gh-aw improvement opportunities by Copilot · Pull Request #49609 · github/gh-aw

Copilot · 2026-08-01T19:05:16Z

Daily Claude workflow that skims latest arXiv papers (cs.AI/SE/LG), extracts actionable gh-aw improvement opportunities, creates a research discussion when findings exist, and maintains a permanent per-paper ledger in repo-memory.

New files

.github/workflows/daily-arxiv-researcher.md — daily scheduled workflow (Claude engine, max-ai-credits: 250)
.github/scripts/arxiv-fetch-and-filter.py — DataOps helper: parses arXiv Atom XML, deduplicates against cache-memory before agent starts; extracted to a separate file to avoid YAML/Python indentation conflicts in heredoc blocks
.github/workflows/daily-arxiv-researcher.lock.yml — compiled output

Optimization design

Technique	What it buys
DataOps step: `curl` + Python parse	Zero AI credits for HTTP fetch and XML parsing
`cache-memory` dedup (90-day key)	Already-seen paper IDs are skipped across daily runs
`paper-screener` sub-agent (`model: small`)	Parallel cheap relevance filter; frontier model never sees irrelevant papers
`opportunity-extractor` sub-agent (`model: large`, max 8)	Strong reasoning bounded to survivors only
Staged noop exits	Stops at the first stage with nothing to do
Abstract truncated to 1,200 chars	Bounds per-paper input cost without losing signal
`repo-memory` on `memory/arxiv-paper-ledger`	Indefinite, Git-auditable ledger of every investigated paper

Flow

schedule daily ~8:00 UTC
  step: curl export.arxiv.org → 40 papers (cs.AI + cs.SE + cs.LG)
  step: parse Atom XML, filter seen IDs
  agent:
    ├─ 0 new → noop
    ├─ paper-screener × N (small, parallel)
    │    └─ none relevant → update ledger + noop
    ├─ opportunity-extractor × ≤8 (large)
    ├─ update repo-memory ledger (all processed papers)
    └─ create-discussion with opportunities  OR  noop

Security

ANTHROPIC_API_KEY: standard Claude engine secret, used by other Claude workflows in this repo
export.arxiv.org: public read-only API, no credentials sent

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-08-01T19:26:14Z

@@ -0,0 +1,94 @@
+"""


@copilot use JavaScript/github-script inlined into the AW, then remove this file

Done in 85e3b9b. The Python script has been removed and its logic is now inlined as an actions/github-script@v9.0.0 step in the workflow. The JavaScript uses Node.js built-in https and fs to fetch and parse the arXiv Atom XML, with the same dedup logic against cache-memory. The export.arxiv.org entry was also dropped from network: since custom steps run outside the AWF firewall sandbox.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot

Pull request overview

Adds a daily Claude workflow to discover arXiv research relevant to gh-aw improvements.

Changes:

Fetches, parses, and deduplicates recent arXiv papers.
Uses sub-agents to screen papers and extract opportunities.
Persists a research ledger and creates discussions for findings.

Show a summary per file

File	Description
`.github/workflows/daily-arxiv-researcher.md`	Defines fetching, analysis, memory, and reporting behavior.
`.github/workflows/daily-arxiv-researcher.lock.yml`	Compiled GitHub Actions workflow.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 1/2 changed files
Comments generated: 5
Review effort level: Balanced

+
+safe-outputs:
+  create-discussion:
+    category: "research"


+        while ((em = entryRe.exec(xml)) !== null) {
+          const entry = em[1];
+          const idUrl = getText('id', entry);
+          const arxivId = idUrl.replace(/.*abs\//, '').trim();


+        try {
+          xml = await new Promise((resolve, reject) => {
+            const req = https.get(ARXIV_URL, { timeout: 30000 }, res => {
+              const chunks = [];
+              res.on('data', c => chunks.push(c));
+              res.on('end', () => resolve(Buffer.concat(chunks).toString('utf8')));
+            });
+            req.on('error', reject);
+            req.on('timeout', () => { req.destroy(); reject(new Error('request timed out')); });
+          });
+        } catch (e) {
+          core.warning(`arXiv fetch failed: ${e.message}`);
+        }


+Write the updated ledger to `/tmp/gh-aw/repo-memory/default/paper-ledger.md`.
+Write the updated index to `/tmp/gh-aw/repo-memory/default/paper-index.json`.


+  - name: Fetch and parse arXiv papers
+    uses: actions/github-script@v9.0.0


github-actions · 2026-08-03T05:28:25Z

🎉 This pull request is included in a new release.

Release: v0.84.3

feat: add daily-arxiv-researcher workflow for arXiv paper scanning

1430043

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI assigned Copilot and pelikhan Aug 1, 2026

Copilot AI changed the title ~~feat: daily-arxiv-researcher — scan arXiv papers for gh-aw improvement opportunities~~ feat: daily arXiv researcher — scan papers for gh-aw improvement opportunities Aug 1, 2026

Copilot finished work on behalf of pelikhan August 1, 2026 19:06

Copilot AI requested a review from pelikhan August 1, 2026 19:06

Copilot created this pull request from a session on behalf of pelikhan August 1, 2026 19:06 View session

pelikhan reviewed Aug 1, 2026

View reviewed changes

Copilot started work on behalf of pelikhan August 1, 2026 19:26 View session

feat: inline arxiv fetch+parse as github-script JS, remove Python script

77c62fa

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI requested a review from pelikhan August 1, 2026 19:43

Copilot finished work on behalf of pelikhan August 1, 2026 19:43

pelikhan marked this pull request as ready for review August 1, 2026 19:44

Copilot AI review requested due to automatic review settings August 1, 2026 19:44

pelikhan merged commit 10e21b7 into main Aug 1, 2026
1 check passed

pelikhan deleted the copilot/create-daily-agentic-workflow branch August 1, 2026 19:44

Copilot started reviewing on behalf of pelikhan August 1, 2026 19:45 View session

Copilot AI reviewed Aug 1, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: daily arXiv researcher — scan papers for gh-aw improvement opportunities - #49609

feat: daily arXiv researcher — scan papers for gh-aw improvement opportunities#49609
pelikhan merged 2 commits into
mainfrom
copilot/create-daily-agentic-workflow

Copilot AI commented Aug 1, 2026 •

edited

Loading

Uh oh!

pelikhan Aug 1, 2026

Uh oh!

Copilot AI Aug 1, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		Write the updated ledger to `/tmp/gh-aw/repo-memory/default/paper-ledger.md`.
		Write the updated index to `/tmp/gh-aw/repo-memory/default/paper-index.json`.

		- name: Fetch and parse arXiv papers
		uses: actions/github-script@v9.0.0

Uh oh!

Conversation

Copilot AI commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New files

Optimization design

Flow

Security

Uh oh!

pelikhan Aug 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Aug 1, 2026 •

edited

Loading