PyMuPDFScraper ignores the User-Agent session — all PDF downloads go out as python-requests, SEC EDGAR returns 403

[aleps001]

Summary

PyMuPDFScraper receives the User-Agent-bearing requests.Session that Scraper builds, but then downloads PDFs with the bare requests module instead. Every .pdf request goes out with the default python-requests/2.x User-Agent, so sites that require an identified client — notably SEC EDGAR (sec.gov fair-access policy) — return 403 Forbidden for every PDF, while HTML pages from the same host scrape fine with the configured UA.

For finance/regulatory research this silently drops the primary sources (SEC rule releases, filing specifications, etc. are all PDFs).

Present in 0.15.1 and current master. Independent of #1846, found in the same retrieval-quality investigation.

Mechanism

Scraper correctly builds and passes a session with the configured UA (gpt_researcher/scraper/scraper.py):

self.session = requests.Session()
self.session.headers.update({"User-Agent": user_agent})
...
contents = await asyncio.gather(
    *(self.extract_data_from_url(url, self.session) for url in self.urls)
)

PyMuPDFScraper stores it — and never uses it (gpt_researcher/scraper/pymupdf/pymupdf.py):

def __init__(self, link, session=None):
    self.link = link
    self.session = session          # <-- stored...

def scrape(self):
    ...
    response = requests.get(self.link, timeout=(5, 30), stream=True)   # <-- ...ignored

The SSL-fallback retry a few lines below has the same problem.

Reproduction

• USER_AGENT configured (any identified UA).
• Research any query whose sources include SEC PDFs, e.g. https://www.sec.gov/files/rules/proposed/2018/33-10515.pdf.

Observed: Error loading PDF : https://www.sec.gov/files/rules/proposed/2018/33-10515.pdf 403 Client Error: Forbidden, while https://www.sec.gov/investment/exchange-traded-funds-small-entity-compliance-guide (HTML, same host, same run) scrapes successfully.

Fix that worked for us

+                http = self.session or requests
                 try:
-                    response = requests.get(self.link, timeout=(5, 30), stream=True)
+                    response = http.get(self.link, timeout=(5, 30), stream=True)
                     response.raise_for_status()
                 except requests.exceptions.SSLError:
                     ...
-                    response = requests.get(self.link, timeout=(5, 30), stream=True, verify=False)
+                    response = http.get(self.link, timeout=(5, 30), stream=True, verify=False)

requests.Session.get and requests.get share the same signature, so self.session or requests is a drop-in.

Measured: the exact PDF above went from 403 to a successful download + parse (640k chars extracted).

Happy to open a PR if the approach looks right.