You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It assumes that link have to end with the .pdf extension to be parsed as pdf documents using PyMuPDFScraper. Not all pdf links returned by SemanticScholar end in the .pdf extension. Right now, those PDFs that do not end in the .pdf extension are being scraped as if they were regular web urls using the incorrect scrapper.
The text was updated successfully, but these errors were encountered:
Describe the bug
PDFs are not always scraped correctly.
To Reproduce
Attempt to fetch content using the SemanticScholar retriever
Expected behavior
PDFs are scraped correctly
Additional context
The issue lies here:
https://github.com/assafelovic/gpt-researcher/blob/0034ad08d364b34306ab9c412afb61f93eccbf11/gpt_researcher/scraper/scraper.py#L84C1-L84C34
It assumes that link have to end with the
.pdf
extension to be parsed as pdf documents usingPyMuPDFScraper
. Not all pdf links returned by SemanticScholar end in the .pdf extension. Right now, those PDFs that do not end in the.pdf
extension are being scraped as if they were regular web urls using the incorrect scrapper.The text was updated successfully, but these errors were encountered: