Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scraper creates "https/" articles directory #1

Open
carlgieringer opened this issue Nov 1, 2017 · 1 comment
Open

scraper creates "https/" articles directory #1

carlgieringer opened this issue Nov 1, 2017 · 1 comment

Comments

@carlgieringer
Copy link
Member

When running the scraper from scratch, there appears a directory articles/https/. There are some articles under this directory, and I don't think they match up with articles not under this directory in the browse view. E.g. articles/https//www.nytimes.com/ don't appear along with articles/www.nytimes.com.

@carlgieringer
Copy link
Member Author

This is due to a legacy artifact in models.Article#filename:

elif ans.startswith('https://'):
            # Terrible hack for backwards compatibility from when https was stored incorrectly,
            # perpetuating the problem
            return 'https:/' + ans[len('https://'):]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant