New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rewrite tools as Scrapy spiders #20

Open

juhanurmi opened this issue Sep 11, 2014 · 0 comments

Owner

juhanurmi commented Sep 11, 2014

Scrapy: The crawler is developed using Scrapy spider framework [1]. This onion site crawler is called OnionBot[2].

At the moment we have many different test tools for hidden services [3]. It might be cleaner solution to rewrite these tools with Scrapy.

[1] http://scrapy.org/
[2] https://github.com/juhanurmi/ahmia/tree/master/onionbot
[3] https://github.com/juhanurmi/ahmia/tree/master/tools

bsloan mentioned this issue

Rewriting some tools as scrapy spiders #30

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment