Rewriting some tools as scrapy spiders

[bsloan]

For what it's worth, here's an attempt to rewrite some of Ahmia's test tools as Scrapy spiders (#20). The original test tools are unchanged.

Needs review and additional testing to ensure any differences between my own environment and Ahmia production environment don't cause problems, and generally to assess whether these changes are worthwhile. A list of rewritten tools plus use / notes for each is below (commands are executed from ahmia/tools directory).

• finder_spider

Collects links to onions and then PUTs the list of links to Ahmia endpoint. To run:

$ scrapy crawl finder_spider

• t2w_domain_spider

Iterates a list of tor2web nodes and saves their visited domains lists to disk. To run:

$ scrapy crawl t2w_domain_spider

• t2w_filter_spider

Downloads the list of hashes of blacklisted onions from a set of tor2web nodes and saves the blacklist to disk.

$ scrapy crawl t2w_filter_spider

• backlink_spider

A backlink counting utility; accepts 4 arguments: target, host, links, count (boolean).

The only required argument is target, which instructs the crawler what URL to search backlinks for. Args are passed in the format expected by scrapy, for example:

$ scrapy crawl backlinkspider -a target="http://www.nytimes.com/" -a count=true

• test_services_spider

Tests whether hidden services are online.

$ scrapy crawl test_services_spider

[juhanurmi]

Wau! Sound like very great work and effort! I will test to code as soon as possible :)