Canonical T-Res resources

[thobson88]

The resources/ subdirectory is .gitignored due to the large size of the data files on which T-Res depends, but there is currently no shared storage location containing a canonical set of resources.

This makes deployment more difficult and also causes problems with integration tests which depend on the resources and cannot be run with consistent results unless a common resources are used.

Steps:

• identify a canonical set of resources (on existing VMs/disks),
• collect these resources into a central storage location (with appropriate compression),
• write a script to automatically download & unpack them into the resources/ subdirectory,
• separately include a subset of resources under the tests/ subdirectory, for unit test fixtures,
• mark any tests depending on the full, canonical resources to be skipped by default when running pytest,
• repeat these steps for the experiments/ subdirectory.

[thobson88]

Identify a canonical set of resources

Latest and most complete resources (afaics) are to be found at:

• deployed-toponym-resolution VM, under /home/lukehare/T-Res/resources
• toponymVM2.0 under /resources/home/mcollardanuy/toponym-resolution/resources

[thobson88]

Collect resources into a central storage location (with appropriate compression)

Script for bundling up the resources:

#!/bin/bash

# Script that creates a zip file of T-Res resources.

# Run this script from the resources/ directory and pass the output directory
# in which the file `resources.zip` will be created.

zip "$1/resources.zip" publication_metadata.json
zip -ur "$1/resources.zip" deezymatch/
zip -ur "$1/resources.zip" models/
zip -ur "$1/resources.zip" rel_db/
zip -ur "$1/resources.zip" wikidata/
zip -ur "$1/resources.zip" wikipedia/
echo "Created T-Res resources.zip at $1/resources.zip"

To be run from /home/lukehare/T-Res/resources on the deployed-toponym-resolution VM.

Note: this script takes several minutes to run because some subfolders contain many small files (e.g. deezymatch/candidate_vectors/wkdtalts_w2v_ocr/embeddings/). It may be worth tweaking so that those subfolders are not compressed but instead added with level 0 compression. This would save time when uncompressing (at the expense of a large zip file).

Update: files now on Azure in tres storage account, resources container.

[thobson88]

With these resources in place (on Azure, at least), I'm now making sure all tests pass on Rosie's dev_rw branch, in preparation for merging that PR.

[thobson88]

Script to fetch resources from Azure

Added to the existing PR as resources/fetch_resources.py. Requires AzCopy and a valid SAS token.

This is a stop-gap solution intended for internal use, pending public release of the resource files #279.

[thobson88]

All tests now pass on a new checkout (of dev_rw) after running the script fetch_resources.py with the command:

python resources/fetch_resources.py