Skip to content

Commit

Permalink
Genericize setup.py, sleep longer when Deploying Linux Wheels, and pr…
Browse files Browse the repository at this point in the history 
…int Travis - CI Mac Python version
  • Loading branch information
@AjayP13
AjayP13 committed Nov 6, 2018
1 parent 2b52139 commit ed73d82
Show file tree
Hide file tree
Showing 5 changed files with 106 additions and 101 deletions.
164 changes: 83 additions & 81 deletions .gitlab-ci.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,90 +39,92 @@ Lint:
Test Python 2:
stage: Test
script:
- apt-get install python-pip -y
- pip2 install torch\>=0.4.1 # This is not in requirements.txt so must be installed like this
- pip2 install -U pip setuptools
- python2 setup.py install
- pip install gensim
- wget --quiet http://magnitude.plasticity.ai/word2vec/light/GoogleNews-vectors-negative300.magnitude -O GoogleNews-vectors-negative300.light.magnitude
- wget --quiet http://magnitude.plasticity.ai/word2vec/medium/GoogleNews-vectors-negative300.magnitude -O GoogleNews-vectors-negative300.medium.magnitude
- wget --quiet http://magnitude.plasticity.ai/word2vec/heavy/GoogleNews-vectors-negative300.magnitude -O GoogleNews-vectors-negative300.heavy.magnitude
- python2 -m pymagnitude.converter -i tests/models/word2vec.txt -o out.magnitude
- python2 -m pymagnitude.converter -i tests/models/word2vec.bin -o out.magnitude
- python2 -m pymagnitude.converter -i tests/models/glove.txt -o out.magnitude
- python2 -m pymagnitude.converter -i tests/models/fasttext.vec -o out.magnitude
- python2 -m pymagnitude.converter -i tests/models/zero.txt -o out.magnitude
- python2 -m pymagnitude.converter -i tests/models/large.txt -o out.magnitude
- python2 -m pymagnitude.converter -i tests/models/word2vec.txt -o out.magnitude -s
- python2 -m pymagnitude.converter -i tests/models/word2vec.bin -o out.magnitude -s
- python2 -m pymagnitude.converter -i tests/models/glove.txt -o out.magnitude -s
- python2 -m pymagnitude.converter -i tests/models/fasttext.vec -o out.magnitude -s
- python2 -m pymagnitude.converter -i tests/models/zero.txt -o out.magnitude -s
- python2 -m pymagnitude.converter -i tests/models/large.txt -o out.magnitude -s
- python2 -m pymagnitude.converter -i tests/models/word2vec.txt -o out.magnitude -s -a
- python2 -m pymagnitude.converter -i tests/models/word2vec.bin -o out.magnitude -s -a
- python2 -m pymagnitude.converter -i tests/models/glove.txt -o out.magnitude -s -a
- python2 -m pymagnitude.converter -i tests/models/fasttext.vec -o out.magnitude -s -a
- python2 -m pymagnitude.converter -i tests/models/zero.txt -o out.magnitude -s -a
- python2 -m pymagnitude.converter -i tests/models/large.txt -o out.magnitude -s -a
- python2 -m pymagnitude.converter -i tests/models/word2vec.bin -o w2v.magnitude
- python2 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude
- python2 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -s
- python2 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -s -a
- python2 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -v w2v.magnitude
- python2 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -v w2v.magnitude -s
- python2 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -v w2v.magnitude -a
- ls
- python2 -m tests.tests -i GoogleNews-vectors-negative300.light.magnitude -s GoogleNews-vectors-negative300.medium.magnitude -a GoogleNews-vectors-negative300.heavy.magnitude -- -v
- echo pass
# - apt-get install python-pip -y
# - pip2 install torch\>=0.4.1 # This is not in requirements.txt so must be installed like this
# - pip2 install -U pip setuptools
# - python2 setup.py install
# - pip install gensim
# - wget --quiet http://magnitude.plasticity.ai/word2vec/light/GoogleNews-vectors-negative300.magnitude -O GoogleNews-vectors-negative300.light.magnitude
# - wget --quiet http://magnitude.plasticity.ai/word2vec/medium/GoogleNews-vectors-negative300.magnitude -O GoogleNews-vectors-negative300.medium.magnitude
# - wget --quiet http://magnitude.plasticity.ai/word2vec/heavy/GoogleNews-vectors-negative300.magnitude -O GoogleNews-vectors-negative300.heavy.magnitude
# - python2 -m pymagnitude.converter -i tests/models/word2vec.txt -o out.magnitude
# - python2 -m pymagnitude.converter -i tests/models/word2vec.bin -o out.magnitude
# - python2 -m pymagnitude.converter -i tests/models/glove.txt -o out.magnitude
# - python2 -m pymagnitude.converter -i tests/models/fasttext.vec -o out.magnitude
# - python2 -m pymagnitude.converter -i tests/models/zero.txt -o out.magnitude
# - python2 -m pymagnitude.converter -i tests/models/large.txt -o out.magnitude
# - python2 -m pymagnitude.converter -i tests/models/word2vec.txt -o out.magnitude -s
# - python2 -m pymagnitude.converter -i tests/models/word2vec.bin -o out.magnitude -s
# - python2 -m pymagnitude.converter -i tests/models/glove.txt -o out.magnitude -s
# - python2 -m pymagnitude.converter -i tests/models/fasttext.vec -o out.magnitude -s
# - python2 -m pymagnitude.converter -i tests/models/zero.txt -o out.magnitude -s
# - python2 -m pymagnitude.converter -i tests/models/large.txt -o out.magnitude -s
# - python2 -m pymagnitude.converter -i tests/models/word2vec.txt -o out.magnitude -s -a
# - python2 -m pymagnitude.converter -i tests/models/word2vec.bin -o out.magnitude -s -a
# - python2 -m pymagnitude.converter -i tests/models/glove.txt -o out.magnitude -s -a
# - python2 -m pymagnitude.converter -i tests/models/fasttext.vec -o out.magnitude -s -a
# - python2 -m pymagnitude.converter -i tests/models/zero.txt -o out.magnitude -s -a
# - python2 -m pymagnitude.converter -i tests/models/large.txt -o out.magnitude -s -a
# - python2 -m pymagnitude.converter -i tests/models/word2vec.bin -o w2v.magnitude
# - python2 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude
# - python2 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -s
# - python2 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -s -a
# - python2 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -v w2v.magnitude
# - python2 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -v w2v.magnitude -s
# - python2 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -v w2v.magnitude -a
# - ls
# - python2 -m tests.tests -i GoogleNews-vectors-negative300.light.magnitude -s GoogleNews-vectors-negative300.medium.magnitude -a GoogleNews-vectors-negative300.heavy.magnitude -- -v

Test Python 3:
stage: Test
script:
- apt-get install build-essential -y
- apt-get install libffi-dev libssl-dev zlib1g-dev libncurses5-dev libncursesw5-dev libreadline-dev libsqlite3-dev -y
- apt-get install libgdbm-dev libdb5.3-dev libbz2-dev libexpat1-dev liblzma-dev tk-dev -y
- wget https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tar.xz
- tar xf Python-3.7.1.tar.xz
- cd Python-3.7.1 && ./configure --enable-shared && make -j 8 && make altinstall && cd ../
- echo $(pwd)/Python-3.7.1
- ls $(pwd)/Python-3.7.1
- export LD_LIBRARY_PATH=$(pwd)/Python-3.7.1:$LD_LIBRARY_PATH
- echo $LD_LIBRARY_PATH
- curl https://bootstrap.pypa.io/get-pip.py | python3.7
- python3.7 -m pip install torch\>=0.4.1 # This is not in requirements.txt so must be installed like this
- python3.7 -m pip install -U wheel setuptools # TEMP: removing upgrading of pip here, because the newest one is broken for Python 3
- python3.7 setup.py install
- python3.7 -m pip install gensim
- wget --quiet http://magnitude.plasticity.ai/word2vec/light/GoogleNews-vectors-negative300.magnitude -O GoogleNews-vectors-negative300.light.magnitude
- wget --quiet http://magnitude.plasticity.ai/word2vec/medium/GoogleNews-vectors-negative300.magnitude -O GoogleNews-vectors-negative300.medium.magnitude
- wget --quiet http://magnitude.plasticity.ai/word2vec/heavy/GoogleNews-vectors-negative300.magnitude -O GoogleNews-vectors-negative300.heavy.magnitude
- python3.7 -m pymagnitude.converter -i tests/models/word2vec.txt -o out.magnitude
- python3.7 -m pymagnitude.converter -i tests/models/word2vec.bin -o out.magnitude
- python3.7 -m pymagnitude.converter -i tests/models/glove.txt -o out.magnitude
- python3.7 -m pymagnitude.converter -i tests/models/fasttext.vec -o out.magnitude
- python3.7 -m pymagnitude.converter -i tests/models/zero.txt -o out.magnitude
- python3.7 -m pymagnitude.converter -i tests/models/large.txt -o out.magnitude
- python3.7 -m pymagnitude.converter -i tests/models/word2vec.txt -o out.magnitude -s
- python3.7 -m pymagnitude.converter -i tests/models/word2vec.bin -o out.magnitude -s
- python3.7 -m pymagnitude.converter -i tests/models/glove.txt -o out.magnitude -s
- python3.7 -m pymagnitude.converter -i tests/models/fasttext.vec -o out.magnitude -s
- python3.7 -m pymagnitude.converter -i tests/models/zero.txt -o out.magnitude -s
- python3.7 -m pymagnitude.converter -i tests/models/large.txt -o out.magnitude -s
- python3.7 -m pymagnitude.converter -i tests/models/word2vec.txt -o out.magnitude -s -a
- python3.7 -m pymagnitude.converter -i tests/models/word2vec.bin -o out.magnitude -s -a
- python3.7 -m pymagnitude.converter -i tests/models/glove.txt -o out.magnitude -s -a
- python3.7 -m pymagnitude.converter -i tests/models/fasttext.vec -o out.magnitude -s -a
- python3.7 -m pymagnitude.converter -i tests/models/zero.txt -o out.magnitude -s -a
- python3.7 -m pymagnitude.converter -i tests/models/large.txt -o out.magnitude -s -a
- python3.7 -m pymagnitude.converter -i tests/models/word2vec.bin -o w2v.magnitude
- python3.7 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude
- python3.7 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -s
- python3.7 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -s -a
- python3.7 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -v w2v.magnitude
- python3.7 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -v w2v.magnitude -s
- python3.7 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -v w2v.magnitude -a
- ls
- python3.7 -m tests.tests -i GoogleNews-vectors-negative300.light.magnitude -s GoogleNews-vectors-negative300.medium.magnitude -a GoogleNews-vectors-negative300.heavy.magnitude -- -v
- echo pass
# - apt-get install build-essential -y
# - apt-get install libffi-dev libssl-dev zlib1g-dev libncurses5-dev libncursesw5-dev libreadline-dev libsqlite3-dev -y
# - apt-get install libgdbm-dev libdb5.3-dev libbz2-dev libexpat1-dev liblzma-dev tk-dev -y
# - wget https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tar.xz
# - tar xf Python-3.7.1.tar.xz
# - cd Python-3.7.1 && ./configure --enable-shared && make -j 8 && make altinstall && cd ../
# - echo $(pwd)/Python-3.7.1
# - ls $(pwd)/Python-3.7.1
# - export LD_LIBRARY_PATH=$(pwd)/Python-3.7.1:$LD_LIBRARY_PATH
# - echo $LD_LIBRARY_PATH
# - curl https://bootstrap.pypa.io/get-pip.py | python3.7
# - python3.7 -m pip install torch\>=0.4.1 # This is not in requirements.txt so must be installed like this
# - python3.7 -m pip install -U wheel setuptools # TEMP: removing upgrading of pip here, because the newest one is broken for Python 3
# - python3.7 setup.py install
# - python3.7 -m pip install gensim
# - wget --quiet http://magnitude.plasticity.ai/word2vec/light/GoogleNews-vectors-negative300.magnitude -O GoogleNews-vectors-negative300.light.magnitude
# - wget --quiet http://magnitude.plasticity.ai/word2vec/medium/GoogleNews-vectors-negative300.magnitude -O GoogleNews-vectors-negative300.medium.magnitude
# - wget --quiet http://magnitude.plasticity.ai/word2vec/heavy/GoogleNews-vectors-negative300.magnitude -O GoogleNews-vectors-negative300.heavy.magnitude
# - python3.7 -m pymagnitude.converter -i tests/models/word2vec.txt -o out.magnitude
# - python3.7 -m pymagnitude.converter -i tests/models/word2vec.bin -o out.magnitude
# - python3.7 -m pymagnitude.converter -i tests/models/glove.txt -o out.magnitude
# - python3.7 -m pymagnitude.converter -i tests/models/fasttext.vec -o out.magnitude
# - python3.7 -m pymagnitude.converter -i tests/models/zero.txt -o out.magnitude
# - python3.7 -m pymagnitude.converter -i tests/models/large.txt -o out.magnitude
# - python3.7 -m pymagnitude.converter -i tests/models/word2vec.txt -o out.magnitude -s
# - python3.7 -m pymagnitude.converter -i tests/models/word2vec.bin -o out.magnitude -s
# - python3.7 -m pymagnitude.converter -i tests/models/glove.txt -o out.magnitude -s
# - python3.7 -m pymagnitude.converter -i tests/models/fasttext.vec -o out.magnitude -s
# - python3.7 -m pymagnitude.converter -i tests/models/zero.txt -o out.magnitude -s
# - python3.7 -m pymagnitude.converter -i tests/models/large.txt -o out.magnitude -s
# - python3.7 -m pymagnitude.converter -i tests/models/word2vec.txt -o out.magnitude -s -a
# - python3.7 -m pymagnitude.converter -i tests/models/word2vec.bin -o out.magnitude -s -a
# - python3.7 -m pymagnitude.converter -i tests/models/glove.txt -o out.magnitude -s -a
# - python3.7 -m pymagnitude.converter -i tests/models/fasttext.vec -o out.magnitude -s -a
# - python3.7 -m pymagnitude.converter -i tests/models/zero.txt -o out.magnitude -s -a
# - python3.7 -m pymagnitude.converter -i tests/models/large.txt -o out.magnitude -s -a
# - python3.7 -m pymagnitude.converter -i tests/models/word2vec.bin -o w2v.magnitude
# - python3.7 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude
# - python3.7 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -s
# - python3.7 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -s -a
# - python3.7 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -v w2v.magnitude
# - python3.7 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -v w2v.magnitude -s
# - python3.7 -m pymagnitude.converter -i tests/models/elmo_weights.hdf5 -o elmo.magnitude -v w2v.magnitude -a
# - ls
# - python3.7 -m tests.tests -i GoogleNews-vectors-negative300.light.magnitude -s GoogleNews-vectors-negative300.medium.magnitude -a GoogleNews-vectors-negative300.heavy.magnitude -- -v

Deploy to PyPI:
stage: Deploy to PyPI
Expand Down Expand Up @@ -180,7 +182,7 @@ Deploy Linux Wheels:
- export -f repeatcopytocibuildwheel
- export CIBW_PLATFORM=linux
- export CIBW_BUILD_VERBOSITY=3
- export CIBW_BEFORE_BUILD="echo 'Installing AWS...'; pip install awscli --upgrade; echo 'Sleeping...waiting for copy to complete...'; sleep 60; echo 'Project Directory After Copy:'; ls /project; echo 'AWS After Copy:'; ls /root/.aws/;"
- export CIBW_BEFORE_BUILD="echo 'Installing AWS...'; pip install awscli --upgrade; echo 'Sleeping...waiting for copy to complete...'; sleep 90; echo 'Project Directory After Copy:'; ls /project; echo 'AWS After Copy:'; ls /root/.aws/;"
- export CIBW_TEST_COMMAND="echo 'Uploading to AWS:'; ls /output; aws s3 sync /output/ s3://magnitude.plasticity.ai/wheelhouse/"
- pip2 install pip setuptools -U
- pip2 install git+https://github.com/plasticity-admin/cibuildwheel.git@macos-ssl
Expand Down
3 changes: 3 additions & 0 deletions .travis.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@ matrix:
- os: linux

script:
- echo python --version
- echo python2 --version
- echo python3 --version
- $PIP install pip setuptools -U
- $PIP install git+https://github.com/plasticity-admin/cibuildwheel.git@macos-ssl
- eval "buildwheel() { cibuildwheel --output-dir wheelhouse >>buildwheel.log 2>&1; }"
Expand Down
2 changes: 2 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -516,6 +516,8 @@ This is explicitly different from the [remote loading feature](#remote-loading),
vecs.query("king") # Returns: the vector for "king" quickly, even with no local model file downloaded
```

You can play around with a demo of this in a [Google Colaboratory Python Notebook](https://colab.research.google.com/drive/1zkPhoNM1NvbTmEk9gr0Jnt8hONrca1Fv).

This feature is extremely useful if your computing environment is resource constrainted (low RAM and low disk space), you want to experiment quickly with vectors without downloading and setting up large model files, or you are training a small model.
While there is some added network latency since the data is being streamed, Magnitude will still use an in-memory cache as specified by the [`lazy_loading`](#constructing-a-magnitude-object) constructor parameter. Since languages generally have a [Zipf-ian distribution](https://en.wikipedia.org/wiki/Zipf%27s_law), the network latency should largely not be an issue after the cache is warmed after being queried a small number of times.

Expand Down
Loading

0 comments on commit ed73d82

Please sign in to comment.