downloader: catch more error situations

[ParthS007]

addresses #98 #99

[tiborsimko]

It would be nice to list all the exceptions instead of one single catch all.

BTW the scenario described in #98 does not for me fully with this PR. I get:

ConnectionResetError: [Errno 104] Connection reset by peer
...
During handling of the above exception, another exception occurred:
...
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
...
During handling of the above exception, another exception occurred:
...
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Is this improved with sequel PRs? Shall we address them in one go?

[ParthS007]

Yes, This PR addresses #98 and will be completed in the sequel of PRs as we discussed.

[tiborsimko]

So let's perhaps squash them together, it'll be easier to test that way!

[tiborsimko]

The general except catch-all is not the best paradigm, as it can mask some error situations, but it is OK for now for partial addressing...

[tiborsimko]

Repetitive download leads to an issue. Here's a test case. The current behaviour is:

$ git clean -d -ff -x
$ cernopendata-client download-files --recid 5500 --verify
==> Downloading file 1 of 11
  -> File: ./5500/BuildFile.xml
  -> Progress: 0/0 KiB (100%)
==> Verifying file BuildFile.xml...
  -> Expected size 305, found 305
  -> Expected checksum adler32:ff63668a, found adler32:ff63668a
...
==> Downloading file 11 of 11
  -> File: ./5500/mass4l_combine.png
  -> Progress: 90/90 KiB (100%)
==> Verifying file mass4l_combine.png...
  -> Expected size 93152, found 93152
  -> Expected checksum adler32:62e0c299, found adler32:62e0c299
==> Success!
$ $ cernopendata-client download-files --recid 5500 --verify

==> Downloading file 1 of 11
  -> File {} is complete. Skip download.

Expected behaviour:

• do not stop after 1st file
• say properly which file is OK (f-string), something like:
  "File ./5500/foo.py is up to date; skipping download
• note that for xrootd protocol, the files are always re-downloaded, there is no check. It would be good to harmonise the behaviour, e.g. always-re-download or always-skip for both HTTP and XRootD protocols

[ParthS007]

• Pushed more changes, now you can check resuming-downloading with the requests library.

• Added a couple of functions separating the functionalities, names can be changed.

• I have harmonized the re-download now files will always-re-download if they are all present for now. We can change it once we are done with adding resuming downloads for all libs.

[tiborsimko]

'record id' -> 'record ID'

Can you please update this part, even if it is not part of the PR? Also in other places:

$ rg 'record id'
cernopendata_client/searcher.py
34:    :return: Boolean after verifying the record id
49:                msg="The record id number you supplied is not valid.",
73:            msg="The record id number you supplied is not valid.",
80:    """Return record id by either title or doi.
89:    :return: record id

cernopendata_client/validator.py
29:            msg="You must supply a record id number as an " "input using -r flag.",

Could be a separate commit.

[tiborsimko]

Tests of the "resume interrupted download" feature passed locally with requests downloaded, and otherwise in general:

  py27: commands succeeded
  py36: commands succeeded
  py37: commands succeeded
  py38: commands succeeded
  py39: commands succeeded
  congratulations :)