Predict command fails

[liranringel]

First I want to thank the authors for this great work! I might find it useful for my research.

I encountered 3 problems:

1. in evaluator/dataset_evaluator.py, in the usage of hf_hub_download, I got an exception because it was used that way: hf_hub_download(repo_id="datasets/tau/scrolls", filename="metrics/scrolls.py") instead of hf_hub_download(repo_id="tau/scrolls", filename="metrics/scrolls.py", repo_type="dataset"). I don't know why it worked for you, perhaps there was a breaking change in the datasets library recently. Would you want me to open a PR for that?
2. The generate script (python scripts/execute.py scripts/commands/generate.py {dataset}_{model}_{split} --checkpoint_path path/to/model/folder) took a very long time, much more than the fine-tuning of 256-bart. There was a warning that might be related saying:

Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector

Edit: I now noticed that this warning is emitted only when I use more than one GPU. However, it is still slower than expected.
3. It failed with the following exception:

Traceback (most recent call last):
File "/home/liranringel/scrolls/baselines/scripts/execute.py", line 53, in
main(command_dict, unknown)
File "/home/liranringel/scrolls/baselines/scripts/execute.py", line 33, in main
runpy.run_module(module_name, run_name="main")
File "/home/liranringel/miniconda3/envs/mem/lib/python3.9/runpy.py", line 228, in run_module
return _run_code(code, {}, init_globals, run_name, mod_spec)
File "/home/liranringel/miniconda3/envs/mem/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/liranringel/scrolls/baselines/src/run.py", line 789, in
main()
File "/home/liranringel/scrolls/baselines/src/run.py", line 656, in main
metrics = trainer.evaluate(metric_key_prefix="eval")
File "/home/liranringel/miniconda3/envs/mem/lib/python3.9/site-packages/transformers/trainer_seq2seq.py", line 131, in evaluate
eval_preds = self._post_process_function(untokenized_eval_dataset, eval_loop_output.predictions)
File "/home/liranringel/miniconda3/envs/mem/lib/python3.9/site-packages/transformers/trainer_seq2seq.py", line 326, in _post_process_function
assert len(untokenized_eval_dataset) == len(self.eval_dataset)
AssertionError

[eladsegal]

Hi Liran, thanks for bringing these issues to our attention!

1. You are exactly right, the huggingface-hub package which is installed unpinned via the datasets package installation had a breaking change. I made the changes as you suggested.

2+3. I tried replicating the issue you described with the qasper dataset in the environment where our original experiments were carried out and also in a freshly installed environment, but couldn't encounter any problems. Can you specify the dataset and model that you used?

[jzhang38]

I also encountered problem 3.
Model: T5 (by passing "t5-base" to model_name_or_path). Dataset: NarrativeQA.
Command:
CUDA_VISIBLE_DEVICES=0 python src/run.py --dataset_config_name narrative_qa
--dataset_name tau/scrolls --drop_duplicates_in_eval True
--greater_is_better False --metric_for_best_model loss
--model_name_or_path t5-base
--num_train_epochs 2 --logging_steps 10 --preprocessing_num_workers 1
--predict_with_generate True --num_beams 1 --trim_very_long_strings
--attention_window 1024 --max_target_length 1024 --fp16 True
--per_device_eval_batch_size 512 --do_eval True --global_attention_first_token True
--max_source_length 512
--output_dir outputs
--run_name outputs

[eladsegal]

@jzhang38 Can you please provide the error message and the output from pip freeze?

[jxiw]

I also have this problem follow using the same command in the repo. @liranringel Did you find the solution for this?

export dataset=quality
export model=led-1024
export split=validation

CUDA_VISIBLE_DEVICES=0 python scripts/execute.py scripts/commands/generate.py ${dataset}_${model}_${split} --checkpoint_path outputs/allenai-led-base-16384_global_1024_1_2e-05_16384_scrolls_quality_town-common-2/

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 1262/1262 [04:22<00:00,  5.18it/s]
Traceback (most recent call last):
  File "scripts/execute.py", line 53, in <module>
    main(command_dict, unknown)
  File "scripts/execute.py", line 33, in main
    runpy.run_module(module_name, run_name="__main__")
  File "/miniconda/lib/python3.8/runpy.py", line 208, in run_module
    return _run_code(code, {}, init_globals, run_name, mod_spec)
  File "/miniconda/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/xxx/scrolls/baselines/src/run.py", line 789, in <module>
    main()
  File "/xxx/scrolls/baselines/src/run.py", line 656, in main
    metrics = trainer.evaluate(metric_key_prefix="eval")
  File "/miniconda/lib/python3.8/site-packages/transformers/trainer_seq2seq.py", line 131, in evaluate
    eval_preds = self._post_process_function(untokenized_eval_dataset, eval_loop_output.predictions)
  File "/miniconda/lib/python3.8/site-packages/transformers/trainer_seq2seq.py", line 326, in _post_process_function
    assert len(untokenized_eval_dataset) == len(self.eval_dataset)
AssertionError
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 1262/1262 [04:23<00:00,  4.80it/s]

[liranringel]

@jxiw no, sorry.
I left my work on it.

[jxiw]

@eladsegal thank the authors for this great work! Is it possible for you to release those model numbers in validation dataset? That be very helpful for me! thank.

[eladsegal]

Hi @jxiw, I see @UriSha already gave you an answer regarding validation scores in #7.
Regarding the assertion error you get, I wasn't able to reproduce it in the original SCROLLS environment and also not in a freshly installed one.
Can you please share the output from pip freeze?