Issue with hydra - Error merging data/dataset=cstm Key 'defaults' not in 'FairseqConfig'

[aprzez]

Hello,

I left the below comment under issue #3 but since it's closed I am not sure the comment will be seen.

I am trying to run the training on a custom dataset and also experiencing this issue.
The file cstm.yaml is placed in here auto_avsr/conf/data/dataset and looks like this:
`defaults:

self
root: "/content/drive/MyDrive/sepedi/data/preprocess_datasets"
label_dir: "labels"
train_file: "train_labels.csv"
val_file: "val_labels.csv"
test_file: "test_labels.csv"`

As suggested above, I tried renaming the conf/config.yaml file. However, when I run:

!python main.py exp_dir=exp \
exp_name=trainaudio \
data.modality=audio \
ckpt_path='content/drive/MyDrive/LRS3_A_WER1.0/model.pth' \
+data/dataset=cstm \ trainer.num_nodes=1

I get:

Error merging data/dataset=cstm Key 'defaults' not in 'FairseqConfig' full_key: defaults reference_type=Optional[FairseqConfig] object_type=FairseqConfig

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

When running:

%env HYDRA_FULL_ERROR=1 !python main.py exp_dir=exp \ exp_name=trainaudio \ data.modality=audio \ ckpt_path='content/drive/MyDrive/LRS3_A_WER1.0/model.pth' \ +data/dataset=cstm \ trainer.num_nodes=1

I get:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 720, in _merge_config
ret = OmegaConf.merge(cfg, loaded_cfg)
File "/usr/local/lib/python3.10/dist-packages/omegaconf/omegaconf.py", line 321, in merge
target.merge_with(*others[1:])
File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 331, in merge_with
self._format_and_raise(key=None, value=None, cause=e)
File "/usr/local/lib/python3.10/dist-packages/omegaconf/base.py", line 95, in _format_and_raise
format_and_raise(
File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 629, in format_and_raise
_raise(ex, cause)
File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 610, in _raise
raise ex # set end OC_CAUSE=1 for full backtrace
File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 329, in merge_with
self._merge_with(*others)
File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 347, in _merge_with
BaseContainer._map_merge(self, other)
File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 314, in _map_merge
dest[key] = src._get_node(key)
File "/usr/local/lib/python3.10/dist-packages/omegaconf/dictconfig.py", line 258, in setitem
self._format_and_raise(
File "/usr/local/lib/python3.10/dist-packages/omegaconf/base.py", line 95, in _format_and_raise
format_and_raise(
File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 629, in format_and_raise
_raise(ex, cause)
File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 610, in _raise
raise ex # set end OC_CAUSE=1 for full backtrace
omegaconf.errors.ConfigKeyError: Key 'defaults' not in 'FairseqConfig'
full_key: defaults
reference_type=Optional[FairseqConfig]
object_type=FairseqConfig

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/content/auto_avsr/main.py", line 74, in
main()
File "/usr/local/lib/python3.10/dist-packages/hydra/main.py", line 32, in decorated_main
_run_hydra(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 346, in _run_hydra
run_and_report(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 201, in run_and_report
raise ex
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 198, in run_and_report
return func()
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 347, in
lambda: hydra.run(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py", line 100, in run
cfg = self.compose_config(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py", line 507, in compose_config
cfg = self.config_loader.load_configuration(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 151, in load_configuration
return self._load_configuration(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 256, in _load_configuration
cfg = self._merge_defaults_into_config(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 805, in _merge_defaults_into_config
hydra_cfg = merge_defaults_list_into_config(hydra_cfg, user_list)
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 777, in merge_defaults_list_into_config
merged_cfg = self._merge_config(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 724, in _merge_config
raise ConfigCompositionException(
hydra.errors.ConfigCompositionException: Error merging data/dataset=cstm

I am running this on colab due to issues installing fairseq editable locally.

Thank you in advance!

[aprzez]

To add on, I am seeing the same issue running this in a conda environment and have not been able to solve it yet.

[mpc001]

Hi @aprzez, If you move the + in front of cstm, will it work? For instance,

!python main.py exp_dir=exp \
exp_name=trainaudio \
data.modality=audio \
ckpt_path='content/drive/MyDrive/LRS3_A_WER1.0/model.pth' \
data/dataset=cstm \ trainer.num_nodes=1

If not, it would be great if you could provide the package versions of hydra and fairseq. I will try to reproduce the error you have.

[aprzez]

Hi @mpc001, I added the plus because when I run

python main.py exp_dir=exp
exp_name=LRS3_A_WER1
data.modality=audio
ckpt_path='/crex/proj/uppmax2023-2-2/agnie/auto_avsr/models/LRS3_A_WER1.0/model.pth'
data/dataset=cstm
trainer.num_nodes=1 > nchlt_LRS3_A_WER1.txt 2>&1

I get:
hydra.errors.ConfigCompositionException: Could not override 'data/dataset'. No match in the defaults list.
To append to your default list use +data/dataset=cstm

Just now trying to run with "+" again:
hydra.errors.ConfigCompositionException: In 'data/dataset/cstm': ConfigKeyError raised while composing config:
Key 'data' not in 'FairseqConfig'
full_key: data
object_type=FairseqConfig

In both cases I renamed the "config" file to to "cf".

Package versions in my conda environment:
fairseq 0.12.3
hydra-core 1.3.0

Thank you!

[mpc001]

It appears that the main hydra configuration file is conflicting with the configuration file for fairseq (similar to #3). Can you please verify the version of omegaconf as well? I have included a list of my packages below. Also make sure that fairseq folder is placed at the root directory of this repository.

omegaconf                 2.3.0                    pypi_0    pypi
hydra-core                1.3.0                    pypi_0    pypi
fairseq                   0.12.2                   pypi_0    pypi

[aprzez]

Hi, confirming the packages I have:

omegaconf 2.3.0 pyhd8ed1ab_0 conda-forge
fairseq 0.12.3 py38h668f88f_3 conda-forge
hydra-core 1.3.0 pypi_0 pypi

I when installing fairseq editable I was getting this error:
ERROR: Failed building wheel for fairseq
Failed to build fairseq
ERROR: Could not build wheels for fairseq, which is required to install pyproject.toml-based projects

And the only way I managed to install it was by running:
conda install -c conda-forge fairseq=0.12

Not sure if that makes a difference. The fairseq folder is placed inside auto_asvr.

Thank you for looking into this.

[zyjcsf]

Hi, confirming the packages I have:

omegaconf 2.3.0 pyhd8ed1ab_0 conda-forge fairseq 0.12.3 py38h668f88f_3 conda-forge hydra-core 1.3.0 pypi_0 pypi

I when installing fairseq editable I was getting this error: ERROR: Failed building wheel for fairseq Failed to build fairseq ERROR: Could not build wheels for fairseq, which is required to install pyproject.toml-based projects

And the only way I managed to install it was by running: conda install -c conda-forge fairseq=0.12

Not sure if that makes a difference. The fairseq folder is placed inside auto_asvr.

Thank you for looking into this.

I also happen this problem. Do you have solved it ?

[SUNDULE]

Hi, confirming the packages I have:
omegaconf 2.3.0 pyhd8ed1ab_0 conda-forge fairseq 0.12.3 py38h668f88f_3 conda-forge hydra-core 1.3.0 pypi_0 pypi
I when installing fairseq editable I was getting this error: ERROR: Failed building wheel for fairseq Failed to build fairseq ERROR: Could not build wheels for fairseq, which is required to install pyproject.toml-based projects
And the only way I managed to install it was by running: conda install -c conda-forge fairseq=0.12
Not sure if that makes a difference. The fairseq folder is placed inside auto_asvr.
Thank you for looking into this.

I also happen this problem. Do you have solved it ?

Install the pip==23.3.1 version :)