Averaging model parameters

[patrik-lambert]

Hi. I have seen the possibility (for example in generate-lines.lua) to decode with several models, whose softmax and attention scores are averaged to generate predictions. This has the inconvenient that decoding time increases nearly linearly with the number of models.

Since I need to keep decoding time low, I would like to build a new model by averaging parameters of several models. Are there methods/pieces of code in fairseq that I could use do to this?

Thanks!

[michaelauli]

You can load the checkpoints of the models you would like to average and save the mean of all the parameters as a new checkpoint.

[patrik-lambert]

Thank you for the tips! I have written the attached script. It seems to do the job. Does it seem correct to you? Does it include all the parameters?
average_models.lua.txt