Bad over-fitting

[SajjadAemmi]

i used vsr_trlrs3_23h_base.pth as pre-trained ckeckpoint and trained VSR model on a very small persian dataset (10 video with 10 seconds duration of each video)

My goal was over-fitting on the dataset to get correct prediction on the same dataset.

But the over-fitted model only learnd 1 text, and produced exactly the same text for every input videos in the inference time.

what is the problem?

[mpc001]

I assume that the 23-hour pre-trained checkpoint is not a good one and it cannot generalize well to other datasets. Have you tried using the checkpoint pre-trained on the entire dataset for fine-tuning?

[SajjadAemmi]

I assume that the 23-hour pre-trained checkpoint is not a good one and it cannot generalize well to other datasets. Have you tried using the checkpoint pre-trained on the entire dataset for fine-tuning?

Thank you for your awesome project and your reply. I was using vsr_trlrs2lrs3vox2avsp_base.pth checkpoint.
I found the solution: I have to freeze encoder to get a little better results.