Significant Gap in Reproduced CoTracker3 Results with baseline_offline.pth

[sinkers-lan]

Dear authors,

Thank you for sharing your code and pre-trained checkpoints! While reproducing the results reported in your paper, I noticed a significant discrepancy in the performance of CoTracker3 when using the provided checkpoint baseline_offline.pth. Below is a comparison specifically for the baseline checkpoint, as well as the results for other settings.

---

Results Comparison: baseline_offline.pth

	Validation Dataset	Protocol	Model	Mode	AJ	Delta	OA
Paper: CoTracker3	DAVIS	First	Base	Offline	63.3	76.2	88.0
Reproduced: CoTracker3	DAVIS	First	Base	Offline	51.2	71.1	73.0

---

Results Comparison: Other Settings

	Validation Dataset	Protocol	Model	Mode	AJ	Delta	OA
Paper: CoTracker3	DAVIS	First	Scaled	Offline	64.4	76.9	91.2
Reproduced: CoTracker3	DAVIS	First	Scaled	Offline	64.0	76.4	90.9
Paper: CoTracker3	DAVIS	First	Base	Online	64.5	76.7	89.7
Reproduced: CoTracker3	DAVIS	First	Base	Online	62.8	76.4	88.4
Paper: CoTracker3	DAVIS	First	Scaled	Online	63.8	76.3	90.2
Reproduced: CoTracker3	DAVIS	First	Scaled	Online	64.4	77.1	90.9

---

Questions and Request

1. Could you confirm whether the baseline_offline.pth checkpoint provided in the repository is the same as the one used in the paper?
2. Is it possible that an incorrect or incomplete version of the checkpoint was uploaded? If so, could you kindly update the checkpoint to the correct one?
3. I used the default configuration from the repository with no modifications.

Looking forward to your response. Thank you for your time and help!

[Jafonso-sudo]

To add a question that is related to reproducibility. The results that I am getting don't seem deterministic, as I am getting different results in different runs. Is there any source of non-determinism in your pipeline? I have tried setting a seed for PyTorch manually, but it does not seem to fix it.

[sinkers-lan]

Follow-up on issue regarding baseline_offline.pth

Hi @nikitakaraevv,

I hope you're doing well. I recently opened this issue regarding the baseline_offline.pth file, as I was unable to reproduce the results reported in the paper using the provided checkpoint. I wanted to check if you could kindly verify the checkpoint and update it if necessary.

Please let me know if you need any additional details or logs from my side to assist in the process.

Thank you for your time and effort in maintaining this work!