Difference between the rna004_130bps_sup@v5.0.0 and rna004_130bps_sup@v5.1.0 ? #1208
Labels
performance
Issues related to basecalling performance
Comments
|
Hi @VBHarrisN, there's no architectural difference between the rna004_130bps_sup v5.0.0 and 5.1.0 models and as such they should run at the same speed. The v5.1.0 model gained improvements from better training. The models are listed on both the README and the Dorado-docs Model List To help us resolve the unexpected performance regression can you share the following information requested in the issue template:
Kind regards, |
|
Run environment: Operating system: WSL Hardware (CPUs, Memory, GPUs): 13th Gen Intel(R) I9-13900K, 64GB, NVIDIA RTX A5000 V5.1.0 verbose output: v5.0.0 Verbose Output: Only difference I notice is that the CUDA batch size is much larger in the v5.1.0 model output, which seems like it would be faster.... I also can't help but notice that the modal tail length for the poly-a estimation is very different between the two models. Let me know if I can provide any more information! |
|
Hi @VBHarrisN, Thank you for the detailed logs - we'll look into this and get back to you once we've investigated. Edit: Could you run this side-by-side benchmark:
Best regards, |
|
Here is V5.0.0 Here is v5.1.0 |
|
Those numbers suggest that it is the polyA calculation that is causing the difference in speeds, rather than the models. The amount of time polyA estimation takes is going to be a little data dependent, but a 60% slowdown seems excessive. Dorado 0.9.0 should already be refusing to estimate reads that have selected an implausibly large region to search for the polyA signal, but it's possible this needs tightening up. Are you able to isolate a subset of reads that replicate this? It looks like some of the data in |
|
Unfortunately we are unable to share that data as it is sensitive information. However, I can say that even with the poly-a-estimation flag turned off, the v5.1.0 model still take about 60% longer to run when run on the full dataset. I can also validate that this happens on other datasets, not just this most recent one. Short of sharing the data, we are happy to assist in any capacity with resolving this issue! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Just curious, what is the difference between the rna004_130bps_sup@v5.0.0 and rna004_130bps_sup@v5.1.0 models? We noticed that the new v5.1.0 model takes approximately 4x the time that the v5.0.0 model does, and were curious about the difference between the two models. We are using a pretty standard RTX A5000 GPU. The v5.1.0 model is not noted in the model list on the GitHub nor in the changelog from my brief glance at them. Any clarification would be appreciated!
The text was updated successfully, but these errors were encountered: