-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The decoder returned a frame that is past the expected one #233
Comments
Are your videos variable frame rate? You can check using We have disabled the vfr check by default, since previously we did not find any issues doing this with variable frame rate videos - however, it seems that might not generally be the case. One potential reason for why it only happens occasionally is the way we handle the unlabeled data - one "epoch" is one pass through the labeled data; once we hit the end of that epoch we randomly re-initialize both the labeled dataloader and the unlabeled dataloader. So it is not guaranteed that you will ever make a full pass through the unlabeled data, and therefore not guaranteed that you will encounter the last batch of the unlabeled video. Even though we set the seed for the unlabeled dataloader we have not actually checked in a while if this leads to the same sequence of unlabeled batches. |
Here is the output for that particular video:
It seems that in this case the frame rate is fixed. |
something to note in your output is the "duration" field - "nb_frames" / "r_frame_rate" = 900 seconds, but your "duration" field is "899.999000" instead of an even "900". I checked a few of my videos and "nb_frames" / "r_frame_rate" always exactly equals "duration". I'm not really sure if this is the issue, or how to interpret this; 26999 / 30 = 899.9666666666667, which isn't exactly equal to "duration" either, so it doesn't appear as if there's a single dropped frame. Do you see this discrepancy with all of your videos? |
Interestingly, on some recording days, all the videos have a duration of 899.999000 while on other dats its a round 900 seconds. I am using ffmpeg segments to create 900 second segments when recording. |
I'd be curious if that's the issue; if you only retain the 900 second videos for training do you run into the issue? I realize it's a hard one to test given the randomness of the video access :/ |
Currently I am just re-running failed attempts. But I will add more labels from another video and retry |
I am getting the following error:
140297388422912 Exception in thread: The decoder returned a frame that is past the expected one. The most likely cause is variable frame rate video. Filename: <path>/sub-463_strain-B6_2024-12-09T17_45_03.mp4
The weird thing is that it only happens occasionally. And I am unable to reproduce it. I ran multiple models with the --multirun option and it only affected one out of the 16 jobs (so far, the rest is still running).
This one was running a temporal loss with a heatmap model type. The training passed the first epoch and then the error showed up by the second epoch. Another model with a temporal loss and heatmap_mhcrn did not get the error.
I tired to look for the error. Here it is mentioned as something that is still under investigation: NVIDIA/DALI#5665
I checked the encoding of the file, it is h264. So its not that. The vfr check is disabled.
Any idea where I can check?
(I removed the path)
The text was updated successfully, but these errors were encountered: