Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The decoder returned a frame that is past the expected one #233

Open
hummuscience opened this issue Jan 13, 2025 · 6 comments
Open

The decoder returned a frame that is past the expected one #233

hummuscience opened this issue Jan 13, 2025 · 6 comments

Comments

@hummuscience
Copy link
Contributor

hummuscience commented Jan 13, 2025

I am getting the following error:

140297388422912 Exception in thread: The decoder returned a frame that is past the expected one. The most likely cause is variable frame rate video. Filename: <path>/sub-463_strain-B6_2024-12-09T17_45_03.mp4

The weird thing is that it only happens occasionally. And I am unable to reproduce it. I ran multiple models with the --multirun option and it only affected one out of the 16 jobs (so far, the rest is still running).

This one was running a temporal loss with a heatmap model type. The training passed the first epoch and then the error showed up by the second epoch. Another model with a temporal loss and heatmap_mhcrn did not get the error.

I tired to look for the error. Here it is mentioned as something that is still under investigation: NVIDIA/DALI#5665

I checked the encoding of the file, it is h264. So its not that. The vfr check is disabled.

Any idea where I can check?

(I removed the path)

@themattinthehatt
Copy link
Collaborator

Are your videos variable frame rate? You can check using ffprobe -v quiet -print_format json -show_streams input and check the "avg_frame_rate" field; if the denominator is something other than 1 the video has variable frame rate.

We have disabled the vfr check by default, since previously we did not find any issues doing this with variable frame rate videos - however, it seems that might not generally be the case.

One potential reason for why it only happens occasionally is the way we handle the unlabeled data - one "epoch" is one pass through the labeled data; once we hit the end of that epoch we randomly re-initialize both the labeled dataloader and the unlabeled dataloader. So it is not guaranteed that you will ever make a full pass through the unlabeled data, and therefore not guaranteed that you will encounter the last batch of the unlabeled video. Even though we set the seed for the unlabeled dataloader we have not actually checked in a while if this leads to the same sequence of unlabeled batches.

@hummuscience
Copy link
Contributor Author

Here is the output for that particular video:

{
    "streams": [
        {
            "index": 0,
            "codec_name": "h264",
            "codec_long_name": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10",
            "profile": "Main",
            "codec_type": "video",
            "codec_tag_string": "avc1",
            "codec_tag": "0x31637661",
            "width": 1920,
            "height": 1080,
            "coded_width": 1920,
            "coded_height": 1080,
            "closed_captions": 0,
            "film_grain": 0,
            "has_b_frames": 2,
            "sample_aspect_ratio": "1:1",
            "display_aspect_ratio": "16:9",
            "pix_fmt": "yuv420p",
            "level": 40,
            "chroma_location": "left",
            "field_order": "progressive",
            "refs": 1,
            "is_avc": "true",
            "nal_length_size": "4",
            "id": "0x1",
            "r_frame_rate": "30/1",
            "avg_frame_rate": "30/1",
            "time_base": "1/16000",
            "start_pts": 1600,
            "start_time": "0.100000",
            "duration_ts": 14399984,
            "duration": "899.999000",
            "bit_rate": "692548",
            "bits_per_raw_sample": "8",
            "nb_frames": "27000",
            "extradata_size": 50,
            "disposition": {
                "default": 1,
                "dub": 0,
                "original": 0,
                "comment": 0,
                "lyrics": 0,
                "karaoke": 0,
                "forced": 0,
                "hearing_impaired": 0,
                "visual_impaired": 0,
                "clean_effects": 0,
                "attached_pic": 0,
                "timed_thumbnails": 0,
                "non_diegetic": 0,
                "captions": 0,
                "descriptions": 0,
                "metadata": 0,
                "dependent": 0,
                "still_image": 0
            },
            "tags": {
                "language": "und",
                "handler_name": "VideoHandler",
                "vendor_id": "[0][0][0][0]",
                "encoder": "Lavc61.3.100 h264_nvenc"
            }
        }
    ]
}

It seems that in this case the frame rate is fixed.

@themattinthehatt
Copy link
Collaborator

something to note in your output is the "duration" field - "nb_frames" / "r_frame_rate" = 900 seconds, but your "duration" field is "899.999000" instead of an even "900". I checked a few of my videos and "nb_frames" / "r_frame_rate" always exactly equals "duration". I'm not really sure if this is the issue, or how to interpret this; 26999 / 30 = 899.9666666666667, which isn't exactly equal to "duration" either, so it doesn't appear as if there's a single dropped frame. Do you see this discrepancy with all of your videos?

@hummuscience
Copy link
Contributor Author

Interestingly, on some recording days, all the videos have a duration of 899.999000 while on other dats its a round 900 seconds.

I am using ffmpeg segments to create 900 second segments when recording.

@themattinthehatt
Copy link
Collaborator

I'd be curious if that's the issue; if you only retain the 900 second videos for training do you run into the issue? I realize it's a hard one to test given the randomness of the video access :/

@hummuscience
Copy link
Contributor Author

Currently I am just re-running failed attempts. But I will add more labels from another video and retry

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants