Setting any value in "Hardware Acceleration" causes conversion to fail immediately

[galtalmor]

Setting any value (other than "none") in "Hardware Acceleration (-hwaccel)" causes conversion to fail immediately on start with the standard message "Video processing failed for: ...".

Turning on logs on any level doesn't show anything.

Tried all the values from ffmpeg -hwaccels output.

ffmpeg version 2022-07-31-git-1368b5a725-essentials_build-www.gyan.dev Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 12.1.0 (Rev2, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-zlib --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-sdl2 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libgme --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libtheora --enable-libvo-amrwbenc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-librubberband
  libavutil      57. 30.100 / 57. 30.100
  libavcodec     59. 40.100 / 59. 40.100
  libavformat    59. 29.100 / 59. 29.100
  libavdevice    59.  8.101 / 59.  8.101
  libavfilter     8. 46.101 /  8. 46.101
  libswscale      6.  8.101 /  6.  8.101
  libswresample   4.  8.100 /  4.  8.100
  libpostproc    56.  7.100 / 56.  7.100
Hardware acceleration methods:
cuda
dxva2
qsv
d3d11va

Video2X version 6.0.0 on Windows 11.

[k4yt3x]

That's somewhat expected right now. Right now Video2X only initializes and uses the hardware context, but for it to work fully, we'll need to also upload frames to / download frames from hardware when needed, and that's gonna be a complex task that'll be implemented in the future.

If you read the tooltip of the hwaccel field, it says:

Currently not very stable, especially on Linux.

It's for the same reason. I'll leave this ticket open to track the support for proper hwaccel.

[k4yt3x]

Oh also, I should add that hwaccel is only for decoding and encoding the video. Normally speaking the bottleneck would be with the upscaling part, which will always use hardware through Vulkan regardless of FFmpeg hwaccel settings. This is why I didn't put it on highest priority.

For just H.264+MP4, slow preset, CRF 20 (Video2X default settings), it won't make any difference. If you want to encode using some advanced hardware encoder, hwaccel will help and it will be addressed eventually.

[galtalmor]

Got it. Thanks for the details!
I really appreciate the work you put into this.

Another question, from your experience, what would yield the best (and fastest) results for live-action old TV series ripped from a DVD in MKV format 480i picture?

[k4yt3x]

If you're talking about live action (as in not animation), then I would probably run a deinterlace first with FFmpeg to get rid of the artifacts, then RealESRGAN with the realesrgan-plus model. The other models are all optimized for anime and cartoons. You can also take a short slice of your footage and try the different models, I've seen Anime4K doing better than RealSR on some live action footage in the past.

[galtalmor]

Sounds good. Thank you again! ❤️

[k4yt3x]

I should amend this statement:

Oh also, I should add that hwaccel is only for decoding and encoding the video.

It's actually mostly for decoding the video. For encoding, I'm pretty sure something like hevc_nvenc will use the hardware themselves. I stand to be corrected.

[k4yt3x]

I think I'll close this issue for now since it's not something I plan to work on soon. hwaccel should mostly affect decoding which isn't really a bottleneck for the program's speed right now. Decoding is usually way faster than processing and encoding. It's something I can come back to when multi-threading, batched inferencing and other features are implemented and there's nothing more to improve on. I'll just close it for now so my todo list is clearer.

[k4yt3x]

The other thing is that this is tough to implement properly, as enabling hwaccel requires uploading/downloading frames to/from the GPU whenever appropriate. This adds a lot of complexity, and seems a bit difficult to implement properly for all hw codecs. Since the speed here isn't a big problem, I'd put it on hold for now.

[galtalmor]

Sounds good. Thanks for the update.