In Vapoursynth, it is convenient to download and use plugins to change the frame rate of videos. However, it seems that using FFmpeg has more limitations (or maybe I haven't found the right method). Anime4K is a video enhancement algorithm suitable for anime and similar videos. In MPV, their plugins can be imported and used for real-time frame rate conversion.
In FFmpeg, how can I use the algorithms in glsl files to change the video frame rate, instead of using FFmpeg's built-in algorithms?
In FFmpeg, i can use libplacebo filters can also be used to apply GLSL shaders. In my tests, using command like:
libplacebo=custom_shader_path=Anime4K_Upscale_CNN_x2_VL.glsl
this leads to noticeable GPU usage, but the output video resolution remains the same as the original.
I can pass in the w and h parameters when calling the libplacebo filter:
libplacebo=w=iw*2:h=ih*2:custom_shader_path=shaders/Anime4K_Upscale_CNN_x2_VL.glsl
This achieves the purpose of super resolution, but is it essentially using anime4k for upscaling behind the processing? Is it using the upscaling algorithm in libplacebo first, before applying anime4k? How can I achieve my original goal that only use anime4k to upscale? here's my full command (in python):
[
'ffmpeg -hide_banner',
'-hwaccel', 'cuda',
'-i', '"{input_ab_path}"',
'-filter_complex',
'"[0:v]libplacebo=custom_shader_path=libplacebo=custom_shader_path=shaders/Anime4K_Upscale_CNN_x2_VL.glsl,subtitles="{sub_file}":si=0[out]"',
'-map', '0:a',
'-map', '"[out]"',
'-c:v', 'libsvtav1',
'-svtav1-params', 'scm=2:scd=1:enable-overlays=1:enable-tf=0:tune=0:preset=7:crf=18',
'-c:a', 'libvorbis',
'-qscale:a', '10',
'-pix_fmt', 'yuv420p10le',
'-y', '"{output_ab_path}"'
]
Let me think through and answer this question myself: Since I don't have the energy to go through the source code of libplacebo, but after I did a pixel by pixel comparison of the same frame with different parameter and filter settings, I found:
When passing in w, h, custom_shader_path parameters to libplacebo, it doesn't first scale then apply the shader from the specified path to process the image. Instead it passes w, h into the shader, and after the shader finishes processing, it then compares the final width and height, and if different, uses a filter like spline36 to scale up or down.
So if we want to use Anime4K's upscaling filter, we just need to use:
libplacebo=w=iw*2:h=ih*2:custom_shader_path=shaders/Anime4K_Upscale_CNN_x2_VL.glsl
to successfully call the shader's upscaling algorithm.
However if the command is like this:
libplacebo=custom_shader_path=shaders/Anime4K_Upscale_CNN_x2_VL.glsl
it will still use anime4k to upscale, but since w and h (target resolution and also ,the default resolution) are not specified, the resolution output by the shader is larger, and libplacebo will use its built-in filters (default is spline36) to downsample the shader output to default resolution.
If my statement above is correct, and I want to use a custom shader for upsampling and downsampling, then we would need approximately these types of instructions:
libplacebo=w=iw*2:h=ih*2:custom_shader_path=shaders/Anime4K_Upscale_CNN_x2_VL.glsl, libplacebo=w=iw/2h:h=ih/2:custom_shader_path=shaders/SSimDownscaler.glsl
Attention: Since these are based solely on my observations and guesses, they are not guaranteed to be correct. I hope someone can point out any mistakes I made and provide an answer that is more low-level and detailed, or give a better approach (of course including vapoursynth, though there's not much to say about that)