I am trying to use filter_complex from ffmpeg to stich several video input together into one output.
I use v4l2loopback to create a virtual video device:
sudo modprobe v4l2loopback video_nr=90
In this example, i first try with only one camera:
v4l2-ctl --device=/dev/video0 --info gives the following output:
Driver Info (not using libv4l2):
Driver name : uvcvideo
Card type : HD USB Camera
Bus info : usb-0000:00:14.0-13.2
Driver version: 4.10.17
Capabilities : 0x84200001
Video Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04200001
Video Capture
Streaming
Extended Pix Format
Priority: 2
Video input : 0 (Camera 1: ok)
Format Video Capture:
Width/Height : 640/480
Pixel Format : 'YUYV'
Field : None
Bytes per Line : 1280
Size Image : 614400
Colorspace : sRGB
Transfer Function : Default
YCbCr Encoding : Default
Quantization : Default
Flags :
Crop Capability Video Capture:
Bounds : Left 0, Top 0, Width 640, Height 480
Default : Left 0, Top 0, Width 640, Height 480
Pixel Aspect: 1/1
Selection: crop_default, Left 0, Top 0, Width 640, Height 480
Selection: crop_bounds, Left 0, Top 0, Width 640, Height 480
Streaming Parameters Video Capture:
Capabilities : timeperframe
Frames per second: 30.000 (30/1)
Read buffers : 0
brightness (int) : min=0 max=15 step=1 default=8 value=8
contrast (int) : min=0 max=15 step=1 default=8 value=8
saturation (int) : min=0 max=15 step=1 default=7 value=7
hue (int) : min=-10 max=10 step=1 default=0value=0
white_balance_temperature_auto (bool) : default=1 value=1
gamma (int) : min=1 max=10 step=1 default=7 value=7
gain (int) : min=0 max=0 step=0 default=0 value=0
power_line_frequency (menu) : min=0 max=2 default=2 value=2
white_balance_temperature (int) : min=2800 max=6500 step=1 default=2800 value=2800 flags=inactive
sharpness (int) : min=0 max=15 step=1 default=6 value=6
backlight_compensation (int) : min=0 max=1 step=1 default=0 value=0
exposure_auto (menu) : min=0 max=3 default=3 value=3
exposure_absolute (int) : min=4 max=5000 step=1 default=625 value=625 flags=inactive
focus_absolute (int) : min=0 max=21 step=1 default=16 value=16 flags=inactive
focus_auto (bool) : default=1 value=1
Then i use this command to ouput /dev/video0 to /dev/video90 on the left side with a specific resolution:
ffmpeg -loglevel verbose -y \
-f v4l2 -thread_queue_size 512 -i /dev/video0 \
-filter_complex "nullsrc=size=960x240 [base]; \
[0:v] format=pix_fmts=yuva420p, setpts=PTS-STARTPTS, scale=320x240 [left]; \
[base][left] overlay=shortest=1" \
-f v4l2 /dev/video90
Which results in the following error:
[v4l2 @ 0x55fa67d02f60] Unknown V4L2 pixel format equivalent for yuva420p
Could not write header for output file #0 (incorrect codec parameters ?): Invalid argument
Error initializing output stream 0:0 --
Conversion failed!
Full logs:
ffmpeg version 3.4.4-1~16.04.york0 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
configuration: --prefix=/usr --extra-version='1~16.04.york0' --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
WARNING: library configuration mismatch
avcodec configuration: --prefix=/usr --extra-version='1~16.04.york0'--toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared --enable-version3 --disable-doc --disable-programs --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libtesseract --enable-libvo_amrwbenc
libavutil 55. 78.100 / 55. 78.100
libavcodec 57.107.100 / 57.107.100
libavformat 57. 83.100 / 57. 83.100
libavdevice 57. 10.100 / 57. 10.100
libavfilter 6.107.100 / 6.107.100
libavresample 3. 7. 0 / 3. 7. 0
libswscale 4. 8.100 / 4. 8.100
libswresample 2. 9.100 / 2. 9.100
libpostproc 54. 7.100 / 54. 7.100
[video4linux2,v4l2 @ 0x55fa67cfca00] fd:5 capabilities:84200001
[video4linux2,v4l2 @ 0x55fa67cfca00] Querying the device for the current frame size
[video4linux2,v4l2 @ 0x55fa67cfca00] Setting frame size to 640x480
Input #0, video4linux2,v4l2, from '/dev/video0':
Duration: N/A, start: 273229.829138, bitrate: 147456 kb/s
Stream #0:0: Video: rawvideo, 1 reference frame (YUY2 / 0x32595559), yuyv422, 640x480, 147456 kb/s, 30 fps, 30 tbr, 1000k tbn, 1000k tbc
[Parsed_nullsrc_0 @ 0x55fa67d00680] size:960x240 rate:25/1 duration:-1.000000 sar:1/1
[Parsed_scale_3 @ 0x55fa67d02240] w:320 h:240 flags:'bilinear' interl:0
Stream mapping:
Stream #0:0 (rawvideo) -> format
overlay -> Stream #0:0 (rawvideo)
Press [q] to stop, [?] for help
[Parsed_nullsrc_0 @ 0x55fa67cfc760] size:960x240 rate:25/1 duration:-1.000000 sar:1/1
[Parsed_scale_3 @ 0x55fa67d055c0] w:320 h:240 flags:'bilinear' interl:0
[graph 0 input from stream 0:0 @ 0x55fa67d06a40] w:640 h:480 pixfmt:yuyv422 tb:1/1000000 fr:30/1 sar:0/1 sws_param:flags=2
[auto_scaler_0 @ 0x55fa67d05d80] w:iw h:ih flags:'bilinear' interl:0
[Parsed_format_1 @ 0x55fa67d04360] auto-inserting filter 'auto_scaler_0' between the filter 'graph 0 input from stream 0:0' and the filter 'Parsed_format_1'
[auto_scaler_0 @ 0x55fa67d05d80] w:640 h:480 fmt:yuyv422 sar:0/1 -> w:640 h:480 fmt:yuva420p sar:0/1 flags:0x2
[Parsed_format_1 @ 0x55fa67d04360] TB:0.000001 FRAME_RATE:30.000000 SAMPLE_RATE:nan
[Parsed_scale_3 @ 0x55fa67d055c0] w:640 h:480 fmt:yuva420p sar:0/1 -> w:320 h:240 fmt:yuva420p sar:0/1 flags:0x2
[Parsed_overlay_4 @ 0x55fa67d05e80] main w:960 h:240 fmt:yuva420p overlay w:320 h:240 fmt:yuva420p
[Parsed_overlay_4 @ 0x55fa67d05e80] [framesync @ 0x55fa67d05fa8] Selected 1/1000000 time base
[Parsed_overlay_4 @ 0x55fa67d05e80] [framesync @ 0x55fa67d05fa8] Sync level 2
[v4l2 @ 0x55fa67d02f60] Unknown V4L2 pixel format equivalent for yuva420p
Could not write header for output file #0 (incorrect codec parameters ?): Invalid argument
Error initializing output stream 0:0 --
Conversion failed!
What I tried and didn't work:
-f h264 for the output, which ends in a av_interleaved_write_frame(): Invalid argument errorSince your overlay input has alpha, the overlay filter requests an alpha format from the nullsrc filter. Now, the v4l2 output device doesn't accept YUV formats with alpha and it can't auto-negotiate with the filter, unlike most encoders, hence the error shown. Force a supported format after the overlay to avoid this.
Use
ffmpeg -loglevel verbose -y \
-f v4l2 -thread_queue_size 512 -i /dev/video0 \
-filter_complex "nullsrc=size=960x240, format=yuv420p [base]; \
[0:v] format=pix_fmts=yuva420p, setpts=PTS-STARTPTS, scale=320x240 [left]; \
[base][left] overlay=shortest=1,format=yuv420p" \
-f v4l2 /dev/video90