I encoded a fMP4-Video (HEVC) in Swift using VideoToolbox and CoreMedia. The resulting fragmented MP4 is only playing in VLC.
The library I am using to write the fMP4 is an HEVC-adapted version of this GitHub-Project: https://github.com/krad/morsel
Process of encoding and writing:
SampleBuffer
from the camera (VTCompressionSession
, as described in WWDC 2014 Session 513)CoreMedia
-Functions (e.g. CMVideoFormatDescriptionGetHEVCParameterSetAtIndex
, CMVideoFormatDescriptionGetDimensions
) to get the encoded streams metadata. I am writing the content of CMFormatDescriptionGetExtension(description, extensionKey: "SampleDescriptionExtensionAtoms" as CFString)["hvcC"]
directly to the hvcC box. This box is obviously written correctly, when I alter one bit of the box of the mp4, QuickTime throws an error.The result file is playable in VLC, when I just let the playback "run" without doing anything. When I scroll the time in VLC, VLC crashes and the playback stops.
It is also sort of "playable" in Safari and QuickTime (no error message shown, the playback window opens and the correct length of the file is shown, I can even change to playback time / play/pause, but there is no video shown. The window remains empty. And that is the problem I have. I need to get an fMP4 running in Safari and QuickTime.
The file is not playing at all in ffplay (same problem when converting with ffmpeg). The line where the playback time is shown remains nan M-V: nan fd= 0 aq= 0KB vq= 0KB sq= 0B f=0/0
, there are no error messages. The playback just does not start.
When playing the file with ffplay, this is the output:
ffplay version 4.3 Copyright (c) 2003-2020 the FFmpeg developers
built with Apple clang version 11.0.3 (clang-1103.0.32.62)
configuration: --prefix=/usr/local/Cellar/ffmpeg/4.3_2 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
libavutil 56. 51.100 / 56. 51.100
libavcodec 58. 91.100 / 58. 91.100
libavformat 58. 45.100 / 58. 45.100
libavdevice 58. 10.100 / 58. 10.100
libavfilter 7. 85.100 / 7. 85.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 7.100 / 5. 7.100
libswresample 3. 7.100 / 3. 7.100
libpostproc 55. 7.100 / 55. 7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mp4':
Metadata:
major_brand : mp42
minor_version : 1
compatible_brands: mp41mp42isomhlsf
creation_time : 2020-08-03T20:41:08.000000Z
Duration: N/A, bitrate: N/A
Stream #0:0(und): Video: hevc (Main) (hvc1 / 0x31637668), yuv420p(tv), 1920x1080, SAR 1:1 DAR 16:9, 1000000000.00 tbr, 1000000000.00 tbn, 1000000000.00 tbc (default)
Metadata:
creation_time : 2020-08-03T20:41:08.000000Z
handler_name : video
nan M-V: nan fd= 0 aq= 0KB vq= 0KB sq= 0B f=0/0
My file has this structure:
ftyp
moov
mvhd
trak
tkhd
mdia
mdhd
hdlr
minf
vmhd
dinf
dref
stbl
stsd
stts
stsc
stsz
stco
mvex
trex
moof-(1)
mdat
moof-(2)
mdat
...
The result file I want to create should be like the fMP4 served within this HLS playlist: (Example: https://developer.apple.com/streaming/examples/advanced-stream-hevc.html, Playlist: https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_adv_example_hevc/master.m3u8) This file has the following structure:
ftyp
moov
mvhd
trak
tkhd
mdia
mdhd
hdlr
minf
vmhd
dinf
dref
stbl
stsd
stts
stsc
stsz
stco
trak (same structure as above)
mvex
trex
moof-(1)
mdat
moof-(2)
mdat
...
This file from Apple plays perfectly in ffmpeg and QuickTime/Safari. Except for the second trak
atom, it has the same structure.
The "good" file is: https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_adv_example_hevc/v14/main.mp4 (It is intended for use in a HLS Playlist I've downloaded it with curl)
The "bad" file is: https://www.transfernow.net/ddl/fmp4_bad (link should work, tested it just now :) ). The video shows a slowly rotating sheet of paper.
Any suggestions what the problem with my file is? Thanks in advance!
The duration in moof->traf->trun->entries->duration is wrong.
So is the moof->traf->tfhd->default_sample_duration.
It looks more like a timestamp but it should be the frame duration.
You only have one sample per fragment which is legal but pretty wasteful.