video ffmpeg http-live-streaming m3u8 libav

FFmpeg (libav) is taking a lot of time to execute avformat_open_input over a master with few variants

I am trying to use libav to demux an HLS stream from youtube, it works so far but my problem is that the call to avformat_open_input takes A LOT of time, sometimes even 1 minute.

Doing same with Exoplayer (for example) it works without any issues.. I have the feeling of missing something big on the usage of libav for hls demuxing..

The master file that I am using looks like the following:

#EXTM3U
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-MEDIA:URI="http://localhost:35835/itag/233/mediadata.m3u8",TYPE=AUDIO,GROUP-ID="233",DEFAULT=YES,AUTOSELECT=YES,NAME="Default"
#EXT-X-MEDIA:URI="http://localhost:35835/itag/234/mediadata.m3u8",TYPE=AUDIO,GROUP-ID="234",DEFAULT=YES,AUTOSELECT=YES,NAME="Default"
#EXT-X-STREAM-INF:BANDWIDTH=1354423,CODECS="avc1.4D401F,mp4a.40.2",RESOLUTION=854x480,AUDIO="234",FRAME-RATE=30,VIDEO-RANGE=SDR,CLOSED-CAPTIONS=NONE
http://localhost:35835/itag/231/mediadata.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=318204,CODECS="avc1.4D4015,mp4a.40.5",RESOLUTION=426x240,AUDIO="233",FRAME-RATE=30,VIDEO-RANGE=SDR,CLOSED-CAPTIONS=NONE
http://localhost:35835/itag/229/mediadata.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=402807,CODECS="avc1.4D4015,mp4a.40.2",RESOLUTION=426x240,AUDIO="234",FRAME-RATE=30,VIDEO-RANGE=SDR,CLOSED-CAPTIONS=NONE
http://localhost:35835/itag/229/mediadata.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=811077,CODECS="avc1.4D401E,mp4a.40.2",RESOLUTION=640x360,AUDIO="234",FRAME-RATE=30,VIDEO-RANGE=SDR,CLOSED-CAPTIONS=NONE
http://localhost:35835/itag/230/mediadata.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=177519,CODECS="avc1.4D400C,mp4a.40.5",RESOLUTION=256x144,AUDIO="233",FRAME-RATE=30,VIDEO-RANGE=SDR,CLOSED-CAPTIONS=NONE
http://localhost:35835/itag/269/mediadata.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3851098,CODECS="avc1.4D4020,mp4a.40.2",RESOLUTION=1280x720,AUDIO="234",FRAME-RATE=60,VIDEO-RANGE=SDR,CLOSED-CAPTIONS=NONE
http://localhost:35835/itag/311/mediadata.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=6324518,CODECS="avc1.64002A,mp4a.40.2",RESOLUTION=1920x1080,AUDIO="234",FRAME-RATE=60,VIDEO-RANGE=SDR,CLOSED-CAPTIONS=NONE
http://localhost:35835/itag/312/mediadata.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=95792,CODECS="vp09.00.10.08,mp4a.40.5",RESOLUTION=256x144,AUDIO="233",FRAME-RATE=15,VIDEO-RANGE=SDR,CLOSED-CAPTIONS=NONE
http://localhost:35835/itag/602/mediadata.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=170851,CODECS="vp09.00.11.08,mp4a.40.5",RESOLUTION=256x144,AUDIO="233",FRAME-RATE=30,VIDEO-RANGE=SDR,CLOSED-CAPTIONS=NONE
http://localhost:35835/itag/603/mediadata.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=292033,CODECS="vp09.00.20.08,mp4a.40.5",RESOLUTION=426x240,AUDIO="233",FRAME-RATE=30,VIDEO-RANGE=SDR,CLOSED-CAPTIONS=NONE
http://localhost:35835/itag/604/mediadata.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=376636,CODECS="vp09.00.20.08,mp4a.40.2",RESOLUTION=426x240,AUDIO="234",FRAME-RATE=30,VIDEO-RANGE=SDR,CLOSED-CAPTIONS=NONE
http://localhost:35835/itag/604/mediadata.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=716510,CODECS="vp09.00.21.08,mp4a.40.2",RESOLUTION=640x360,AUDIO="234",FRAME-RATE=30,VIDEO-RANGE=SDR,CLOSED-CAPTIONS=NONE
http://localhost:35835/itag/605/mediadata.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1145989,CODECS="vp09.00.30.08,mp4a.40.2",RESOLUTION=854x480,AUDIO="234",FRAME-RATE=30,VIDEO-RANGE=SDR,CLOSED-CAPTIONS=NONE
http://localhost:35835/itag/606/mediadata.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3217654,CODECS="vp09.00.40.08,mp4a.40.2",RESOLUTION=1280x720,AUDIO="234",FRAME-RATE=60,VIDEO-RANGE=SDR,CLOSED-CAPTIONS=NONE
http://localhost:35835/itag/612/mediadata.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=6078208,CODECS="vp09.00.41.08,mp4a.40.2",RESOLUTION=1920x1080,AUDIO="234",FRAME-RATE=60,VIDEO-RANGE=SDR,CLOSED-CAPTIONS=NONE
http://localhost:35835/itag/617/mediadata.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=14086878,CODECS="vp09.00.50.08,mp4a.40.2",RESOLUTION=2560x1440,AUDIO="234",FRAME-RATE=60,VIDEO-RANGE=SDR,CLOSED-CAPTIONS=NONE
http://localhost:35835/itag/623/mediadata.m3u8

And to give an example of how a EXT-STREAM-INF link content looks, here is a sample:

#EXTM3U
#YT-EXT-CONDENSED-URL:BASE-URI="https://rr6---sn-gqn-h5qs.googlevideo.com/videoplayback/id/189d2f25f6103cad/itag/233/source/youtube/cpn/2TC6rJxXeKRmE8UD/expire/1696647973/ei/xXYgZczEE-idp-oPltGVuAI/ip/2a0c:5a84:f209:1800:b842:b4cf:48d7:ca1c/requiressl/yes/ratebypass/yes/goi/133/sgoap/clen%3D4352437%3Bdur%3D713.642%3Bgir%3Dyes%3Bitag%3D139%3Blmt%3D1677742885047413/hls_chunk_host/rr6---sn-gqn-h5qs.googlevideo.com/mh/M-/mm/31,29/mn/sn-gqn-h5qs,sn-h5qzen7y/ms/au,rdu/mv/m/mvi/6/pl/36/ctier/A/pfa/5/initcwndbps/1466250/hightc/yes/siu/1/vprv/1/playlist_type/DVR/txp/5532434/mt/1696626071/fvip/2/keepalive/yes/fexp/24007246/beids/24350018/sparams/expire,ei,ip,id,itag,source,requiressl,ratebypass,goi,sgoap,ctier,pfa,hightc,siu,vprv,playlist_type/sig/AGM4YrMwRQIhAIxbCFktH4mxamWq1_maS3YbKeNCpYaOd8JCh0fMaCblAiBFrvxzoCzIjaLpMkxvv6Aw0dSNpYDl_KI-P1aNw7McyQ%3D%3D/lsparams/hls_chunk_host,mh,mm,mn,ms,mv,mvi,pl,initcwndbps/lsig/AK1ks_kwRQIhAOKU632ETDnLoNcjq8c7oB8fd2bxioPtYANy3Bo7LKrvAiBfywDIDKfA4RY7c0CzqpRWOso0X0Gkbx6aLox0EG11KQ%3D%3D/playlist/index.m3u8",PARAMS="begin,len,goap,gosq",PREFIX="s/"
#EXT-X-VERSION:3
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-TARGETDURATION:7
#EXTINF:3.1,
https://rr6---sn-gqn-h5qs.googlevideo.com/videoplayback/id/189d2f25f6103cad/itag/233/source/youtube/cpn/2TC6rJxXeKRmE8UD/expire/1696647973/ei/xXYgZczEE-idp-oPltGVuAI/ip/2a0c:5a84:f209:1800:b842:b4cf:48d7:ca1c/requiressl/yes/ratebypass/yes/goi/133/sgoap/clen%3D4352437%3Bdur%3D713.642%3Bgir%3Dyes%3Bitag%3D139%3Blmt%3D1677742885047413/hls_chunk_host/rr6---sn-gqn-h5qs.googlevideo.com/mh/M-/mm/31,29/mn/sn-gqn-h5qs,sn-h5qzen7y/ms/au,rdu/mv/m/mvi/6/pl/36/ctier/A/pfa/5/initcwndbps/1466250/hightc/yes/siu/1/vprv/1/playlist_type/DVR/txp/5532434/mt/1696626071/fvip/2/keepalive/yes/fexp/24007246/beids/24350018/sparams/expire,ei,ip,id,itag,source,requiressl,ratebypass,goi,sgoap,ctier,pfa,hightc,siu,vprv,playlist_type/sig/AGM4YrMwRQIhAIxbCFktH4mxamWq1_maS3YbKeNCpYaOd8JCh0fMaCblAiBFrvxzoCzIjaLpMkxvv6Aw0dSNpYDl_KI-P1aNw7McyQ%3D%3D/lsparams/hls_chunk_host,mh,mm,mn,ms,mv,mvi,pl,initcwndbps/lsig/AK1ks_kwRQIhAOKU632ETDnLoNcjq8c7oB8fd2bxioPtYANy3Bo7LKrvAiBfywDIDKfA4RY7c0CzqpRWOso0X0Gkbx6aLox0EG11KQ%3D%3D/playlist/index.m3u8/s/0/3100/slices%3D0-62986/0
#EXTINF:4.266666,
https://rr6---sn-gqn-h5qs.googlevideo.com/videoplayback/id/189d2f25f6103cad/itag/233/source/youtube/cpn/2TC6rJxXeKRmE8UD/expire/1696647973/ei/xXYgZczEE-idp-oPltGVuAI/ip/2a0c:5a84:f209:1800:b842:b4cf:48d7:ca1c/requiressl/yes/ratebypass/yes/goi/133/sgoap/clen%3D4352437%3Bdur%3D713.642%3Bgir%3Dyes%3Bitag%3D139%3Blmt%3D1677742885047413/hls_chunk_host/rr6---sn-gqn-h5qs.googlevideo.com/mh/M-/mm/31,29/mn/sn-gqn-h5qs,sn-h5qzen7y/ms/au,rdu/mv/m/mvi/6/pl/36/ctier/A/pfa/5/initcwndbps/1466250/hightc/yes/siu/1/vprv/1/playlist_type/DVR/txp/5532434/mt/1696626071/fvip/2/keepalive/yes/fexp/24007246/beids/24350018/sparams/expire,ei,ip,id,itag,source,requiressl,ratebypass,goi,sgoap,ctier,pfa,hightc,siu,vprv,playlist_type/sig/AGM4YrMwRQIhAIxbCFktH4mxamWq1_maS3YbKeNCpYaOd8JCh0fMaCblAiBFrvxzoCzIjaLpMkxvv6Aw0dSNpYDl_KI-P1aNw7McyQ%3D%3D/lsparams/hls_chunk_host,mh,mm,mn,ms,mv,mvi,pl,initcwndbps/lsig/AK1ks_kwRQIhAOKU632ETDnLoNcjq8c7oB8fd2bxioPtYANy3Bo7LKrvAiBfywDIDKfA4RY7c0CzqpRWOso0X0Gkbx6aLox0EG11KQ%3D%3D/playlist/index.m3u8/s/3100/4267/slices%3D0-62986/1
h5qs.googlevideo.com/videoplayback/id/189d2f25f6103cad/itag/233/source/youtube/cpn/2TC6rJxXeKRmE8UD/expire/1696647973/ei/xXYgZczEE-idp-oPltGVuAI/ip/2a0c:5a84:f209:1800:b842:b4cf:48d7:ca1c/requiressl/yes/ratebypass/yes/goi/133/sgoap/clen%3D4352437%3Bdur%3D713.642%3Bgir%3Dyes%3Bitag%3D139%3Blmt%3D1677742885047413/hls_chunk_host/rr6---sn-gqn-h5qs.googlevideo.com/mh/M-/mm/31,29/mn/sn-gqn-h5qs,sn-h5qzen7y/ms/au,rdu/mv/m/mvi/6/pl/36/ctier/A/pfa/5/initcwndbps/1466250/hightc/yes/siu/1/vprv/1/playlist_type/DVR/txp/5532434/mt/1696626071/fvip/2/keepalive/yes/fexp/24007246/beids/24350018/sparams/expire,ei,ip,id,itag,source,requiressl,ratebypass,goi,sgoap,ctier,pfa,hightc,siu,vprv,playlist_type/sig/AGM4YrMwRQIhAIxbCFktH4mxamWq1_maS3YbKeNCpYaOd8JCh0fMaCblAiBFrvxzoCzIjaLpMkxvv6Aw0dSNpYDl_KI-P1aNw7McyQ%3D%3D/lsparams/hls_chunk_host,mh,mm,mn,ms,mv,mvi,pl,initcwndbps/lsig/AK1ks_kwRQIhAOKU632ETDnLoNcjq8c7oB8fd2bxioPtYANy3Bo7LKrvAiBfywDIDKfA4RY7c0CzqpRWOso0X0Gkbx6aLox0EG11KQ%3D%3D/playlist/index.m3u8/s/693833/5034/slices%3D0-640,4202012-4262932/148
#EXTINF:4.3,
https://rr6---sn-gqn-h5qs.googlevideo.com/videoplayback/id/189d2f25f6103cad/itag/233/source/youtube/cpn/2TC6rJxXeKRmE8UD/expire/1696647973/ei/xXYgZczEE-idp-oPltGVuAI/ip/2a0c:5a84:f209:1800:b842:b4cf:48d7:ca1c/requiressl/yes/ratebypass/yes/goi/133/sgoap/clen%3D4352437%3Bdur%3D713.642%3Bgir%3Dyes%3Bitag%3D139%3Blmt%3D1677742885047413/hls_chunk_host/rr6---sn-gqn-h5qs.googlevideo.com/mh/M-/mm/31,29/mn/sn-gqn-h5qs,sn-h5qzen7y/ms/au,rdu/mv/m/mvi/6/pl/36/ctier/A/pfa/5/initcwndbps/1466250/hightc/yes/siu/1/vprv/1/playlist_type/DVR/txp/5532434/mt/1696626071/fvip/2/keepalive/yes/fexp/24007246/beids/24350018/sparams/expire,ei,ip,id,itag,source,requiressl,ratebypass,goi,sgoap,ctier,pfa,hightc,siu,vprv,playlist_type/sig/AGM4YrMwRQIhAIxbCFktH4mxamWq1_maS3YbKeNCpYaOd8JCh0fMaCblAiBFrvxzoCzIjaLpMkxvv6Aw0dSNpYDl_KI-P1aNw7McyQ%3D%3D/lsparams/hls_chunk_host,mh,mm,mn,ms,mv,mvi,pl,initcwndbps/lsig/AK1ks_kwRQIhAOKU632ETDnLoNcjq8c7oB8fd2bxioPtYANy3Bo7LKrvAiBfywDIDKfA4RY7c0CzqpRWOso0X0Gkbx6aLox0EG11KQ%3D%3D/playlist/index.m3u8/s/698867/4300/slices%3D0-640,4202012-4323843/149
[.. a lot more parts ...]
#EXT-X-ENDLIST

Note:

I tried already to parse master.m3u8 to filter the streams and produce a new master.m3u8 with just the streams that I want ( 1 of audio + 1 of video ). Timming improves a bit but it's still more than 15 seconds!
I've the feeling that avformat_open_input is reading/opening/parsing all parts into each EXT-X-STREAM-INF or something like that as otherwise this makes no sense. No idea on how to solve this.
I already tried with some options like http_multiple or http_seekable, without any change on results..

Any ideas ?

Solution

I found a way to drastically reduce the time for avformat_open_input to open the stream, get stream info and so on..

The trick is to:

Parse the m3u8 files to get the uri to segments.
Create an avio context to read these segments.
Do av_probe_input_buffer before calling avformat_open_input.

Now the same videos that took my 20 seconds and sometimes more than one minute, are open in less than 1 second.

Here you go a quick and dirty sample to demo the solution, using libav + hlparse:

    #define AVIO_CTX_BUFSIZE 1024 * 32

static int avioContextRead(void *context, uint8_t *buf, int bufSize)
{
    HlsSegmentsParser *demuxer = (HlsSegmentsParser *)context;
    if (demuxer == nullptr)
    {
        return 0;
    }
    return demuxer->readBuffer(buf, bufSize);
}

int HlsSegmentsParser::readBuffer(uint8_t *buf, int bufSize)
{
    while(true){
        if(cur_segment == nullptr){
            cur_segment = segments->data;
            qDebug()<<"AVFORMAT: Reading segment:"<<cur_segment->sequence_num;
            if (avio_open2(&segment_avio_context, cur_segment->uri, AVIO_FLAG_READ, NULL, NULL) < 0) {
                // Errol;
                return -1;
            }
        }

        int bytesRead = avio_read(segment_avio_context, buf, bufSize);
        if(bytesRead<0)
        {
            qDebug()<<"AVFORMAT: Finished segment:"<<cur_segment->sequence_num;
            avio_close(segment_avio_context);
            segment_avio_context = nullptr;

            cur_segment = nullptr;
            segments = segments->next;
            continue;
        }

        return bytesRead;
    }
}

HlsSegmentsParser::HlsSegmentsParser()
{
    // Inicializa libavformat
    avformat_network_init();
}

int HlsSegmentsParser::init(const std::string &filePath)
{

    m_formatContext = avformat_alloc_context();
    m_avioCtxBuffer = static_cast<uint8_t *>(av_malloc(AVIO_CTX_BUFSIZE));
    m_avioContext = avio_alloc_context(m_avioCtxBuffer, AVIO_CTX_BUFSIZE, 0, this, &avioContextRead, nullptr, nullptr);

    auto *formatContext = (AVFormatContext *)m_formatContext;
    formatContext->pb = static_cast<AVIOContext *>(m_avioContext);
    formatContext->flags |= AVFMT_FLAG_CUSTOM_IO;


    auto content = readURLContent(filePath.c_str());
    qDebug()<<"AVFORMAT: Content: " << content.c_str();

    master_t master_playlist;
    if (HLS_OK == hlsparse_master_init(&master_playlist)) {
         if (hlsparse_master(content.c_str(), content.length(), &master_playlist)) {

            auto videoUri = master_playlist.stream_infs.data->uri;
            auto stream_contnet = readURLContent(videoUri);

            qDebug()<<"AVFORMAT: StreamContent: " << stream_contnet.c_str();
            media_playlist_t media_playlist;
            if(HLS_OK == hlsparse_media_playlist_init(&media_playlist)){
                if(hlsparse_media_playlist(stream_contnet.c_str(), stream_contnet.length(), &media_playlist)){
                    segments = &media_playlist.segments;
                    qDebug() << "AVFORMAT: Opening Input (video): " << videoUri;
                }
            }

        }
    }

    int ret = 0;

    qDebug()<<"AVFORMAT: av_probe_input_buffer: ";
    const AVInputFormat * fmt = nullptr;
    ret = av_probe_input_buffer(m_avioContext, &fmt,  nullptr, nullptr, 0 , 0);
    if (ret < 0)
    {
        qDebug() << "AVFORMAT: Unable to av_probe_input_buffer: " << ret << av_err2str(ret);
    }
    qDebug()<<"AVFORMAT: av_probe_input_buffer done! ";

    qDebug()<<"AVFORMAT: avformat_open_input: ";
    ret = avformat_open_input(&formatContext, nullptr, fmt, nullptr);
    if (ret < 0)
    {
        qDebug() << "AVFORMAT: Unable to avformat_open_input: " << ret << av_err2str(ret);
    }

    qDebug()<<"AVFORMAT: avformat_open_input done! ";


    qDebug()<<"AVFORMAT: avformat_find_stream_info" ;
    ret = avformat_find_stream_info(formatContext, nullptr);
    if (ret < 0)
    {
        qDebug() << "AVFORMAT: Unable to avformat_find_stream_info: " << ret << av_err2str(ret);
    }

    qDebug()<<"AVFORMAT: avformat_find_stream_info done!";

    const AVCodec *vcodec = nullptr;
    auto m_vStreamIdx = av_find_best_stream(formatContext,
                                       AVMEDIA_TYPE_VIDEO,
                                       -1,
                                       -1,
                                       &vcodec,
                                            0);


    qDebug()<<"AVFORMAT: m_vStreamIdx"<<m_vStreamIdx;

    AVCodecParameters *codecParam = formatContext->streams[m_vStreamIdx]->codecpar;
    AVRational fpsRatio = formatContext->streams[m_vStreamIdx]->avg_frame_rate;

    m_width = codecParam->width;
    m_height = codecParam->height;
    m_vCodec = codecParam->codec_id;

    qDebug()<<"AVFORMAT: VCodec "<<avcodec_get_name(codecParam->codec_id)<<m_width<<"x"<<m_height<<fpsRatio.den<<fpsRatio.num;

    return -1;
}

std::string HlsSegmentsParser::readURLContent(const char* url) {
    AVIOContext *avioCtx = NULL;
    uint8_t buffer[8192];
    int bytesRead;
    std::string content;

    if (avio_open2(&avioCtx, url, AVIO_FLAG_READ, NULL, NULL) < 0) {
        fprintf(stderr, "No se pudo abrir la URL.\n");
        avformat_network_deinit();
        return "";
    }

    do {
        bytesRead = avio_read(avioCtx, buffer, sizeof(buffer));
        if (bytesRead > 0) {
            content.append(reinterpret_cast<char*>(buffer), bytesRead);
        }
    } while (bytesRead > 0);

    avio_close(avioCtx);
    return content;
}

This is producing outputs like the following:

2023-10-09 23:00:36.228  D  AVFORMAT: Opening Input (video):  http://localhost:42257/itag/311/mediadata.m3u8
2023-10-09 23:00:36.228  D  AVFORMAT: av_probe_input_buffer: 
2023-10-09 23:00:36.228  D  AVFORMAT: Reading segment: 0
2023-10-09 23:00:36.423  D  AVFORMAT: av_probe_input_buffer done! 
2023-10-09 23:00:36.424  D  AVFORMAT: avformat_open_input: 
2023-10-09 23:00:36.425  D  AVFORMAT: avformat_open_input done! 
2023-10-09 23:00:36.425  D  AVFORMAT: avformat_find_stream_info
2023-10-09 23:00:36.454  D  AVFORMAT: avformat_find_stream_info done!
2023-10-09 23:00:36.454  D  AVFORMAT: m_vStreamIdx 0
2023-10-09 23:00:36.454  D  AVFORMAT: VCodec  h264 1280 x 720 1001 60000