VLC huge buffering times over rtp for local H264 stream

I'm outputting an H264 stream, encoded by my application using ffmpeg. I can display it using ffplay, but when trying to view the stream in VLC, I only get the first frame, or it looks like that's the case.

The messages output shows that it is "buffering", taking around a minute to get to 100% when the frame updates. When using ffplay, the latency is about 50-100ms at worst.

I am sending to rtp://127.0.0.1:6666?pkt_size=1316 with the format rtp_mpegts. I am new to this and it's highly likely I haven't set the frame up completely correctly. The process is (minus declarations and error checking)

codec_name = "libx264";
codec = avcodec_find_encoder_by_name(codec_name.c_str());
context = avcodec_alloc_context3(codec);
pkt = av_packet_alloc();
context->bit_rate = 5 * Mega;
context->width = info.DisplayWidth;
context->height = info.DisplayHeight;
context->time_base = { 1, FPS };
context->framerate = { FPS, 1 };
context->gop_size = 100;
context->max_b_frames = 1;            
context->pix_fmt = AV_PIX_FMT_YUV420P;
if (codec->id == AV_CODEC_ID_H264)
            {
                check_ret("set option: preset", av_opt_set(context->priv_data, "preset", "fast", 0));
                check_ret("set option: tune", av_opt_set(context->priv_data, "tune", "zerolatency", 0));
                check_ret("set option: profile", av_opt_set(context->priv_data, "profile", "baseline", 0));                
            }
check_ret("open codec", avcodec_open2(context, codec, NULL));

// setup the stream 
fmt = (AVOutputFormat*)av_guess_format("rtp_mpegts", NULL, NULL);

avformat_alloc_output_context2(&avfctx, fmt, fmt->name,
            "rtp://127.0.0.1:6666?pkt_size=1316"); 
        
avio_open(&avfctx->pb, avfctx->url, AVIO_FLAG_WRITE);
AVStream* stream = avformat_new_stream(avfctx, codec);
avcodec_parameters_from_context(stream->codecpar, context);
stream->time_base.num = 1;
stream->time_base.den = FPS;
avformat_write_header(avfctx, NULL);

// then the encoding (in an output loop)
<not shown: get frame from swapchain, sws_scale from rgba to yuv>
yuvFrame->pts = i++; // i is incremented every frame
avcodec_send_frame(enc_ctx, yuvFrame);
 while (ret >= 0) {
  ret = avcodec_receive_packet(enc_ctx, pkt);          
  //ret = av_interleaved_write_frame(avfctx, pkt); was using this, don't seem to need it
  ret = av_write_frame(avfctx, pkt);
  av_packet_unref(pkt);
}

The VLC output looks like this:

main debug: using hw decoder module "d3d11va"
avcodec info: Using D3D11VA (NVIDIA GeForce RTX 2080 Super with Max-Q Design, vendor 10de(NVIDIA), device 1e93, revision a1) for hardware decoding
qt debug: Logical video size: 1280x720
main debug: resized to 1280x720
main debug: VoutDisplayEvent 'resize' 1280x720
main debug: Received first picture
main debug: Buffering 1%
main debug: Buffering 2%
main debug: Buffering 3%
main debug: auto hiding mouse cursor
main debug: Buffering 4%
main debug: Buffering 5%
main debug: Buffering 6%
main debug: Buffering 7%
main debug: Buffering 8%
main debug: Buffering 9%
main debug: Buffering 10%
main debug: auto hiding mouse cursor
main debug: Buffering 11%
rtp warning: 1 packet(s) lost
rtp warning: 1 packet(s) lost
rtp warning: 1 packet(s) lost
ts warning: discontinuity received 0x3 instead of 0xd (pid=256)
ts warning: discontinuity received 0x5 instead of 0xf (pid=256)
ts warning: discontinuity received 0x1 instead of 0xb (pid=256)
main debug: Buffering 12%
main debug: Buffering 13%
main debug: Buffering 14%
main debug: Buffering 15%
main debug: Buffering 16%
main debug: Buffering 17%
main debug: Buffering 18%
main debug: auto hiding mouse cursor
main debug: Buffering 19%
main debug: Buffering 20%

The problem with my approach above was that it was based on the ffmpeg example encode_video.c with some bits for stream output borrowed from google.

Thanks to @rotem I started putting together a standalone executable and stumbled on the example muxing.c in the ffmpeg examples.

This let me find the steps I was missing: set the stream index on the packet, and rescale the time:

av_packet_rescale_ts(pkt, context->time_base, stream->time_base);
pkt->stream_index = stream->index;
int ret = av_interleaved_write_frame(avfctx, pkt);