Speed problems with realtime recording with libavcodec and libavformat

I am trying to use libavcodec and libavformat to write an mp4 video file in realtime using h264. I am using an approach heavily inspired by this answer here This works well as a non-realtime solution however, avcodec_receive_packet() starts running much slower after 20 frames or so (this is usually around the first time it returns success and thus av_interleaved_write_frame() is called for the first time). This is so slow that my writing cannot work in realtime.

Solutions I have tried:

• Enabling multithreading on my codec context
• Running avcodec_receive_packet() and av_interleaved_write_frame() on a separate thread to my capture from the realtime video source
• Changing the gop_size in the video context
• Lowering my bitrate in the video context

Is there anything I'm missing? Possibly some fundamental rules to capturing video in realtime. I am not very experienced with programming with video.

I have solved this by not using h.264 encoding and instead using libavcodec's mpeg2video encoder. This leads to a much larger file size however the frame by frame encoding has a much more consistent processing time. Thanks to @G.M.'s comment for that. I have not yet tested any other encoders so possibly those could be helpful too.

Another possible solution was using GPU acceleration as mentioned by @VC.One, however for my use case, this was not feasible due to the target hardware for this to run not being known. A possible hybrid method is to determine the target hardware in the code and enable H.264 with GPU processing if a powerful enough GPU is available and to use mpeg2video encoding if not.