I split video and audio from an mp4 using
ffmpeg -i orig.mp4 -an -c copy orig_video.mp4
ffmpeg -i orig.mp4 -vn -c copy orig_audio.aac
I subsequently merge again using
ffmpeg -i orig_video.mp4 -i orig_audio.aac -c copy orig_rejoined.mp4
The problem is, in the rejoined mp4, audio and video are out of sync. The video was recorded with my android phone. I also converting the video to a constant framerate, but this didn't change anything. Any ideas? What I want to do in the end is replace the audio by a mic recording, which is automatically aligned. I want to do this using moviepy, however this uses ffmpeg under the hood, introducing this strange alignment bug.
the problem was a non-zero start time of the audio stream in the original mp4 container. it is not immediatly visible if ffprobing the original container i think (although it can be revealed with targeted probing), but what it revealed it to me was splitting of the audio in a separate mp4 container and then using ffprobe
ffmpeg -i orig.mp4 -vn -c copy orig-audio.mp4
ffprobe orig-audio.mp4
which gives "Duration: 00:02:17.24, start: 0.224000, bitrate: 257 kb/s", ie. a non-zero start time of 0.224s of the audio stream. this delay is lost when splitting and merged without accounting for it. gpt explained to me, that it is common to add such a delay to the audio stream to account for delays present in the video. we can account for it in the merge using:
ffmpeg -i orig-video.mp4 -itsoffset 00:00:00.224 -i orig-audio.aac -c copy merged_with_offest.mp4
note that i still noted some out of sync when sending the video with whatsapp, but otherwise it works.