objective-cavfoundationavassetwriteravassetexportsessionaudio-video-sync

AVAssetExportSession causes audio video to desync when trying to merge multiple video/audio clips


I am trying to merge a bunch of video clips containing both audio and video into one longer video file.
AVMutableComposition *mainComposition = [[AVMutableComposition alloc] init]; AVMutableCompositionTrack *compositionVideoTrack = [mainComposition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];

AVMutableCompositionTrack *soundtrackTrack = [mainComposition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:kCMPersistentTrackID_Invalid];
CMTime insertTime = kCMTimeZero;


NSArray *videoFileNames;
for (NSString  *videoPathString in videoFileNames) {

    AVAsset *videoAsset = [AVAsset assetWithURL:[NSURL fileURLWithPath:[NSString stringWithFormat:@"/Volumes/videoClips/CLIPS/%@",videoPathString]]];

                NSError *err1;
                NSError *err2;
                [compositionVideoTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, videoAsset.duration) ofTrack:[[videoAsset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0] atTime:insertTime error:&err1];

                [soundtrackTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, videoAsset.duration) ofTrack:[[videoAsset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0] atTime:insertTime error:&err2];
                insertTime = CMTimeAdd(insertTime, videoAsset.duration);
}


NSURL *outptVideoUrl = [NSURL fileURLWithPath:@"/Volumes/mergedVideo/finalizedMoviev.mp4"];
AVAssetExportSession *exporter = [[AVAssetExportSession alloc] initWithAsset:mainComposition presetName:AVAssetExportPreset1280x720];

exporter.outputURL= outptVideoUrl;
exporter.outputFileType = AVFileTypeMPEG4;   
exporter.shouldOptimizeForNetworkUse = NO;
[exporter exportAsynchronouslyWithCompletionHandler:^{
    dispatch_async(dispatch_get_main_queue(), ^{
        NSLog(@"Completed video export");
    });
}];`

This works however there is a desync between the audio and video after the first clip. Video if you watch the first clip the audio is perfectly synced then if you skip ahead towards the 15 minute mark you will notice a definite desync between video and audio. After a little bit of searching I was able to this postiOS AVFoundation audio/video out of sync

Which has a link inside of it that explains what is going on that audio enques blank samples when compressing. On both the front and back of the compressed audio file. Technical Note TN2258 AAC Audio - Encoder Delay and Synchronization

Now I think what I can do to solve this problem is set up an AVAssetReader with both video and audio outputs read through each CMSamplebuffer of audio and video frame decompress them then and put it into an AVAssetWritter and repeat this for each clip. But what I am unsure of is how to keep audio and video synced the whole time. Am I just taking each audio frame and adding it in as is or am I trying to trim out the blank priming and remainding packets frames?

Thanks for any help!


Solution

  • I was right in what I assumed by looking at each frame individually and writing them through an avassetwriter does work. However if clips are not the same audio sample rate you run into trouble thats the new issue I am working through now.