avfoundationcifilteravvideocomposition

How do you access the frame number inside AVAsynchronousCIImageFilteringRequest?


When using a CIFilter with an AVVideoComposition, how can I access the current frame number in the AVAsynchronousCIImageFilteringRequest callback?

The best I was able to do is estimate the frame number from the time:

^(AVAsynchronousCIImageFilteringRequest * _Nonnull request) {
    double seconds = CMTimeGetSeconds(request.compositionTime);
    double fps = [[_avAsset tracksWithMediaType:AVMediaTypeVideo] firstObject].nominalFrameRate;
    int frame = round(seconds * fps);

    // (Calculate filter parameters based on frame number)
}

But this isn't accurate enough for me. Is there a way to access the frame number?


Solution

  • The key is to set the frameDuration on the AVMutableVideoComposition to match that of the underlying AVAssetTrack. if you do this, then the values of compositionTime will be even multiples of minFrameDuration and there won't be any rounding issues.

    AVMutableVideoComposition *vidComp =
        [AVMutableVideoComposition videoCompositionWithAsset:self.avAsset
                                applyingCIFiltersWithHandler:
            ^(AVAsynchronousCIImageFilteringRequest * _Nonnull request) {
                CMTime frame = self.avAssetTrack.minFrameDuration;
                int frameNum = (request.compositionTime.value * frame.timescale) /
                               (request.compositionTime.timescale * frame.value);
    
                // (Calculate filter parameters based on frame number)
            }];
    
    vidComp.frameDuration = self.avAssetTrack.minFrameDuration;
    

    In my case of a ~60 fps MP4 movie, avAssetTrack.minFrameDuration = (1001/60000), but vidComp.frameDuration = (1501/90000) before I update it. This resulted in slow drift of 'compositionTime' in the callback, which resulted in occasional repeated frames.