Swift: Play a local video and apply CIFilter in realtime

I'm trying to play a local video and apply a CIFilter in realtime with no lag. How can I do that? I already know how to apply a CIFilter to a video in AVPlayer but the performance it's not as fast as I want.

This is my current code:

@objc func exposure(slider: UISlider, event: UIEvent) {
    if let touchEvent = event.allTouches?.first {
        switch touchEvent.phase {
        case .moved:
            player.currentItem?.videoComposition = AVVideoComposition(asset: player.currentItem!.asset, applyingCIFiltersWithHandler: { request in
             
                let exposureFilter = CIFilter.exposureAdjust()
                exposureFilter.inputImage = request.sourceImage.clampedToExtent()
                exposureFilter.ev = slider.value
             
                let output = self.exposureFilter.outputImage!.cropped(to: request.sourceImage.extent)

                // Provide the filter output to the composition
                request.finish(with: output, context: nil)
         })
        default:
            break
        }
    }
}

Solution

The problem is that you re-create and re-assign the video composition to the player item every time the slider value changes. This is very costly and unnecessary. You can do the following instead:

Create the filter somewhere outside the composition block and keep a reference to it, for instance in a property.
Also, create the composition only once and let it apply the referenced filter (instead of creating a new one with every frame).
When the slider value changes, only set the corresponding parameter value of the filter. The next time the composition will render a frame, it will automatically use the new parameter value because it uses a reference to the just-changed filter.

Something like this:

let exposureFilter = CIFilter.exposureAdjust()

init() {
    // set initial composition
    self.updateComposition()
}

func updateComposition() {
    player.currentItem?.videoComposition = AVVideoComposition(asset: player.currentItem!.asset, applyingCIFiltersWithHandler: { request in
        self.exposureFilter.inputImage = request.sourceImage.clampedToExtent()
        let output = self.exposureFilter.outputImage!.cropped(to: request.sourceImage.extent)
        request.finish(with: output, context: nil)
    })
}

@objc func exposureChanged(slider: UISlider) {
    self.exposureFilter.ev = slider.value
    // we need to re-set the composition if the player is paused to cause an update (see remark below)
    if player.rate == 0.0 {
        self.updateComposition()
    }
}

(By the way, you can just do slider.addTarget(self, action:#selector(exposureChanged(slider:)), for: .valueChanged) to get notified when the slider value changes. No need to evaluate events.)

One final note: There actually is a use case when you want to re-assign the composition, which is when the player is currently paused but you still want to show a preview of the current frame with the filter values change. Please refer to this technical note from Apple on how to do that.