Fastest way to get the RGB average inside of a non-rectangular contour in the CMSampleBuffer

I am trying to get the RGB average inside of a non-rectangular multi-edge (closed) contour generated over a face landmark region in the frame (think of it as a face contour) from AVCaptureVideoDataOutput. I currently have the following code,

        let landmarkPath = CGMutablePath()
        let landmarkPathPoints = landmark.normalizedPoints
            .map({ landmarkPoint in
                CGPoint(
                    x: landmarkPoint.y * faceBoundingBox.height + faceBoundingBox.origin.x,
                    y: landmarkPoint.x * faceBoundingBox.width + faceBoundingBox.origin.y)
            })
        landmarkPath.addLines(between: landmarkPathPoints)
        landmarkPath.closeSubpath()

        let averageFilter = CIFilter(name: "CIAreaAverage", parameters: [kCIInputImageKey: frame, kCIInputExtentKey: landmarkPath])!
        let outputImage = averageFilter.outputImage!

However, it currently throws *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[__NSCFType CGRectValue]: unrecognized selector sent to instance 0x283a57a80' terminating with uncaught exception of type NSException. I suspect this is as the kCIInputExtentKey is not a proper CIVector rectangular object. Is there anyway to fix this? How can I define a non-rectangular region for the CIAreaAverage filter? If not possible, what's the most efficient way of getting the average RGB across the region of interest?

Thanks a lot in advance!

If you could make all pixels outside of the contour transparent then you could use CIKmeans filter with inputCount equal 1 and the inputExtent set to the extent of the frame to get the average color of the area inside the contour (the output of the filter will contain 1-pixel image and the color of the pixel is what you are looking for).

Now, to make all pixels transparent outside of the contour, you could do something like this:

1. Create a mask image but setting all pixels inside the contour white and black outside (set background to black and fill the path with white).
2. Use CIBlendWithMask filter where:
   • inputBackgroundImage is a fully transparent (clear) image
   • inputImage is the original frame
   • inputMaskImage is the mask you created above

The output of that filter will give you the image with all pixels outside the contour fully transparent. And now you can use the CIKMeans filter with it as described at the beginning.

BTW, if you want to play with every single of the 230 filters out there check this app out: https://apps.apple.com/us/app/filter-magic/id1594986951

UPDATE:

CIFilters can only work with CIImages. So the mask image has to be a CIImage as well. One way to do that is to create a CGImage from CAShapeLayer containing the mask and then create CIImage out of it. Here is how the code could look like:

// Create the closed contour path from points
let path = CGMutablePath()
path.addLines(between: points)
path.closeSubpath()

// Create CAShapeLayer matching the dimensions of the input frame
let layer = CAShapeLayer()
layer.frame = frame.extent // Assuming frame is the input CIImage with the face

// Set background and fill color and set the path
layer.fillColor = UIColor.white.cgColor
layer.backgroundColor = UIColor.black.cgColor
layer.path = path

// Render the contents of the CAShapeLayer to CGImage
let width = Int(layer.bounds.width)
let height = Int(layer.bounds.height)
let context = CGContext(data: nil,
                        width: width,
                        height: height,
                        bitsPerComponent: 8,
                        bytesPerRow: 4 * width,
                        space: CGColorSpaceCreateDeviceRGB(),
                        bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue)!

layer.render(in: context)
let cgImage = context.makeImage()!

// Create CIImage out of it
let ciImage = CIImage(cgImage: cgImage)

// To create clear background CIImage just do this:
let bgImage = CIImage.clear.cropped(to: frame.extent)