Filter on CALayer except for a shape which is an union of (non necessarily distinct) rectangles

I want to apply a CIFilter to a CALayer except for an area which is an union of rectangles. I post my not working code, it does exactly the opposite, i.e. the filter is applied only in the rectangles and not outside!

func refreshTheFrameWithEffect() {
    self.layer!.masksToBounds = true
    self.layerUsesCoreImageFilters = true
    self.layer!.needsDisplayOnBoundsChange = true

    let filter = CIFilter(name: "CICircleSplashDistortion")
    filter!.setDefaults()
    self.layer!.backgroundFilters = [filter!]

    var excludedRects: [CGRect] = [] //INITIALISE THEM HOW YOU PREFER

    let maskLayer = CAShapeLayer()
    maskLayer.frame = self.bounds
    self.layer!.fillMode = kCAFillRuleEvenOdd
    var maskPath = NSBezierPath()
    for rect in excludedRects {
        maskPath.append(NSBezierPath(rect: rect))
    }

    maskLayer.path = maskPath.CGPath
    self.layer!.mask = maskLayer
    self.layer!.needsDisplay()
}

and then the following code from the Internet since NSBezierPath does not have the attribute CGPath, unlike UIBezierPath.

public var CGPath: CGPath {
    let path = CGMutablePath()
    var points = [CGPoint](repeating: .zero, count: 3)
    for i in 0 ..< self.elementCount {
        let type = self.element(at: i, associatedPoints: &points)
        switch type {
        case .moveToBezierPathElement: path.move(to: CGPoint(x: points[0].x, y: points[0].y) )
        case .lineToBezierPathElement: path.addLine(to: CGPoint(x: points[0].x, y: points[0].y) )
        case .curveToBezierPathElement: path.addCurve(      to: CGPoint(x: points[2].x, y: points[2].y),
                                                            control1: CGPoint(x: points[0].x, y: points[0].y),
                                                            control2: CGPoint(x: points[1].x, y: points[1].y) )
        case .closePathBezierPathElement: path.closeSubpath()
        }
    }
    return path
}

AFAIK, you can't mask a layer to the inverse of a path.

A couple of observations:

1. If you were just trying to knock out the paths inside the whole view, you could do that with the trick of creating a path that consists of the whole CGRect plus the various interior paths and leverage the even/odd winding/fill rule, but that won't work if your interior paths overlap with each other.

2. You can mask images to the inverse of a path (by creating a separate "image mask"), but that won't work for dynamic CALayer masking. It's used for masking a NSImage. So, if you were OK using a snapshot for the filtered part of the view, that's an option.

   See code snippet below for example of using image masks.

3. Another approach is to apply your filter to the whole view, snapshot it, put that snapshot underneath the view in question and then mask the top level view to your interior paths. In effect, mask the un-filtered view to your interior paths, revealing a filtered snapshot of your view below it.

4. Yet approach would be to create a path representing the outline of the union of all of your interior paths. If the paths are simple (e.g. non-rotated rectangles) this is pretty easy. If the paths are complex (some rotated, some non-rectangular paths, etc.), this gets hairy. But the trivial scenario isn't too bad. Anyway, if you do that, then you can fall back to that even-odd trick.

I'm not wholly satisfied with any of these approaches, but I don't see any other way to accomplish what you're looking for. Hopefully someone will suggest some better ways to tackle this.

To expand on option 2 (using an image mask created by drawing a few paths, possibly overlapping), in Swift 3 you can do something like:

private func maskImageWithPaths(image: NSImage, size: CGSize) -> NSImage {
    // define parameters for two overlapping paths
    
    let center1 = CGPoint(x: size.width * 0.40, y: size.height / 2)
    let radius1 = size.width * 0.20
    
    let center2 = CGPoint(x: size.width * 0.60 , y: size.height / 2)
    let radius2 = size.width * 0.20
    
    // create these two overlapping paths
    
    let path = CGMutablePath()
    path.move(to: center1)
    path.addArc(center: center1, radius: radius1, startAngle: -.pi / 2, endAngle: .pi * 3 / 2, clockwise: false)
    path.move(to: center2)
    path.addArc(center: center2, radius: radius2, startAngle: -.pi / 2, endAngle: .pi * 3 / 2, clockwise: false)
    
    // create image from these paths
    
    let imageRep = NSBitmapImageRep(bitmapDataPlanes: nil, pixelsWide: Int(size.width), pixelsHigh: Int(size.height), bitsPerSample: 8, samplesPerPixel: 4, hasAlpha: true, isPlanar: false, colorSpaceName: NSDeviceRGBColorSpace, bitmapFormat: .alphaFirst, bytesPerRow: 4 * Int(size.width), bitsPerPixel: 32)!
    let context = NSGraphicsContext(bitmapImageRep: imageRep)!
    
    context.cgContext.addPath(path)
    context.cgContext.setFillColor(NSColor.blue.cgColor)
    context.cgContext.fillPath()
    let maskImage = context.cgContext.makeImage()!
    
    let mask = CGImage(maskWidth: Int(size.width), height: Int(size.height), bitsPerComponent: 8, bitsPerPixel: 32, bytesPerRow: 4 * Int(size.width), provider: maskImage.dataProvider!, decode: nil, shouldInterpolate: true)!
    
    let finalImage = image.cgImage(forProposedRect: nil, context: nil, hints: nil)!.masking(mask)!
    return NSImage(cgImage: finalImage, size: size)
}

That yields:

This masks out the drawn paths.