How to combine / nest CALayer masks?

Is it possible use nested CALayer mask to create one combined mask? My goal is to create a complex shapes by combining / masking other complex shapes and applying a gradient to the result.

For example one CAShapeLayer my hold a path which draws a unicorn head and a second CAShapeLayer my hold a path which draws a flower. By applying the flower layer as mask to the unicorn layer, I get a "flowerly unicorn". In then next step I could than apply this complex shape to a CAGradientLayer to get a "flowerly unicorn rainbow".

However, when applying the unicorn layer (with the flower mask) as mask for the gradient, the result is a unicorn rainbow. The flower mask already applied to the unicorn is skipped.

Is there any other way to combine / nest layer mask to create such shapes?

The following code illustrates the problem using some simpler shapes:

let rectLayer = CAShapeLayer()
rectLayer = // ... a simple rect shape

let circleLayer = CAShapeLayer()
circleLayer = // ... a circle shape, half overlapping the rect

// Apply rect as mask to create half circle
circleLayer.mask = rectLayer

// Some gradient
let gradientLayer = CAGradientLayer()
gradientLayer = // ... set up gradient with some colors

// Apply masked circle as mask to get a half circle with gradient
gradientLayer.mask = circleLayer

Now I would assume to get a half circle shape with a gradient. However, I get a circle with a gradient instead. It seem that when applying the circleLayer as mask to the gradientLayer the rect mask is not considered / applied...

Am I doing something wrong? Is there another way to solve this? Masking and combining masks is quite a common task in many different applications. Thus I wonder if such a powerful framework as CA has a way of doing this.

Solution

As @DuncanC pointed out mask cannot be nested. So, applying Layer A as mask to Layer B and than applying Layer B as mask to Layer C does not work as expected. This will only use the original content of B as mask and not taking account the changed made by Layer A.

I was able to get around this limitation by first "flattening" the result of Layer B + Mask Layer A. This is done by creating an image of this result and than applying this as mask to Layer C:

let layerA = CAShapeLayer() // ... add some shapes
let layerB = CAShapeLayer() // ... add some shapes

// Apply A as mask to B
layerB.mask = layerA

// Flatten B
let flatB = layerB.flatten()

// Apply flatB as mask to C
let layerC = CAShapeLayer() // ... add some shapes
layerC.mask = flatB

The flattening is done using this CALayer extension:

extension CALayer {
    func flatten() -> CALayer {
        guard let colorSpace = CGColorSpace(name: CGColorSpace.sRGB),
              let ctx = CGContext(data: nil, width: Int(bounds.width), height: Int(bounds.height), bitsPerComponent: 8, bytesPerRow: 4*Int(bounds.width), space: colorSpace, bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue) else { return CALayer() }
    
        ctx.translateBy(x: 0, y: bounds.height)
        ctx.scaleBy(x: 1.0, y: -1.0)

        render(in: ctx)
        let image = ctx.makeImage()
    
        let flattenedLayer = CALayer()
        flattenedLayer.frame = frame
        flattenedLayer.contents = image
    
        return flattenedLayer
    }
}

This might not be the perfect solution but it works very well in my case. Maybe this might help others as well.