swiftmacosimage-processingcocoaios-vision

CGImage.crop(to:) returns a weird crop on MacOS


I was facing a weird behavior while using the method CGImage.crop(to:).

My goal is cropping a user face using the apple vision-framework to train AI-based models on it, but while running the vision on the utkface dataset I got a completely weird crop, see examples above.

Here is the relevant piece of code:


        let image = NSImage(byReferencingFile: imagePath)!
        let cgImage = image.cgImage(forProposedRect: nil, context: nil, hints: nil)!
        
        let visionRequest = VNDetectFaceRectanglesRequest()

        let handler = VNImageRequestHandler(cgImage: cgImage, orientation: .up, options: [:])
        do {
            try await handler.perform([visionRequest])
        } catch {
            print("Failed ... \(error.localizedDescription)")
            return
        }

        let observations = visionRequest.results?
            .filter {
                $0.confidence >= request.faceCaptureThreshold &&
                ($0.boundingBox.size.width >= 0.1 || $0.boundingBox.size.height >= 0.1)
            } ?? []

        for (index, observation) in observations.enumerated() {
            let normalizedBoundingBox = observation.boundingBox
            let boundingBox = VNImageRectForNormalizedRect(normalizedBoundingBox, cgImage.width, cgImage.height)
            let croppedImage = cgImage.cropping(to: boundingBox)!
            // Redacted: store croppedImage on Disk
        }

While debugging this, I was able to discovery that everything is going fine, until the function crop(to:) be called, what's weird, because when I draw a rect above the original image using CoreImage-related functions the rect is on the right place, but the crop is completely different.

I was able to work around this, recreating the image using CoreImage, something like:

            let croppedImage = DrawImageInCGContext(size: boundingBox.size) { (context) -> () in
                context.draw(cgImage, in: .init(origin: .init(x: -boundingBox.minX, y: -boundingBox.minY), size: CGSize(width: cgImage.width, height: cgImage.height)), byTiling: true)
            }

But I didn't understand what I was doing wrong when using the CGImage.crop(to:), or if it's a bug on Apple-side.

Any ideas?

Original Image

Image with rect

Weird Crop


Solution

  • What's happening here is that macOS uses a different coordinate system starting from bottom left corner instead of up left corner, like iOS. But CGImage.crop(to:) expects an up left corner based rect to crop the image, so a translation is needed to crop it correctly.

    A small change is enough to fix that:

                let normalizedBoundingBox = observation.boundingBox
                let boundingBox = VNImageRectForNormalizedRect(normalizedBoundingBox, cgImage.width, cgImage.height)
                
                let flippedY = CGFloat(cgImage.height) - boundingBox.maxY
                
                let macOSBoundingBox = CGRect(
                    origin: .init(x: boundingBox.minX, y: flippedY),
                    size: boundingBox.size
                )