I was facing a weird behavior while using the method CGImage.crop(to:)
.
My goal is cropping a user face using the apple vision-framework to train AI-based models on it, but while running the vision on the utkface
dataset I got a completely weird crop, see examples above.
Here is the relevant piece of code:
let image = NSImage(byReferencingFile: imagePath)!
let cgImage = image.cgImage(forProposedRect: nil, context: nil, hints: nil)!
let visionRequest = VNDetectFaceRectanglesRequest()
let handler = VNImageRequestHandler(cgImage: cgImage, orientation: .up, options: [:])
do {
try await handler.perform([visionRequest])
} catch {
print("Failed ... \(error.localizedDescription)")
return
}
let observations = visionRequest.results?
.filter {
$0.confidence >= request.faceCaptureThreshold &&
($0.boundingBox.size.width >= 0.1 || $0.boundingBox.size.height >= 0.1)
} ?? []
for (index, observation) in observations.enumerated() {
let normalizedBoundingBox = observation.boundingBox
let boundingBox = VNImageRectForNormalizedRect(normalizedBoundingBox, cgImage.width, cgImage.height)
let croppedImage = cgImage.cropping(to: boundingBox)!
// Redacted: store croppedImage on Disk
}
While debugging this, I was able to discovery that everything is going fine, until the function crop(to:)
be called, what's weird, because when I draw a rect above the original image using CoreImage-related functions the rect is on the right place, but the crop is completely different.
I was able to work around this, recreating the image using CoreImage
, something like:
let croppedImage = DrawImageInCGContext(size: boundingBox.size) { (context) -> () in
context.draw(cgImage, in: .init(origin: .init(x: -boundingBox.minX, y: -boundingBox.minY), size: CGSize(width: cgImage.width, height: cgImage.height)), byTiling: true)
}
But I didn't understand what I was doing wrong when using the CGImage.crop(to:)
, or if it's a bug on Apple-side.
Any ideas?
What's happening here is that macOS uses a different coordinate system starting from bottom left corner instead of up left corner, like iOS. But CGImage.crop(to:)
expects an up left corner based rect to crop the image, so a translation is needed to crop it correctly.
A small change is enough to fix that:
let normalizedBoundingBox = observation.boundingBox
let boundingBox = VNImageRectForNormalizedRect(normalizedBoundingBox, cgImage.width, cgImage.height)
let flippedY = CGFloat(cgImage.height) - boundingBox.maxY
let macOSBoundingBox = CGRect(
origin: .init(x: boundingBox.minX, y: flippedY),
size: boundingBox.size
)