swiftcifacefeature

Swift - Add carnival mask to a photo which contains face


I have photo with face on it.

I have carnival mask:

carnival mask

With this function I detect the face:

   let ciImage = CIImage(cgImage: photo)
   let options = [CIDetectorAccuracy: CIDetectorAccuracyHigh]
   let faceDetector = CIDetector(ofType: CIDetectorTypeFace, context: nil, options: options)!
   let faces = faceDetector.features(in: ciImage)
   if let face = faces.first as? CIFaceFeature {

   }

How to detect the holes in the mask ?

How I can place the mask on the face after detecting the holes of the mask?


Solution

  • I'd probably try this approach:

    Get leftEyePosition, rightEyePosition, and the faceAngle value. (All part of CIFaceFeature)

    Calculate the distance in between the left and right eye.

    Here is a link on how to calculate the distance: https://www.hackingwithswift.com/example-code/core-graphics/how-to-calculate-the-distance-between-two-cgpoints

    Create constants with the original dimensions of the mask as well as the x and y distance to the center of one of the eyes.

    With the distance of the eyes you calculate the new width of your mask proportionally.

    That should get you a mask with the right size. Also calculate the new x and y distances to the center of one of eyes of the mask the same way.

    Adjust all values proportionally again to fit the final intended size on the screen.

    Place the mask on the photo using the coordinates of the eyes, offsetting by the mask eye to corner distance.

    Use the faceAngle value to rotate the mask.

    Before importing the mask into the project, convert it to a png with transparent background, remove the white background. You could do that in code, but that would be a lot of work and depending on the masks source file it might not turn out as well.

    UPDATE, I've tried my solution. It's a simple iOS one screen app, just copy the code into the ViewController.swift file, add your mask as png and a photo of a face as photo.jpg into the project and it should work.

    Here is a link to your photo as png if you want to try:

    QPTF1.png

       import UIKit
    
    class ViewController: UIViewController {
    
        override func viewDidLoad() {
            super.viewDidLoad()
            imageMethod()
        }
    
        func imageMethod() {
    
        let uiMaskImage = UIImage(named: "QPTF1.png") //converted to png with transperancy before adding to the project
        let maskOriginalWidth = CGFloat(exactly: 655.0)!
        let maskOriginalHeight = CGFloat(exactly: 364.0)!
        let maskOriginalEyeDistance = CGFloat(exactly: 230.0)! //increase or decrease value to change the final size of the mask
        let maskOriginalLeftEyePossitionX = CGFloat(exactly: 203.0)! //increase or decrease to fine tune mask possition on x axis
        let maskOriginalLeftEyePossitionY = CGFloat(exactly: 200.0)! //increase or decrease to fine tune mask possition on y axis
    
    
        //This code assumes the image AND face orientation is always matching the same orientation!
        //The code needs to be adjusted for other cases using UIImage.Orientation to get the orientation and adjusts the coordinates accordingly.
        //CIDetector might also not detect faces which don't have the same orientation as the photo. Try to use CIDetectorImageOrientation to look for other orientations of no face has been detected.
        //Also you might want to use other orientation points and scale values (right eye, nose etc.) in case the left eye, and left to right eye distance is not available.
        //Also this code is very wordy, pretty sure it can be cut down to half the size and made simpler on many places.
    
        let uiImageFace = UIImage(named: "photo.jpg")
        let ciImageFace = CIImage(image: uiImageFace!)
        let options = [CIDetectorAccuracy: CIDetectorAccuracyHigh]
        let faceDetector = CIDetector(ofType: CIDetectorTypeFace, context: nil, options: options)!
        let faces = faceDetector.features(in: ciImageFace!)
        if let face = faces.first as? CIFaceFeature {
    
    
            /*
            Getting the distances and angle based on the original photo
            */
            let faceAngle = face.faceAngle
            let rotationAngle = CGFloat(faceAngle * .pi / 180.0)
    
            //The distance in between the eyes of the original photo.
            let originalFaceEyeDistance = CGPointDistance(from: face.leftEyePosition, to: face.rightEyePosition)
    
    
            /*
            Adjusting the mask and its eye coordinates to fit the original photo.
            */
    
            //Setting the scale mask : original.
            let eyeDistanceScale = maskOriginalEyeDistance / originalFaceEyeDistance
    
            //The new dimensions of the mask.
            let newMaskWidth = maskOriginalWidth/eyeDistanceScale
            let newMaskHeight = maskOriginalHeight/eyeDistanceScale
    
            //The new mask coordinates of the left eye in relation to the original photo.
            let newMaskLeftEyePossitionX = maskOriginalLeftEyePossitionX / eyeDistanceScale
            let newMaskLeftEyePossitionY = maskOriginalLeftEyePossitionY / eyeDistanceScale
    
            /*
            Adjusting the size values to fit the desired final size on the screen.
            */
    
            //Using the width of the screen to calculate the new scale.
            let screenScale = uiImageFace!.size.width / view.frame.width
    
            //The new final dimensions of the mask
            let scaledToScreenMaskWidth = newMaskWidth / screenScale
            let scaledToScreenMaskHeight = newMaskHeight / screenScale
    
            //The new final dimensions of the photo.
            let scaledToScreenPhotoHeight = uiImageFace!.size.height / screenScale
            let scaledToScreenPhotoWidth = uiImageFace!.size.width / screenScale
    
            //The new eye coordinates of the photo.
            let scaledToScreenLeftEyeFacePositionX = face.leftEyePosition.x / screenScale
            let scaledToScreenLeftEyeFacePositionY = (uiImageFace!.size.height - face.leftEyePosition.y) / screenScale //reversing the y direction
    
            //The new eye to corner distance of the mask
            let scaledToScreenMaskLeftEyeX = newMaskLeftEyePossitionX / screenScale
            let scaledToScreenMaskLeftEyeY = newMaskLeftEyePossitionY / screenScale
    
            //The final coordinates for the mask
            let adjustedMaskLeftEyeX = scaledToScreenLeftEyeFacePositionX - scaledToScreenMaskLeftEyeX
            let adjustedMaskLeftEyeY = scaledToScreenLeftEyeFacePositionY - scaledToScreenMaskLeftEyeY
    
            /*
            Showing the image on the screen.
            */
    
            let baseImageView = UIImageView(image: uiImageFace!)
            //If x and y is not 0, the mask x and y need to be adjusted too.
            baseImageView.frame = CGRect(x: CGFloat(exactly: 0.0)!, y: CGFloat(exactly: 0.0)!, width: scaledToScreenPhotoWidth, height: scaledToScreenPhotoHeight)
            view.addSubview(baseImageView)
    
            let maskImageView = UIImageView(image: uiMaskImage!)
            maskImageView.frame = CGRect(x: adjustedMaskLeftEyeX, y: adjustedMaskLeftEyeY, width: scaledToScreenMaskWidth, height: scaledToScreenMaskHeight)
                maskImageView.transform = CGAffineTransform(rotationAngle: rotationAngle)
                view.addSubview(maskImageView)
            }
    
        }
    
        func CGPointDistanceSquared(from: CGPoint, to: CGPoint) -> CGFloat {
            return (from.x - to.x) * (from.x - to.x) + (from.y - to.y) * (from.y - to.y)
        }
    
        func CGPointDistance(from: CGPoint, to: CGPoint) -> CGFloat {
            return sqrt(CGPointDistanceSquared(from: from, to: to))
        }
    
    }
    

    Result:

    enter image description here

    Here is my uncommented approach to scan for the eyes. It has still some quirks but should be a starting point.

     import UIKit
    
    class ViewController: UIViewController {
    
        override func viewDidLoad() {
            super.viewDidLoad()
            imageMethod()
        }
    
        func imageMethod() {
    
            struct coords {
                let coord: (x: Int, y: Int)
                let size: Int
            }
    
        let uiMaskImage = UIImage(named: "QPTF1.png") //converted to png with transperancy before adding to the project
    
        let uiMaskImage2 = UIImage(named: "QPTF1.png")
        let ciMaskImage2 = CIImage(image: uiMaskImage2!)
        let context = CIContext(options: nil)
        let cgMaskImage = context.createCGImage(ciMaskImage2!, from: ciMaskImage2!.extent)
    
        let pixelData = cgMaskImage!.dataProvider!.data
        let data: UnsafePointer<UInt8> = CFDataGetBytePtr(pixelData)
    
        let alphaLevel: CGFloat = 0.0 //0.0 - 1.0 set higher to allow images with partially transparent eyes, like sunglasses.
    
        var possibleEyes: [coords] = []
    
        let frame = 10
        var detailLevel = 6
    
        let sizeX = Int((uiMaskImage?.size.width)!)
        let sizeY = Int((uiMaskImage?.size.height)!)
    
        var points: [(x: Int, y: Int)] = []
    
        var pointA_X = sizeX / 4
        var pointA_Y = sizeY / 4
        var pointB_X = sizeX / 4
        var pointB_Y = sizeY * 3 / 4
        var pointC_X = sizeX * 3 / 4
        var pointC_Y = sizeY / 4
        var pointD_X = sizeX * 3 / 4
        var pointD_Y = sizeY * 3 / 4
    
        var nextXsmaller = pointA_X / 2
        var nextYsmaller = pointA_Y / 2
    
        points.append((x: pointA_X, y: pointA_Y))
        points.append((x: pointB_X, y: pointB_Y))
        points.append((x: pointC_X, y: pointC_Y))
        points.append((x: pointD_X, y: pointD_Y))
    
        func transparentArea(_ x: Int, _ y: Int) -> Bool {
            let pos = CGPoint(x: x, y: y)
            let pixelInfo: Int = ((Int(uiMaskImage2!.size.width) * Int(pos.y)) + Int(pos.x)) * 4
            let a = CGFloat(data[pixelInfo+3]) / CGFloat(255.0)
            if a <= alphaLevel {
                return true
            } else {
                return false
            }
        }
    
        func createPoints(point: (x: Int, y: Int)) {
    
            pointA_X = point.x - nextXsmaller
            pointA_Y = point.y - nextYsmaller
    
            pointB_X = point.x - nextXsmaller
            pointB_Y = point.y + nextYsmaller
    
            pointC_X = point.x + nextXsmaller
            pointC_Y = point.y - nextYsmaller
    
            pointD_X = point.x + nextXsmaller
            pointD_Y = point.y + nextYsmaller
    
            points.append((x: pointA_X, y: pointA_Y))
            points.append((x: pointB_X, y: pointB_Y))
            points.append((x: pointC_X, y: pointC_Y))
            points.append((x: pointD_X, y: pointD_Y))
    
        }
    
        func checkSides(point: (x: Int, y: Int)) {
    
            var xNeg = (val: 0, end: false)
            var xPos = (val: 0, end: false)
            var yNeg = (val: 0, end: false)
            var yPos = (val: 0, end: false)
    
            if transparentArea(point.x, point.y) {
    
                xNeg.val = point.x
                xPos.val = point.x
                yNeg.val = point.y
                yPos.val = point.y
    
                while true {
    
                    if transparentArea(xNeg.val, point.y) {
                        xNeg.val -= 1
                        if xNeg.val <= frame {
                            break
                        }
                    } else {
                        xNeg.end = true
                    }
                    if transparentArea(xPos.val, point.y) {
                        xPos.val += 1
                        if xPos.val >= sizeX-frame {
                            break
                        }
                    } else {
                        xPos.end = true
                    }
    
                    if transparentArea(point.x, yNeg.val) {
                        yNeg.val -= 1
                        if yNeg.val <= frame {
                            break
                        }
                    } else {
                        yNeg.end = true
                    }
    
                    if transparentArea(point.x, yPos.val) {
                        yPos.val += 1
                        if yPos.val >= sizeY-frame {
                            break
                        }
                    } else {
                        yPos.end = true
                    }
    
                    if xNeg.end && xPos.end && yNeg.end && yPos.end {
    
                        let newEyes = coords(coord: (point.x, point.y), size: (xPos.val - xNeg.val) * (yPos.val - yNeg.val) )
    
                        possibleEyes.append(newEyes)
    
                        break
                    }
                }
            }
        }
    
        while detailLevel > 0 {
    
            print("Run: \(detailLevel)")
    
    
            for (index, point) in points.enumerated().reversed() {
    
                //checking if the point is inside of an transparent area
                checkSides(point: point)
    
                points.remove(at: index)
    
                if detailLevel > 1 {
                        createPoints(point: point)
                }
            }
            detailLevel -= 1
            nextXsmaller = nextXsmaller / 2
            nextYsmaller = nextYsmaller / 2
    
        }
    
        possibleEyes.sort { $0.coord.x > $1.coord.x }
    
        var rightEyes = possibleEyes[0...possibleEyes.count/2]
        var leftEyes = possibleEyes[possibleEyes.count/2..<possibleEyes.count]
    
        leftEyes.sort { $0.size > $1.size }
        rightEyes.sort { $0.size > $1.size }
    
        leftEyes = leftEyes.dropLast(Int(Double(leftEyes.count) * 0.01))
        rightEyes = rightEyes.dropLast(Int(Double(leftEyes.count) * 0.01))
    
        let sumXleft = ( leftEyes.reduce(0) { $0 + $1.coord.x} ) / leftEyes.count
        let sumYleft = ( leftEyes.reduce(0) { $0 + $1.coord.y} ) / leftEyes.count
    
        let sumXright = ( rightEyes.reduce(0) { $0 + $1.coord.x} ) / rightEyes.count
        let sumYright = ( rightEyes.reduce(0) { $0 + $1.coord.y} ) / rightEyes.count
    
    
        let maskOriginalWidth = CGFloat(exactly: sizeX)!
        let maskOriginalHeight = CGFloat(exactly: sizeY)!
        let maskOriginalLeftEyePossitionX = CGFloat(exactly: sumXleft)!
        let maskOriginalLeftEyePossitionY = CGFloat(exactly: sumYleft)!
        let maskOriginalEyeDistance = CGPointDistance(from: CGPoint(x: sumXright, y: sumYright), to: CGPoint(x: sumXleft, y: sumYleft))
    
        //This code assumes the image AND face orientation is always matching the same orientation!
        //The code needs to be adjusted for other cases using UIImage.Orientation to get the orientation and adjusts the coordinates accordingly.
        //CIDetector might also not detect faces which don't have the same orientation as the photo. Try to use CIDetectorImageOrientation to look for other orientations of no face has been detected.
        //Also you might want to use other orientation points and scale values (right eye, nose etc.) in case the left eye, and left to right eye distance is not available.
        //Also this code is very wordy, pretty sure it can be cut down to half the size and made simpler on many places.
    
        let uiImageFace = UIImage(named: "photo3.jpg")
        let ciImageFace = CIImage(image: uiImageFace!)
        let options = [CIDetectorAccuracy: CIDetectorAccuracyHigh]
        let faceDetector = CIDetector(ofType: CIDetectorTypeFace, context: nil, options: options)!
        let faces = faceDetector.features(in: ciImageFace!)
        if let face = faces.first as? CIFaceFeature {
    
    
            /*
            Getting the distances and angle based on the original photo
            */
            let faceAngle = face.faceAngle
            let rotationAngle = CGFloat(faceAngle * .pi / 180.0)
    
            //The distance in between the eyes of the original photo.
            let originalFaceEyeDistance = CGPointDistance(from: face.leftEyePosition, to: face.rightEyePosition)
    
    
            /*
            Adjusting the mask and its eye coordinates to fit the original photo.
            */
    
            //Setting the scale mask : original.
            let eyeDistanceScale = maskOriginalEyeDistance / originalFaceEyeDistance
    
            //The new dimensions of the mask.
            let newMaskWidth = maskOriginalWidth/eyeDistanceScale
            let newMaskHeight = maskOriginalHeight/eyeDistanceScale
    
            //The new mask coordinates of the left eye in relation to the original photo.
            let newMaskLeftEyePossitionX = maskOriginalLeftEyePossitionX / eyeDistanceScale
            let newMaskLeftEyePossitionY = maskOriginalLeftEyePossitionY / eyeDistanceScale
    
            /*
            Adjusting the size values to fit the desired final size on the screen.
            */
    
            //Using the width of the screen to calculate the new scale.
            let screenScale = uiImageFace!.size.width / view.frame.width
    
            //The new final dimensions of the mask
            let scaledToScreenMaskWidth = newMaskWidth / screenScale
            let scaledToScreenMaskHeight = newMaskHeight / screenScale
    
            //The new final dimensions of the photo.
            let scaledToScreenPhotoHeight = uiImageFace!.size.height / screenScale
            let scaledToScreenPhotoWidth = uiImageFace!.size.width / screenScale
    
            //The new eye coordinates of the photo.
            let scaledToScreenLeftEyeFacePositionX = face.leftEyePosition.x / screenScale
            let scaledToScreenLeftEyeFacePositionY = (uiImageFace!.size.height - face.leftEyePosition.y) / screenScale //reversing the y direction
    
            //The new eye to corner distance of the mask
            let scaledToScreenMaskLeftEyeX = newMaskLeftEyePossitionX / screenScale
            let scaledToScreenMaskLeftEyeY = newMaskLeftEyePossitionY / screenScale
    
            //The final coordinates for the mask
            let adjustedMaskLeftEyeX = scaledToScreenLeftEyeFacePositionX - scaledToScreenMaskLeftEyeX
            let adjustedMaskLeftEyeY = scaledToScreenLeftEyeFacePositionY - scaledToScreenMaskLeftEyeY
    
            /*
            Showing the image on the screen.
            */
    
            let baseImageView = UIImageView(image: uiImageFace!)
            //If x and y is not 0, the mask x and y need to be adjusted too.
            baseImageView.frame = CGRect(x: CGFloat(exactly: 0.0)!, y: CGFloat(exactly: 0.0)!, width: scaledToScreenPhotoWidth, height: scaledToScreenPhotoHeight)
            view.addSubview(baseImageView)
    
            let maskImageView = UIImageView(image: uiMaskImage!)
            maskImageView.frame = CGRect(x: adjustedMaskLeftEyeX, y: adjustedMaskLeftEyeY, width: scaledToScreenMaskWidth, height: scaledToScreenMaskHeight)
            maskImageView.transform = CGAffineTransform(rotationAngle: rotationAngle)
            view.addSubview(maskImageView)
            }
    
        }
    
        func CGPointDistanceSquared(from: CGPoint, to: CGPoint) -> CGFloat {
            return (from.x - to.x) * (from.x - to.x) + (from.y - to.y) * (from.y - to.y)
        }
    
        func CGPointDistance(from: CGPoint, to: CGPoint) -> CGFloat {
            return sqrt(CGPointDistanceSquared(from: from, to: to))
        }
    
    }