swiftimage-processingface-detectionapple-vision

Vision framework to detect face looking up/down


i’m using Vision framework to detect face orientation. VNFaceObservation has roll and yaw properties but unfortunately no pitch. How can i calculate pitch value? i need to check if a person looking up or down.

Someone in another forums suggested me to use distance between eye pupils and eye brows but i don't get good result especially when person change distance to the screen as this difference vary. I tried to divide the difference with face boundingBox height but this value is also dynamic, box surrounding the face becomes smaller when user looking up.

People with very different facial features is another story. Has anyone implemented a feature like this with Vision? Thank you for your help.

   var origins: [CGPoint] = []
  //  origin based on left and right pupil
  if let point = result.landmarks?.leftPupil?.normalizedPoints.first {
    let origin = landmark(point: point, to: result.boundingBox)
    origins.append(origin)
  }
  if let point = result.landmarks?.rightPupil?.normalizedPoints.first {
    let origin = landmark(point: point, to: result.boundingBox)
    origins.append(origin)
  }
 
  // Calculate the average y coordinate of the origins.
  let avgY = origins.map { $0.y }.reduce(0.0, +) / CGFloat(origins.count)
  
  // get eyebrow locations
  var eyebrowOrigins: [CGPoint] = []
    var heights: [CGPoint] = []
    
    
  if let point = result.landmarks?.leftEyebrow?.normalizedPoints.first {
    let origin = landmark(point: point, to: result.boundingBox)
    eyebrowOrigins.append(origin)
  }
  if let point = result.landmarks?.rightEyebrow?.normalizedPoints.first {
    let origin = landmark(point: point, to: result.boundingBox)
    eyebrowOrigins.append(origin)
  }
  
  // Calculate the average y coordinate of the eye brows.
  let eyebrowAvgY = eyebrowOrigins.map { $0.y }.reduce(0.0, +) / CGFloat(origins.count)
  
  // compare pupils location to eye brows
  
  var focusY: CGFloat = 0
    let diff = avgY - eyebrowAvgY

  // approximate face height inside the bounding box
    let faceHeight = result.boundingBox.height

Solution

  • Pitch is now a property of VNFaceObservation