pythonface-recognitiondlib

Use of "truncated" large 68-points model for face encoding


face_recognition version: 1.3.0 Python version: 3.92 Operating System: Windows 10 64

Description I've tried to solve "masked" face recognition and identification. An idea was to compare only some parts of faces (eyes, brows, part of nose) using a truncated face landmarks set.. So, I defined a new dlib-class, which returns “original” landmarks for the aforementioned face parts and (0. 0) – for the rest of 68 points of large model. Also I defined a new method face_encodings_masked() to work with new class:

What I Did

import face_recognition
from face_recognition import api as frapi
import numpy as np
import dlib



class Full_object_detection_masked(dlib.full_object_detection):
    def part(self, idx:int):
        if idx in range(2, 15) or idx in range(48, 68):
            return (0, 0)
        return super().part(idx)

    def parts(self):
        lst = dlib.points()
        for idx in range(0, 2):
            old_x = super().part(idx).x
            old_y = super().part(idx).y
            lst.insert(idx, dlib.point(old_x, old_y))
        for idx in range(2, 15):
            lst.insert(idx, dlib.point(0, 0))
        for idx in range(15, 29):
            old_x = super().part(idx).x
            old_y = super().part(idx).y
            lst.insert(idx, dlib.point(old_x, old_y))
        for idx in range(29, 36):
            lst.insert(idx, dlib.point(0, 0))
        for idx in range(36, 48):
            old_x = super().part(idx).x
            old_y = super().part(idx).y
            lst.insert(idx, dlib.point(old_x, old_y))    
        for idx in range(48, 68):
            lst.insert(idx, dlib.point(0, 0))
        return lst
    
    
def face_encodings_masked(face_image, known_face_locations=None, num_jitters=1, model="large"):
    """
    Given an image, return the 128-dimension face encoding for each face in the image.

    :param face_image: The image that contains one or more faces
    :param known_face_locations: Optional - the bounding boxes of each face if you already know them.
    :param num_jitters: How many times to re-sample the face when calculating encoding. Higher is more accurate, but slower (i.e. 100 is 100x slower)
    :param model: Optional - which model to use. "large" (default) or "small" which only returns 5 points but is faster.
    :return: A list of 128-dimensional face encodings (one for each face in the image)
    """
    raw_landmarks = frapi._raw_face_landmarks(face_image,
                                        known_face_locations,
                                        model)
    masked_raw_landmarks = []
    for lm in raw_landmarks:
        masked_raw_landmarks.append(Full_object_detection_masked(lm.rect, lm.parts()))
    return [np.array(
        frapi.face_encoder.compute_face_descriptor(
            face_image,
            raw_landmark_set,
            num_jitters)) for raw_landmark_set in masked_raw_landmarks]


image = face_recognition.load_image_file("1.jpg")
"""calling modified face_encodings with truncated face landmarks.."""
enc_masked = face_encodings_masked(image, known_face_locations=None, num_jitters=1, model="large")
"""calling standard face_encodings with normal face landmarks.."""
enc_full = frapi.face_encodings(image, known_face_locations=None, num_jitters=1, model="large")
print("Difference between face encodings with partial (masked) and full landmarks:")
print(enc_full[0] - enc_masked[0])

I did hoped to see a difference .. But there is no difference:

Difference between face encodings with partial (masked) and full landmarks:

[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0.]

I've tried to explore source code for face_recognition and dlib modules but found that every appropriate method needs face landmarks as an argument... So, if the face landmarks are different - the face encodings should be different, right? But it isn't. Any ideas?

UPDATE (03-2024). A few time ago I've passed from face_recognition / dlib to insightface library. The last one works pretty different and generates 512d face vectors instead of 128d by dlib and so on.


Solution

  • I decided to go another way and use a virtual mask (5 different types) application on a basic face and then - storing additional encodings for the "masked"faces too. .. Well, it works. Not 100% efficiently but .. does. How it works - at Github