androidandroid-studiotensorflowaugmented-realityface-detection

Is it possible to use face_landmark.tflite from Mediapipe to generate face mesh in Android independently?


I'm working on a face tracking app (Android studio / Java) and I need to identify face landmarks. I'm interested using Mediapipe face mesh model. The problem is: I use Windows OS, and Mediapipe is not working on Windows OS.

I have very basic knowledge in Tensorflow, Can anybody explain to me how can i use Mediapipe's face_landmark.tflite model to detect faces in images and generate face mesh in Android studio with Java independently without the whole Mediapipe framework?


Solution

  • You can try look into my notebook below for usage example in python. This only needs tflite model and does not require Mediapipe installation.

    https://nbviewer.jupyter.org/github/quickgrid/CodeLab/blob/master/tensorflow/MediaPipe_Face_Mesh_TFLite_Python_Direct_Inference.ipynb

    This is the output image,

    enter image description here

    This should give a starting point to use android tflite interpreter to get face landmarks and draw them. It will require a face detector such as blazeface to output the face bounding box first.

    As I have not implemented this model in android yet I cannot say what else may be needed. Further details may be found in mediapipe face mesh codes. The notebook is based on this code,

    MediaPipe TensorflowLite Iris Model

    https://github.com/shortcipher3/stackoverflow/blob/master/mediapipe_iris_2d_landmarks.ipynb

    Further references,

    https://github.com/google/mediapipe/blob/master/mediapipe/modules/face_landmark/face_landmark.tflite

    https://google.github.io/mediapipe/solutions/face_mesh

    Model card with input, output details,

    https://drive.google.com/file/d/1QvwWNfFoweGVjsXF3DXzcrCnz-mx-Lha/view

    Alternate option

    Android ML Kit also has face landmarks with very good documentation and code example.

    https://developers.google.com/ml-kit/vision/face-detection

    https://developers.google.com/android/reference/com/google/mlkit/vision/face/package-summary