MP_Blendshapes
Model Description
MP_Blendshapes has been ported to pytorch from Google's mediapipe library using Liam Schoneveld's github repository. The inputs are 146/473 of mediapipe's FaceMeshV2 high density landmark model. The 52 blendshapes are similar to ARKit Face Blendshapes and loosely correspond to Facial Action Unit Coding System.
See the mediapipe model card for more details.
Model Details
- Model Type: MLP-Mixer (Keras)
- Framework: pytorch
Model Sources
- Repository: GitHub Repository
- Model Card: Mediapipe blendshape model card
- Paper: Attention mesh: High-fidelity face mesh prediction in real-time
Citation
If you use the mp_blendshapes model in your research or application, please cite the following paper:
Grishchenko, I., Ablavatski, A., Kartynnik, Y., Raveendran, K., & Grundmann, M. (2020). Attention mesh: High-fidelity face mesh prediction in real-time. arXiv preprint arXiv:2006.10962.
@article{grishchenko2020attention,
title={Attention mesh: High-fidelity face mesh prediction in real-time},
author={Grishchenko, Ivan and Ablavatski, Artsiom and Kartynnik, Yury and Raveendran, Karthik and Grundmann, Matthias},
journal={arXiv preprint arXiv:2006.10962},
year={2020}
}
Example Useage
import torch
import pandas as pd
from huggingface_hub import hf_hub_download
from feat.au_detectors.MP_Blendshapes.MP_Blendshapes_test import MediaPipeBlendshapesMLPMixer
from feat.utils import MP_BLENDSHAPE_MODEL_LANDMARKS_SUBSET, MP_BLENDSHAPE_NAMES
device = 'cpu'
# Load model and weights
blendshape_detector = MediaPipeBlendshapesMLPMixer()
model_path = hf_hub_download(repo_id="py-feat/mp_blendshapes", filename="face_blendshapes.pth")
blendshape_model_file = hf_hub_download(repo_id='py-feat/resmasknet', filename="ResMaskNet_Z_resmasking_dropout1_rot30.pth")
blendshape_checkpoint = torch.load(blendshape_model_file, map_location=device)["net"]
blendshape_detector.load_state_dict(blendshape_checkpoint)
blendshape_detector.eval()
blendshape_detector.to(device)
# Test model
face_image = "path/to/your/test_image.jpg" # Replace with your extracted face image that is [224, 224]
# Extract Landmarks
landmark_detector = torch.load('/Users/lukechang/Dropbox/py-feat/mediapipe/model/face_landmarks_detector_Nx3x256x256_onnx.pth', weights_only=False)
landmark_detector.eval()
landmark_detector.to(device)
landmark_results = landmark_detector(torch.tensor(face_image).to(device))
# Blendshape Classification
landmarks = landmark_results[0].reshape(1,478,3)[:,:,:2]
img_size = torch.tensor((face_image_width, face_image_height)).unsqueeze(0).unsqueeze(0)
landmarks = landmarks * img_size
blendshapes = blendshape_detector(landmarks)
blendshape_results = pd.Series(blendshape_results.squeeze().detach().numpy(), index=BLENDSHAPE_NAMES)