Face Capture Quality (FCQ)
Pose-aware face-capture-quality + head-pose model. Given a detected face crop it predicts
head pose (yaw/pitch/roll in degrees) and a pose-independent usability score, plus
blur/occlusion/exposure flags. Built to gate guided multi-angle ("360") face capture,
where off-the-shelf quality scores (e.g. Apple's VNDetectFaceCaptureQuality) wrongly
penalise non-frontal poses: a sharp, well-lit three-quarter turn should score high, a
blurry frontal face should score low.
- Backbone: MobileNetV3-Small (timm
mobilenetv3_small_100), ~1.9M params. - Output (single tensor, 7):
[yaw, pitch, roll, usable, blur, occluded, bad_exposure]. - Formats:
pytorch_model.bin,model.onnx(opset 17),FaceCaptureQuality.mlpackage(CoreML, fp16).
Input / preprocessing
RGB, NCHW, 160x160, values in [0, 1] (ImageNet normalisation is baked into the
graph; CoreML uses ImageType(scale=1/255)). The crop is a square, axis-aligned
bounding-box crop around the detected face (~0.2 margin), resized to 160 — i.e.
pose-preserving, NOT eye-aligned (a head-pose model must keep roll/yaw/pitch). The host
app should feed the same: detect face -> square bbox + ~20% margin -> resize 160.
How to use (ONNX Runtime)
The model expects a face crop — detect the face first, take a square bounding box with ~20% margin, then resize to 160×160 RGB in [0, 1], NCHW. ImageNet normalisation is baked into the graph, so no manual mean/std subtraction is needed.
import numpy as np, onnxruntime as ort
from PIL import Image
from huggingface_hub import hf_hub_download
onnx_path = hf_hub_download("shafi-afridi/face-capture-quality", "model.onnx")
# face_crop.jpg = an already-detected face (square bbox + ~20% margin)
face = Image.open("face_crop.jpg").convert("RGB").resize((160, 160))
x = (np.asarray(face, np.float32) / 255.0).transpose(2, 0, 1)[None] # (1, 3, 160, 160)
sess = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])
out = sess.run(None, {sess.get_inputs()[0].name: x})
yaw, pitch, roll, usable, blur, occluded, bad_exposure = np.asarray(out).reshape(-1)[:7]
print(f"pose°: yaw={yaw:.1f} pitch={pitch:.1f} roll={roll:.1f}")
print(f"usable={usable:.2f} blur={blur:.2f} occluded={occluded:.2f} bad_exposure={bad_exposure:.2f}")
# yaw/pitch/roll in degrees; usable + flags in [0, 1] (sigmoid). Higher `usable` = better capture.
On Apple devices prefer CoreML: download
FaceCaptureQuality.mlpackageand feed a Vision/Core MLImageTypewithscale=1/255(normalisation is already inside the graph).
Intended use
On-device capture guidance (CoreML / ONNX). NOT a recognition or identity model, and not an authentication or surveillance tool.
Results
| Metric | Value |
|---|---|
| Head-pose MAE — yaw / pitch / roll (deg), AFLW2000-3D | 14.78 / 7.61 / 6.64 |
| Head-pose MAE — mean (deg), AFLW2000-3D (n=1969) | 9.68 |
Quality — Spearman ρ (predicted usable vs known degradation), n=1200 | 0.790 |
| Quality — ERC/EDC verification error, 0% → 20% discard (LFW) | 0.0696 → 0.0490 |
| CoreML size | 3.8 MB |
Numbers are produced by eval.py; see the GitHub repo to reproduce. The ERC injects
controlled degradations into LFW to create a quality range, then shows that discarding the
lowest-usable faces lowers verification error.
Training data & licence
Head pose is trained on 300W-LP (real ground-truth pose Pose_Para; pose-diverse via
synthetic large-pose renders) and evaluated on AFLW2000-3D — the standard head-pose
protocol, cropped identically (square box around the 68 landmarks, no detector). The
quality head is trained on synthetic degradations (controlled blur / JPEG / noise /
exposure / occlusion with known severity) applied to 300W-LP + FFHQ faces — FFHQ supplies
real, high-resolution clean faces so the model calibrates "crisp real face = high usable" —
plus real occlusion/blur/framing negatives from WIDER FACE.
300W-LP, FFHQ, WIDER FACE and AFLW2000-3D are research / non-commercial datasets, so the v1
weights are released cc-by-nc-4.0 — research use only. A commercially-licensed version
would require retraining on cleared data (same architecture). The repository CODE is Apache-2.0.
Limitations & bias
- Quality targets are partly synthetic (controlled degradations), supplemented by real WIDER negatives; pose labels are 300W-LP's 3DMM-fit GT (synthetic large-pose renders).
- Face models can perform unevenly across demographics, skin tones, and lighting; the training data is not demographically balanced. Evaluate on your own population before use.
- Pose accuracy degrades at extreme angles (|yaw| > 90 deg) and under heavy occlusion.
- fp16 CoreML numerics are verified on-device (iPhone), not on the training box.
Files
pytorch_model.bin— training checkpoint (state dict).model.onnx— opset 17, for servers / cross-platform.FaceCaptureQuality.mlpackage— CoreML (fp16), for iOS. Input: RGB image (scale=1/255).config.json— input/output spec.
Attribution
- Datasets: 300W-LP and AFLW2000-3D (Zhu et al., "Face Alignment Across Large Poses: A 3D Solution", CVPR 2016; academic / non-commercial), FFHQ (NVIDIA; CC BY-NC-SA 4.0), WIDER FACE (Yang et al., CVPR 2016; CC BY-NC-ND 4.0).
- Verification embeddings for the ERC eval only: InsightFace (buffalo_l).
- Code: https://github.com/shafi-afridi/face-capture-quality
- Demo Space: https://huggingface.co/spaces/shafi-afridi/face-capture-quality-demo