mlx-community/sam-3d-body-dinov3-bf16
This model was converted to MLX format from facebook/sam-3d-body-dinov3
(revision 11aaa346) for the sam3d_body model in
mlx-vlm. Refer to the
original model card for more details on the model.
This is an unofficial derivative of Meta's SAM 3D Body and its Momentum Human Rig (MHR) body model. It holds weights and metadata only; the inference code lives in mlx-vlm. SAM 3D Body estimates a full-body human mesh (18,439 vertices, 127 joints, 70 keypoints) from a single image and a person box.
Contents
model.safetensors holds 1,203 tensors (2.80 GB): 615 bfloat16, 574 float32, 11 int32, 3 uint8.
Dtypes match Meta's release exactly. The DINOv3-H+ backbone, the bulk of the weights, is
bfloat16 as Meta ships it, including its RoPE periods. The decoders, heads, prompt encoder,
ray conditioning and MHR body model stay float32. Index tensors are int32, and MHR's
parameter masks are uint8. Nothing is cast to float16 or quantized.
| Component | Prefixes | Tensors | Dtypes |
|---|---|---|---|
| DINOv3-H+ backbone | backbone | 615 | bfloat16 |
| Hand-box detection | bbox_embed, hand_box_embedding, hand_cls_embed, hand_pe_layer | 10 | float32 |
| Body decoder | decoder | 194 | float32 |
| Hand decoder, heads, tokens and ray conditioning | decoder_hand, head_camera_hand, head_pose_hand, init_camera_hand, init_pose_hand, init_to_token_mhr_hand, keypoint3d_embedding_hand, keypoint3d_posemb_linear_hand, keypoint_embedding_hand, keypoint_feat_linear_hand, keypoint_posemb_linear_hand, prev_to_token_mhr_hand, ray_cond_emb_hand | 223 | float32 |
| Camera head | head_camera | 4 | float32 |
| MHR body model | head_pose.body_model | 25 | float32, int32, uint8 |
| Pose head | head_pose | 17 | float32, int32 |
| Tokens and keypoint embeddings | init_camera, init_pose, init_to_token_mhr, keypoint3d_embedding, keypoint3d_posemb_linear, keypoint_embedding, keypoint_feat_linear, keypoint_posemb_linear, prev_to_token_mhr | 18 | float32 |
| Prompt encoder | prompt_encoder, prompt_to_token | 94 | float32 |
| Ray conditioning | ray_cond_emb | 3 | float32 |
The checkpoint is complete: it includes the hand decoder and heads (*_hand), hand-box
detection (hand_cls_embed, bbox_embed, hand_pe_layer), the prompt encoder's point and
mask embeddings with mask_downscaling, the MHR expression basis
(head_pose.body_model.face_shape_vectors), mesh UVs and parameter limits. Keys are mlx-vlm
module paths in MLX layout: Conv2d weights are (O, H, W, I), and each backbone qkv
projection is split into q_proj, k_proj and v_proj. Only exact duplicates, empty
tensors and unused placeholders were left out (47 source tensors).
conversion.json lists every tensor's source and transform, every skipped tensor, and MHR's
joint and parameter names.
config.json follows mlx-vlm's SAM3DConfig, with Meta's values, including
image_size: [512, 512]. model_config.yaml is Meta's original configuration.
Use with mlx
pip install -U mlx-vlm
import numpy as np
from mlx_vlm.models.sam3d_body.generate import SAM3DPredictor
from mlx_vlm.utils import get_model_path, load_model
from PIL import Image
model = load_model(get_model_path("mlx-community/sam-3d-body-dinov3-bf16"), strict=False)
predictor = SAM3DPredictor(model, model.config)
image = np.asarray(Image.open("person.jpg").convert("RGB"))
result = predictor.predict(image, bbox=[x1, y1, x2, y2])
Pass strict=False because these tensors have no mlx-vlm module yet:
head_pose.body_model.parameter_limitshead_pose.body_model.pmihead_pose.body_model.texcoord_faceshead_pose.body_model.texcoordsinit_camera_handinit_pose_handkeypoint3d_embedding_handkeypoint_embedding_hand
mlx-vlm's Model.sanitize drops the hand branch and mask_downscaling.
SAM3DPredictor.from_pretrained can't load this checkpoint because it reads tensors through
NumPy, which has no bfloat16 type.
[!WARNING] mlx-vlm's current
sam3d_bodyinference code departs from Meta's reference in several places. Its default is a 512×384 anisotropic crop, where Meta uses a 512×512 isotropic one; this repository'sconfig.jsonsets 512×512. Its decoder FFN uses ReLU instead of GELU. Its CLIFF condition, camera translation and ray map use different box sizes. Expect its outputs to differ from Meta's. The weights here are faithful to Meta's release either way.
Conversion and validation
The converter read model.ckpt and assets/mhr_model.pt at revision 11aaa346c7204874a1cbafe3d39a979080b2c55a. It
renamed keys to mlx-vlm module paths, split the fused qkv weights, and merged each
effective bias (bias * bias_mask) into q_proj and v_proj. k_proj has no bias; its
masked bias is exactly zero. It also moved convolution channels last and narrowed int64
indices to int32. Every stored tensor was read back and matches its source bit for bit after
the layout transform is undone. With mlx-vlm 0.7.2 and MLX 0.32.2, the checkpoint loads
through mlx_vlm.utils.load_model: every model parameter matches its tensor in the file,
none is missing, and no shape differs. A forward pass on a synthetic image produced finite
outputs. This doesn't claim parity with Meta's PyTorch model; see the warning above.
License and credits
SAM 3D Body is by Meta Superintelligence Labs
(code,
paper). Its weights are under the
SAM License, a copy of which is in
LICENSE. The MHR body model is Meta's
Momentum Human Rig
(paper), under the Apache License 2.0 in
LICENSE-APACHE. The MHR tensors here are renamed and re-laid out as
conversion.json describes.
@article{yang2026sam3dbody,
title={SAM 3D Body: Robust Full-Body Human Mesh Recovery},
author={Yang, Xitong and Kukreja, Devansh and Pinkus, Don and Sagar, Anushka and Fan, Taosha
and Park, Jinhyung and Shin, Soyong and Cao, Jinkun and Liu, Jiawei and Ugrinovic,
Nicolas and Feiszli, Matt and Malik, Jitendra and Dollar, Piotr and Kitani, Kris},
journal={arXiv preprint arXiv:2602.15989},
year={2026}
}
@misc{MHR:2025,
title={MHR: Momentum Human Rig},
author={Ferguson, Aaron and others},
year={2025},
eprint={2511.15586},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2511.15586}
}