mlx-community/VGGT-Omega-1B-512-bf16

bf16 MLX conversion of VGGT-Omega-1B-512 (vggt_omega_1b_512.pt from facebook/VGGT-Omega, revision 1041e80f) for the vggt_omega model in mlx-vlm. VGGT-Ω (Meta AI and Oxford VGG, CVPR 2026) reconstructs a scene from a sequence of images in one forward pass: a camera, a depth map and a depth confidence map for every frame. The 512 checkpoint is the one the authors recommend for in-the-wild use.

This is an unofficial derivative. It holds weights and metadata only; the inference code lives in mlx-vlm. The weights are under the FAIR Noncommercial Research License (copy in LICENSE.txt): noncommercial research use only.

What is in model.safetensors (2.29 GB, half the size of the float32 original):

• 1,332 bfloat16 tensors and the two float32 RoPE periods buffers (1.14 B parameters);
• the checkpoint's qkv.bias_mask buffers folded into the qkv biases (the aggregator's masks are all zero, so those projections have no bias, as in the reference), conv kernels in channel-last layout, and the unused mask_token dropped.

On load, mlx-vlm upcasts the camera and depth heads to float32, as the reference runs them. The aggregator runs in bf16 with a float32 residual stream, like the reference under CUDA autocast.

Use with mlx-vlm

Requires an mlx-vlm version that includes the vggt_omega model (added after 0.7.4).

import mlx.core as mx
from mlx_vlm import load
from mlx_vlm.models.vggt_omega.generate import VGGTOmegaPredictor, read_video

model, processor = load("mlx-community/VGGT-Omega-1B-512-bf16")
predictor = VGGTOmegaPredictor(model, processor)

# Paths, URLs, PIL images or RGB arrays; the first image is the reference frame.
out = predictor.infer(["frame_000.png", "frame_001.png", "frame_002.png"])
# or: out = predictor.infer(read_video("clip.mp4", fps=1))
mx.eval(out)

out["extrinsics"]    # (S, 3, 4) camera-from-world, OpenCV convention
out["intrinsics"]    # (S, 3, 3) in pixels of the model input
out["depth"]         # (S, H, W)
out["depth_conf"]    # (S, H, W)
out["world_points"]  # (S, H, W, 3) unprojected depth

Command line, writing all outputs to .npz and a colored point cloud:

python -m mlx_vlm.models.vggt_omega.generate \
    --model mlx-community/VGGT-Omega-1B-512-bf16 \
    --video clip.mp4 --fps 1 --output scene.npz --ply scene.ply

Accuracy and speed

Measured against the float32 PyTorch reference on frames of the upstream example videos (688x384 inputs). This bf16 model is on par with the reference's own bf16 autocast: on 8 frames, depth median relative error 8.6e-4 (reference autocast 1.2e-3), and camera rotations within 0.03 to 0.16 degrees across four scenes. In float32, the MLX port matches the reference to ~1e-5.

Full forward pass on an Apple M5 Max:

Citation

@inproceedings{wang2026vggtomega,
  title={VGGT-{$\Omega$}},
  author={Wang, Jianyuan and Chen, Minghao and Zhang, Shangzhan and Karaev, Nikita and Sch{\"o}nberger, Johannes and Labatut, Patrick and Bojanowski, Piotr and Novotny, David and Vedaldi, Andrea and Rupprecht, Christian},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2026}
}