📂 Part of the Lance MLX collection on mlx-community.

Lance-3B-bf16 (MLX, image specialist)

MLX port of ByteDance Intelligent Creation Lab's Lance unified multimodal model — the image-specialist Lance_3B checkpoint, converted to bf16 for Apple Silicon. ~6.19 B LLM parameters in MoT (Mixture-of-Transformer-Experts) layout, plus the Qwen2.5-VL ViT (~669 M) and Lance's bundled Wan2.2 VAE (~705 M) for full image-task coverage.

Lance is ByteDance's 3B-active unified multimodal model (paper, code, HF original). This is not Lance/LanceDB, the columnar data format.

Status

🟢 Production-ready for image tasks on Apple Silicon as of 2026-05-21.

For video tasks (t2v, video_edit, x2t_video), see mlx-community/Lance-3B-Video-bf16. All six Lance task families are now validated end-to-end on Apple Silicon as of 2026-05-21.

The 48-channel Wan2.2 VAE is bundled here for convenience but also published standalone at mlx-community/Wan2.2-VAE-Lance-bf16 — both image_edit and the video pipelines need it.

Hardware envelope (memory_mode, 2026-06-02)

The lance-mlx source repo's memory_mode knob (auto / parallel / relay) brings bf16 image generation within reach of 8–16 GB Apple Silicon Macs:

relay produces byte-identical output to parallel (MD5-verified on real Lance-3B-bf16) — it sheds the UND tower after prefill and frees the GEN tower before VAE decode, so peak memory ≈ heaviest single phase rather than the sum of all three. Default auto resolves by mx.device_info()'s recommended working-set size with the split at ~18 GiB. The same envelope and modes apply to mlx-community/Lance-3B-Video-bf16 for video tasks.

Lossless streaming VAE decode (lossless_decode, 2026-06-05)

generate(..., lossless_decode=True) (default since 2026-06-05) uses a bit-identical streaming VAE decode that's lighter than the naive whole dec(z) in every measured config — via temporal causal-cache streaming (flat in frame count) and spatial halo-tile + crop (no blend ⇒ exact). 50-case bit-identity test in lance-mlx with a negative control verifies the guarantee. lossless_decode=False keeps the lossy trapezoidal-blend tiling from the prior path (~1.5–4.8 / 255 off the reference) for the one config where lossless exceeds 16 GB (768² video; lossless on 1024² image fits 16 GB cleanly at ~12.2 GB).

Full envelope and ri_phys-measured numbers (true OS-committed footprint, supersedes the older mx.get_peak_memory() quotes which under-report ~2×) in the source repo's LIMITS.md.

For VQA-only use cases on small Macs, the compressed mlx-community/Lance-3B-AWQ-INT4 variant is also available (3.3 GB LLM, 6-9× faster long-form decode; VQA only, not for image generation).

Quickstart

Install from the source repo (will be on PyPI in a follow-up release):

git clone https://github.com/xocialize/lance-mlx
cd lance-mlx && uv sync

Download this checkpoint:

from huggingface_hub import snapshot_download
weights = snapshot_download("mlx-community/Lance-3B-bf16")

Text-to-image

from lance_mlx.pipeline.t2i import TextToImagePipeline

pipe = TextToImagePipeline.from_pretrained(
    lance_weights_dir=weights,
    vae_safetensors=f"{weights}/vae.safetensors",
)
image = pipe.generate(
    "A photorealistic tabby cat holding up a colorful STOP sign on a sunlit street.",
    height=768, width=768, num_steps=30, cfg_scale=4.0, seed=42,
)
image.save("cat_with_stop.png")

Image editing

from lance_mlx.pipeline.image_edit import ImageEditPipeline

pipe = ImageEditPipeline.from_pretrained(
    lance_weights_dir=weights,
    vae_safetensors=f"{weights}/vae.safetensors",
)
edited = pipe.generate(
    input_image="portrait.jpg",
    instruction="Remove the hat from the painting.",
    height=768, width=768, num_steps=30, cfg_scale=4.0, seed=42,
)
edited.save("portrait_no_hat.png")

Image VQA / understanding

from lance_mlx.pipeline.understanding import UnderstandingPipeline
from PIL import Image

pipe = UnderstandingPipeline.from_pretrained(
    lance_weights_dir=weights,
    vit_safetensors=f"{weights}/vit.safetensors",
)
answer = pipe.generate(
    Image.open("license_plate.png"),
    "What is the license plate number visible in this image?",
    max_new_tokens=64, prompt_style="lance",
)
print(answer)

Performance (M5 Max 128 GB, macOS 26.2, MLX bf16)

KV cache scales with answer length: 1.7× speedup on a 5-token answer, 2.8× on a ~100-token answer.

Architecture

• Two expert towers (LLM_UND, LLM_GEN), each initialized from Qwen2.5-VL-3B-Instruct, with per-expert FFN, output projection, and QK-norm.
• Modality-deterministic routing: text + Qwen2.5-VL ViT semantic tokens → LLM_UND (autoregressive next-token); Wan2.2 VAE latent tokens → LLM_GEN (flow-matching velocity prediction). No learned gate.
• MaPE — modality-aware RoPE with per-modality temporal anchor (image-gen tokens re-anchored to t=1000).
• Wan2.2 3D causal VAE (16× spatial / 4× temporal compression, 48-channel latent — Lance bundles its own VAE; do NOT use the public 16-ch wan2.2_vae.safetensors).
• Bidirectional attention within latent block — causal_mask OR full_and_noise_mask per upstream data/data_utils.py::create_sparse_mask. Without this, the noisy-VAE position 0 of a 2304-token image grid can only see itself + text, producing blurry outputs across all prompts.
• Untied LM head.

Files in this repo

Provenance

Source: bytedance-research/Lance/Lance_3B/model.safetensors (1021 tensors, 6.185 B params). Conversion script: scripts/02_convert.py in the lance-mlx repo. The script:

• Loads original PyTorch safetensors, keeps F32 for normalization scales (per Phase 1b notes).
• Strips the language_model. prefix; the MLX LanceModel is the root, not nested.
• Splits the bundled ViT (vit_model.* keys) into a sibling vit.safetensors for parity with the Lance_3B distribution shape.
• Re-keys llm2vae.weight/bias and time_embedder.mlp.{0,2}.{weight,bias} to match scaffolded MLX modules.

Wan2.2 VAE source: bytedance-research/Lance/Wan2.2_VAE.pth → scripts/06_convert_wan_vae.py. Roundtrip MAD on a real photo at 768² is ~7/255 in u8 domain.

Limitations

• bf16 only. 4-bit + 8-bit quantization in progress. Naive INT4 has been observed to degrade the GEN expert (per Reza2kn/lance-quant's findings); quantization needs per-tower calibration.
• English + Chinese prompts work; other languages are training-distribution-limited (Qwen2.5-VL was trained primarily on en + zh).
• No streaming / batching API yet. Single-image, single-prompt generation only.
• CFG runs the LLM twice per step. A future KV-cache for the text + clean-ref prefix would save ~30% on image_edit.

Documented divergences from upstream PyTorch

• Outputs differ in low-level pixel detail from a CUDA reference run on the same seed/prompt (~1–5% per-pixel deviation expected from bf16 vs fp32, MLX RoPE vs PyTorch RoPE rounding, and a small number of intermediate-norm precision steps). Semantic correctness preserved across the 6 x2t_image oracle cases and all visually-verified t2i + image_edit prompts.
• x2t_image answers differ stylistically from Phase 0 oracle (PyTorch) — consistent across all 6 cases. Tracked as a Phase 5 parity follow-up; does not affect content correctness.

License

This MLX port: Apache 2.0.

Underlying weights:

• Lance: Apache 2.0 (ByteDance Intelligent Creation Lab).
• Wan2.2 VAE: Apache 2.0 (Alibaba).
• Qwen2.5-VL: Apache 2.0 (Alibaba).

See NOTICE for full attribution.

Citation

@article{fu2026lance,
  title={Lance: Unified Multimodal Modeling by Multi-Task Synergy},
  author={Fu, Fengyi and Huang, Mengqi and Wu, Shaojin and others},
  journal={arXiv preprint arXiv:2605.18678},
  year={2026}
}

Links

• MLX port code + phase notes: github.com/xocialize/lance-mlx
• Original PyTorch model: bytedance-research/Lance
• Wan2.2 VAE (standalone): mlx-community/Wan2.2-VAE-Lance-bf16
• Video specialist (alpha): mlx-community/Lance-3B-Video-bf16