VLA-JEPA - LIBERO (GGUF for vla.cpp)
GGUF conversion of lerobot/VLA-JEPA-LIBERO
for inference with vla.cpp, a
lightweight C++ inference engine for Vision-Language-Action models built on top of
llama.cpp.
VLA-JEPA is a 3B VLA built on a Qwen3-VL-2B-Instruct vision-language backbone
- a 24-layer ViT (1024d, patch 16, temporal-patch 2, spatial-merge ÷2, deepstack
features fused from layers 5/11/17, 256 px) feeding a 28-layer Qwen3 LM (2048d,
16 query / 8 KV heads × 128, RoPE θ=5e6) - coupled to a DiT-B flow-matching
action head (16 transformer blocks, 12 heads × 64, inner 768d, cross-attending
the 2048d LM stream, 1024d output) that denoises a 7-step action chunk over 4
flow-matching steps from 32 learned future tokens (
action_dim7,state_dim8).
The upstream checkpoint also carries a V-JEPA world-model predictor, but it is not on the action-inference path - so the conversion drops the world model and keeps only the Qwen3-VL backbone + DiT action head. The vision tower is baked into the combined GGUF, so no separate mmproj file is needed.
Files
| File | Size | Description |
|---|---|---|
vla-jepa.gguf | 4.25 GiB | Combined VLA model - Qwen3-VL-2B backbone (ViT + deepstack mergers + Qwen3 LM) + DiT-B flow-matching action head + arch config, BF16. World-model predictor dropped. |
Required normalisation stats (not bundled here)
VLA-JEPA un-normalises with LeRobot processor stats, not a
dataset_statistics.json. The client reads two safetensors from the dir passed to
--stats-json:
| File | Role |
|---|---|
policy_preprocessor_step_3_normalizer_processor.safetensors | observation.state MEAN_STD normaliser |
policy_postprocessor_step_2_unnormalizer_processor.safetensors | action MIN_MAX un-normaliser (+ gripper snap/binarise) |
Both ship in the upstream
lerobot/VLA-JEPA-LIBERO repo.
Copy them next to vla-jepa.gguf (then --stats-json . works) or point
--stats-json at a local copy of the upstream checkpoint.
Usage
Build vla-server from the vla.cpp repo,
then:
# Terminal 1 - serve (use the CUDA build for inference). No mmproj argument.
VLA_JEPA_BF16_WEIGHTS=1 ./build-cuda/vla-server --bind tcp://*:5566 \
vla-jepa.gguf
# Terminal 2 - drive a LIBERO episode (inside the LIBERO uv venv)
python eval/client/run_sim_client_direct.py \
--arch vla_jepa \
--task libero_object --task-id 0 --n-episodes 10 \
--n-action-steps 7 \
--stats-json . \
--vla-addr tcp://localhost:5566
Notes:
vla_jeparuns at 256 px with a 7-step action chunk (--n-action-steps 7) and proprio state dim 8.- Set
VLA_JEPA_BF16_WEIGHTS=1to keep BF16 matmuls (default upcasts weights to F32).VLA_NUM_STEPS=<n>overrides the 4 flow-matching steps. --stats-json <dir>is required - point it at the dir holding the twopolicy_{pre,post}processorsafetensors (see above).- The tokenizer/processor auto-loads from
Qwen/Qwen3-VL-2B-Instructon the Hub (the client expands it with the 28<|action_i|>+<|embodied_action|>tokens). Pass--tokenizer /path/to/Qwen3-VL-2B-Instructto load it offline.
Benchmark
Smoke test - LIBERO sweep, vla-server + run_sim_client_direct.py:
| Hardware | Success rate | client/step | server/call | Peak mem |
|---|---|---|---|---|
| - | 100.0% (10 tasks × 10 episodes = 100/100) | - | - | - |
VRAM and latency have not been measured yet; they will be filled in once profiled on reference hardware.
License
The upstream lerobot/VLA-JEPA-LIBERO
repo declares no explicit license. Review the upstream terms and the licenses
of its components - Qwen3-VL-2B (Apache-2.0) and V-JEPA (Meta research license) -
before use. The vla.cpp conversion tooling and inference engine are
Apache-2.0-licensed.