GR00T N1.5 VLM — graft28
The vision-language path of NVIDIA GR00T N1.5-3B (backbone.eagle_model), made
runnable as a standalone VLM.
The released GR00T checkpoint ships only 12 of the 28 LLM decoder layers — the policy never needs the rest. This repo fills layers 12–27 from public Qwen3-1.7B so the language model is complete and can generate text.
This is not NVIDIA's VLM. Layers 12–27 are public Qwen3-1.7B weights, not GR00T's. Treat it as a research artifact for probing what GR00T's visual encoder and lower LLM layers represent, not as an official release.
What came from where
| Part | Source |
|---|---|
| Vision tower (SigLIP2, 27 layers, 1152-d, patch 14, 224px) | GR00T N1.5-3B checkpoint |
Connector mlp1.0 — a single Linear(1152 → 2048) | GR00T N1.5-3B checkpoint |
| LLM layers 0–11 | GR00T N1.5-3B checkpoint |
| LLM layers 12–27 | public Qwen3-1.7B |
Final norm, lm_head, embed_tokens | GR00T N1.5-3B checkpoint |
28 LLM layers, 2.15B parameters, bfloat16. tie_word_embeddings=True, so
lm_head.weight is not stored separately (it equals embed_tokens.weight in the
source checkpoint, verified).
Build script: build_graft28.py. Sources and verification: provenance.json.
All 760 saved tensors are byte-identical (torch.equal) to the tensor they came
from — no conversion, no re-quantization.
Usage — transformers
from transformers import AutoModel, AutoProcessor
D = "JisuHann/groot-vlm-graft28"
model = AutoModel.from_pretrained(D, trust_remote_code=True, torch_dtype="bfloat16").cuda().eval()
proc = AutoProcessor.from_pretrained(D, trust_remote_code=True)
Tested with transformers 4.51.3 / torch 2.7.1. Two gotchas:
radio_model.py(vendored GR00T code) callstorch.cuda.get_device_capability()at import time, so at least one GPU must be visible even for a CPU-only run.- The config pins
flash_attention_2, so CPU forward passes fail. For CPU, passattn_implementation="eager"for both the vision and text configs.
smoke_test.py loads the model from this repo and re-asks one sample from the
experiment below.
Image tokens
No pixel shuffle (use_pixel_shuffle: false): all 256 patches of a 224px tile go
through mlp1.0, giving 256 image tokens per tile (GR00T's own
preprocessor_config.json says tokens_per_tile: 256). A 640×480 frame at native
resolution tiles into 12 + thumbnail = 13 tiles = 3,328 tokens.
Usage — vLLM
vLLM registers Eagle2_5_VLForConditionalGeneration, but its implementation follows
NVIDIA's public Eagle2.5-VL, which differs from this checkpoint in three ways:
| this model (GR00T) | vLLM's built-in Eagle2.5-VL | |
|---|---|---|
| connector | one Linear(1152 → 2048) | LayerNorm + Linear + GELU + Linear |
| pixel shuffle | never | always, scale 0.5 (256 → 64 tokens/tile) |
| image normalization | SigLIP, mean/std 0.5 | ImageNet, 0.485… / 0.229… |
vllm_plugin/ is a small plugin that fixes exactly those three things and nothing
else. Setting downsample_ratio: 1.0 already makes vLLM's pixel_shuffle a no-op
(the two permutes cancel, every view() keeps its shape) and sets
image_seq_length to 256, so the plugin only overrides the connector shape and the
normalization constants.
pip install -e vllm_plugin/ # registers GrootEagle2_5_VLForConditionalGeneration
bash make_vllm_dir.sh ./graft28-vllm # weights symlinked, config from vllm/config.json
vllm serve ./graft28-vllm --served-model-name groot-vlm-graft28 \
--max-model-len 8192 --gpu-memory-utilization 0.30
vllm/config.json is this repo's config with three keys changed:
downsample_ratio: 1.0, min_dynamic_patch / max_dynamic_patch (vLLM reads the
_patch spelling; GR00T writes _tiles), and the architecture name the plugin
registers. The weights are untouched and shared with the transformers copy.
Where it came from
Built for a probe of what GR00T N1.5 "sees" in RoboCasa safe-navigation scenes: 50 samples (5 layouts × 5 obstacles × blocking/non-blocking), asked to name the obstacle on the floor, at policy resolution (224px) and native 640×480.
Sample result, egocentric view, native resolution: the obstacle is in the model's top-3 for 24/31 visible cases (77%, chance 16.7%), mean rank 2.5, AUROC 0.973 for separating visible from occluded; at 224px that drops to 45% top-3 and mean rank 6.4. Layers 12–27 being Qwen3 rather than GR00T is a confound for anything that depends on the upper LLM.
License
The GR00T-derived weights follow NVIDIA's license for GR00T-N1.5-3B; the vendored modeling code is NVIDIA's. Layers 12–27 come from Qwen3-1.7B (Apache-2.0). Check both before redistributing.