Cosmos-H2R-0918
A post-trained Cosmos3-Nano video world model for robot manipulation, exported as a consolidated Cosmos (Hugging Face safetensors) checkpoint.
Beyond plain video prediction, this checkpoint carries a 3D point-tracking head trained jointly with the flow-matching backbone, and explicit camera conditioning, so a generated rollout comes with per-frame 3D point trajectories and can be driven by a camera pose stream.
Model details
| Base model | nvidia/Cosmos3-Nano (Qwen3-VL-8B MoT backbone + diffusion expert) |
| Architecture | cosmos3_omni, unified_3d_mrope position embeddings |
| Parameters | ~15.2 B (language model + diffusion expert), plus the Qwen3-VL ViT tower |
| Weights | EMA weights, bf16 |
| Training iteration | 785 (final stage of a multi-stage curriculum) |
| Training resolution | 480p, 2 views, clips up to 157 frames |
| Packing budget | 32 000 tokens/sample |
What was trained in the final stage
- Data — DROID episodes cut into per-subtask clips with per-subtask language (~67 k clips, 157-frame window cap), 480p, two views.
- Objective — rectified-flow video loss, plus a 3D point-tracking loss (depth enabled, sequence cap 50) and a track-anchored cross-view consistency loss (weight 0.1).
- Conditioning — 1–5 clean past latents (1/5/9/13/17 context frames) and a 9D
framewise camera action with analytic Plücker-ray camera codes and RoCE rotations
(
camera_roce_mode=qk_vo) in every DiT layer. - Optimization — LR 2e-5, 200-step warmup on a 2000-step cosine cycle, 32× GH200.
Usage
This is a consolidated Cosmos checkpoint (config.json + sharded model*.safetensors +
checkpoint.json), the same layout nvidia/Cosmos3-Nano ships. Point the Cosmos inference
entry point at a local copy:
hf download rooty2020/Cosmos-H2R-0918 --local-dir ./Cosmos-H2R-0918
torchrun --nproc_per_node=<N> -m cosmos_framework.scripts.inference \
-i inputs.json \
-o outputs/ \
--checkpoint-path ./Cosmos-H2R-0918
The camera-conditioning and 3D-tracking heads are specific to this fine-tune; running it from
stock cosmos-predict without those modules will load the video backbone only.
Provenance
Exported from a PyTorch Distributed Checkpoint with
python -m cosmos_framework.scripts.export_model (the run's own config.yaml, EMA weights).
The ViT tower is taken from Qwen/Qwen3-VL-8B-Instruct at the revision pinned by the
Cosmos framework, as the training checkpoint does not carry it.
License
Derived from nvidia/Cosmos3-Nano and governed by the
NVIDIA Open Model License. Training data comes
from DROID; its terms apply to the data, and the usual
caveats about generated video (no guarantee of physical accuracy, not for safety-critical
control) apply here too.