Cosmos3-ours-RB-Y1
A Cosmos3-Nano multi-view video world model (Omni-4D) fine-tuned on RB-Y1 + WujiHand2 teleoperation with ego and exo views. Beyond video prediction it carries a 3D point-tracking head trained jointly with the backbone and explicit camera conditioning, so a rollout comes with per-frame 3D trajectories of the robot's arm and hand points.
Model details
| Base model | nvidia/Cosmos3-Nano (Qwen3-VL-8B MoT backbone + diffusion expert) |
| Architecture | cosmos3_omni, unified_3d_mrope, Omni-4D multi-view packing, + tracking_head, + camera_conditioner |
| Parameters | 15.22 B (incl. the Qwen3-VL ViT tower) |
| Weights | EMA weights, bf16 (tracking head and camera conditioner included) |
| Training iteration | 800 (of a 4000-step schedule) |
| Warm start | the Omni-4D stage-3 multi-view video+tracking model (0902_s3_full, iter 1068) |
Training
- Data — RB-Y1 WujiHand2 teleop, 100 episodes, 2 views (ego + exo) resized 1280×720 → 832×480. Per-episode captions.
- 3D point tracks — 768 points per frame (512 on the arms, 256 on the hands) in the robot base frame, projected into both views; the exo view is the anchor (exact calibration). Visibility is "finite, in front of the camera and in bounds" only — no depth ships, so self-occlusion is not modelled.
- Cameras — both cameras are static in the base frame (verified: zero pose variance on every episode), so the future camera stream is held constant.
- Windows — 4n+1 tiling plus a tail window so every frame is supervised: 741 windows (703 train / 38 val).
- Optimization — LR 1e-4, one linear schedule over 4000 steps.
Files
config.json, model-0000{1..7}-of-00007.safetensors, model.safetensors.index.json
checkpoint.json export provenance (use_ema_weights: true)
training_config.yaml the full training config of the source run
dcp/iter_000000800/ the same checkpoint as a raw DCP (weights only, net + net_ema)
Usage
A standard consolidated Cosmos checkpoint, the same layout nvidia/Cosmos3-Nano ships:
hf download rooty2020/Cosmos3-ours-RB-Y1 --local-dir ./Cosmos3-ours-RB-Y1 --exclude "dcp/*"
torchrun --nproc_per_node=<N> -m cosmos_framework.scripts.inference \
-i inputs.json -o outputs/ --checkpoint-path ./Cosmos3-ours-RB-Y1
The tracking head and camera conditioning are specific to Omni-4D; stock cosmos-predict
without those modules loads the video backbone only.
Provenance
Exported from a PyTorch Distributed Checkpoint (iter 800) with
python -m cosmos_framework.scripts.export_model --use-ema-weights. All 1069 net_ema tensors
of the training checkpoint are present in the export (tracking head and camera conditioner
included). The ViT tower is not in the training checkpoint and is taken from
Qwen/Qwen3-VL-8B-Instruct at the revision pinned by the Cosmos framework.
License
Derived from nvidia/Cosmos3-Nano and governed by the
NVIDIA Open Model License. The usual caveats
about generated video (no guarantee of physical accuracy, not for safety-critical control)
apply.