Cosmos-3 Tactile-MoT v2 — step-250 (interim) checkpoint
⚠️ Interim checkpoint. These are the weights at step 250 of a 3000-step full-SFT run, shared to bring up an evaluation pipeline in parallel with training — not a final model. The predicted video is coherent but still early/soft. Final weights will follow.
Joint video + tactile future prediction on the real 15.9 B NVIDIA Cosmos3-Nano backbone.
Task: past video + past tactile → future video + future tactile (30 Hz source / 10 Hz slow /
30 Hz fast; no action, no hand-pose).
What was trained (v2, full SFT)
Full SFT of the Cosmos3 generation pathway (~6.98 B, FSDP2/ZeRO-3) + a fresh tactile modality; the und/reasoner tower and the Wan VAE are frozen. On top of the v1 fixes (distinct per-view mRoPE, corrected VAE conditioning boundary, removed fast-path future leakage), v2 adds:
- Slow tactile = aligned 10 Hz sample (the 30 Hz sample at the tick time), not a within-block mean.
- Per-view identity embedding (head / wrist-L / wrist-R).
- English text conditioning via the official Cosmos3 prompt format (system + chat template +
duration/resolution +
<|vision_start|>); prompts are English translations of the vendor task. - Official UniPC flow sampler for sampling; trainer uses fp32-master + bf16 compute.
Both the slow expert (gen pathway + tactile head) and the 30 Hz fast expert (codec + FastTactileExpert)
are trained.
Files
model.safetensors— full model weights in bf16 (Cosmos3 backbone + tactile head + view embedding + tactile codec + fast expert + Wan VAE). ~32 GB.config.json— theJointModelConfigused (setsnapshotto your local Cosmos3-Nano base path).keys.json— the exact state-dict keys (for a load sanity check).
How to load & evaluate
Requires the model code and the Cosmos3-Nano base snapshot (for architecture/config + frozen weights the checkpoint overrides).
import torch
from safetensors.torch import load_file
from tacwam.cosmos_tactile.joint_model import JointModelConfig, TactileCosmosModel
cfg = JointModelConfig(snapshot="/path/to/Cosmos3-Nano", train_mode="gen_sft", use_lora=False,
master_dtype="bfloat16", slow_ctx=8, slow_hor=8, num_views=3, use_fast_expert=True)
m = TactileCosmosModel(cfg).to("cuda").eval()
sd = load_file("model.safetensors")
missing, unexpected = m.load_state_dict(sd, strict=False)
assert len(unexpected) == 0 and len(missing) < 8, (len(missing), len(unexpected)) # only tiny non-persistent buffers may miss
Or with the repo's eval/coherence helper (handles .safetensors, single-file .pt, and DCP dirs):
python scripts/cosmos3/smoke_coherence.py --ckpt <dir-with-model.safetensors> --sampler unipc
- Code:
github.com/haohww/tacWAM—tacwam/cosmos_tactile/,tacwam/schema_v03/,scripts/cosmos3/. - Data splits (which recordings to eval on):
haohw/tacwam-tactile-mot-v2-splits. - Base model: NVIDIA Cosmos3-Nano (OpenMDW-1.1). This is a derivative and inherits that license.
Caveats
- Interim (8% of the planned run); expect softness, especially in the close-up wrist views.
- Sampling is rectified-flow; use
--sampler unipc(the checkpoint's shipped scheduler). - The training corpus is egocentric human-subject vendor video; only weights are shared here.