Cosmos3-Edge → RoboTwin 2.0 aloha-agilex action policy

A supervised fine-tune of nvidia/Cosmos3-Edge (3.86B, mixture-of-transformers) into a bimanual manipulation policy for RoboTwin 2.0, aloha-agilex embodiment.

Released by General Instinct.

What it does

Takes one observation and a natural-language instruction, returns a chunk of 16 absolute joint targets for a dual-arm ALOHA-AGILEX robot.

action_normalisation: none is deliberate. Shipping no statistics file means there is no train/serve normaliser that can silently disagree, which is one of the easier ways to get a policy that moves plausibly and scores badly with nothing reporting an error.

How the fine-tune was done

Base. nvidia/Cosmos3-Edge — the general Edge checkpoint, not Cosmos3-Edge-Policy-DROID. Starting from the DROID policy would have meant adapting a single-arm 8-D Franka head to a bimanual 14-D robot; the action heads were instead initialised fresh (keys_to_skip_loading = ["action2llm", "llm2action", "action_modality_embed"], which is what every shipped Cosmos action recipe does) and a previously unused embodiment-domain slot was registered for this robot, so no other embodiment's conditioning was disturbed.

Data. The official RoboTwin 2.0 release archives — the aloha-agilex clean and randomized demonstration sets — converted to LeRobot v3. Nothing was self-collected, and no demonstrations were held out; the model trains on all of them. Training instructions come from the seen instruction pool only.

Observation. RoboTwin exposes three cameras; Cosmos3-Edge consumes a single video stream, so the three views are composed into one frozen mosaic — head camera at full width on top, the two wrist views side by side beneath it. Two details are load-bearing and worth repeating if you build on this:

• Decode JPEG/PNG with cv2.imdecode. RoboTwin stores cv2.imencode output of an RGB array; PIL returns RGB and the resulting channel swap is invisible — the policy still works, just worse, with no error anywhere.
• Resize the wrist views with cv2.INTER_LINEAR. The 2× downscale makes bilinear degenerate to a box average; PIL's BILINEAR and antialias=True use a wider support filter and differ by tens of LSB, which is a real train/serve distribution shift.

The training and serving mosaics were verified byte-identical before training started.

Only the head, left-wrist and right-wrist RGB streams are used. Depth, segmentation, pointclouds, camera intrinsics/extrinsics and every other privileged simulator field are available in RoboTwin's observation dict and are all deliberately excluded — a heavily vision-pretrained model would otherwise get an unfair channel.

Recipe. Standard action-SFT with cosmos-framework: bf16, HSDP fully sharded across one 8×H100 node, selective activation checkpointing on the attention kernels, AdamW at a constant 5e-5 after a warm-up, gradient clipping at 1.0. About a day of wall clock.

The iteration count was fixed in advance from the dataset size rather than tuned — the window count divided by the global batch, targeting roughly four epochs, which is the middle of the range comparable work uses. The evaluated checkpoint is the final one at that pre-registered step count: no best-checkpoint selection, no early stopping, no validation split. Selecting a checkpoint on the same episodes you later report is an easy way to publish a number that does not reproduce.

Denoise steps are chosen at inference time

Cosmos3-Edge's action head is a flow-matching model, so the number of denoising steps is not baked into the weights — the training config carries no step count at all. One checkpoint integrates the sampling ODE with however many steps you ask for, and this repository contains exactly one set of weights that serves any of them.

Practically: --num-steps on the server changes cost and fidelity without touching the model. With guidance = 1.0 the network is evaluated once per step, so the step count is also the number of function evaluations; other guidance scales evaluate it twice per step.

Serving it

Requires cosmos-framework plus a RoboTwin embodiment-domain registration and a policy adapter for the mosaic and action contract described above; these are not included here.

python -m cosmos_framework.scripts.action_policy_server_robotwin \
    --domain-name robotwin_aloha_agilex \
    --checkpoint-path <this repo> \
    --action-chunk-size 16 --guidance 1.0 --num-steps 30

The server rebuilds the training-time input transform by loading the training config recorded in checkpoint.json, so it needs the environment variables that config interpolates. If it cannot load it, the server falls back to a default transform and will serve a prompt format the model never saw — a failure that shows up only as mediocre accuracy. Check the server reports the training-time prompt format on startup and treat any warning about an unreachable training config as fatal rather than cosmetic.

Licence

Derivative of nvidia/Cosmos3-Edge under OpenMDW-1.1, whose grant, notice and outputs clauses permit redistributing derivative weights. This is our reading of the terms, not legal advice.

Citation

@misc{instinctwm_cosmos3edge_robotwin_2026,
  title  = {Cosmos3-Edge fine-tuned as a RoboTwin 2.0 aloha-agilex action policy},
  author = {General Instinct},
  year   = {2026},
  url    = {https://huggingface.co/GM717/cosmos3-edge-robotwin-aloha-agilex}
}