Cosmos-H-Surgical

Model Overview

Cosmos-H-Surgical v0.3.1 provides two checkpoints built on NVIDIA Cosmos3-Nano:

The base Cosmos-H-Surgical model remains the default when --checkpoint-path is omitted. The DMD2 checkpoint is an explicitly selected, student-only safetensors export; it does not contain optimizer, scheduler, teacher, fake-score, trainer, or other DCP training state.

The supported workflows are:

• Video prediction: generate the next 92 surgical video frames from a starting image and a structured text description.
• Simulation-to-real transfer: convert a blur, depth, segmentation, or edge control video into a photorealistic surgical video.
• Four-step transfer: run the same four transfer modalities with the DMD2 student and its fixed SDE schedule.

Each transfer input specification activates exactly one control modality.

This model is ready for commercial and non-commercial use.

License

The model weights and accompanying Cosmos-H-Surgical source code are provided under OpenMDW-1.1. The official license source is available from the OpenMDW repository. Third-party software and upstream model components remain subject to their own license terms.

Intended Use

Expected users include medical researchers, surgical robotics developers, AI developers, and healthcare institutions working on synthetic surgical video, simulation-to-real transfer, physical AI, and downstream robotics research.

These models are not intended for clinical diagnosis or autonomous clinical decision-making. Validate model behavior with use-case-specific data before deployment. Users must have appropriate rights and permissions for all input images and videos.

Architecture and Output

The DMD2 export contains seven safetensors shards, 1,092 indexed tensor keys, 15,171,557,568 parameters, and 288 LoRA tensor keys.

Installation

Clone the v0.3.1 source release and install one CUDA dependency group:

# CUDA 13, recommended
uv sync --group cu130

# CUDA 12.8
uv sync --group cu128

source .venv/bin/activate

Inference

The default base checkpoint can be run without an explicit checkpoint name:

torchrun --nproc-per-node=8 \
  -m cosmos_h_surgical infer \
  --parallelism-preset=latency \
  --dp-shard-size=1 \
  -i inputs/predict/surgical_predict.jsonl \
  -o outputs/cosmos-h-surgical/predict \
  --no-guardrails \
  --seed=0

Select the DMD2 student explicitly for four-step transfer:

torchrun --nproc-per-node=1 \
  -m cosmos_h_surgical infer \
  --checkpoint-path Cosmos-H-Surgical-Transfer-DMD2-4Step \
  --parallelism-preset=latency \
  --dp-shard-size=1 \
  --no-use-torch-compile \
  -i /path/to/dmd2_transfer_input.json \
  -o outputs/cosmos-h-surgical/dmd2-transfer \
  --no-guardrails \
  --seed=0

The DMD2 input uses model_mode: "video2video", exactly one active control, and both guidance: 1.0 and control_guidance: 1.0. The full input schema and examples are in the inference documentation.

DMD2 Distillation Recipe

The v0.3.1 source release includes a public DMD2 training launcher, export wrapper, and tutorial. The public configuration is a recommended starting recipe rather than an exact reconstruction of the internal training run that produced this checkpoint.

The default control sampling configuration is:

control_modalities={
    "edge": 1.0,
    "blur": 1.0,
    "depth": 1.0,
    "seg": 1.0,
}

See the DMD2 distillation tutorial for data preparation, single-node and multi-node launch, resume, export, and inference instructions.

Validation

Transfer FVD

The released DMD2 student was evaluated on 1,010 videos per control modality at 832 x 480, 93 frames, and 16 FPS:

One-H100 Latency

Latency was measured on one H100 80 GB GPU using one edge-transfer sample at 832 x 480, 93 frames, and 16 FPS, with seed 0, BF16, compilation disabled, one warmup, and five measured repetitions. Values are mean +/- sample standard deviation in seconds. CUDA stage times are synchronized; generation and complete-batch times are wall-clock measurements.

The DMD2 run executed four denoiser calls. The base run executed 100 calls (50 steps with two classifier-free guidance evaluations per step). The DMD2 run did not execute an additional CFG branch.

The release validation also checks all four public transfer controls, verifies the exported checkpoint inventory and checksums, and compares a fixed-seed exported-student output with the source DCP student output.

Training and Evaluation Data

Cosmos-H-Surgical was trained using GraSP real surgical video and synthetic surgical video data. The synthetic collection contains 12,600 laparoscopic cholecystectomy videos generated through the Cosmos-H-Surgical transfer pipeline. GraSP contributes 15,043 robot-assisted radical prostatectomy videos. The combined collection uses a 90% training and 10% validation split.

The training collection contains approximately 24,879 videos and the held-out validation collection approximately 2,764 videos. Total video duration is less than 10,000 hours.

Hardware and Software

• Linux and BF16 inference
• NVIDIA Hopper used for release validation
• CUDA 13 recommended; CUDA 12.8 supported by the release environment
• Single H100 80 GB used for the DMD2 latency and transfer smoke validation

Other GPU architectures, precisions, and parallelism settings require separate validation.

References

1. NVIDIA Cosmos: World Foundation Models for Physical AI
2. NVIDIA Cosmos3-Nano
3. Cosmos-H-Surgical source repository

Please report security vulnerabilities through the NVIDIA Product Security process.