Cosmos-H-Surgical

Model Overview

Cosmos-H-Surgical v0.3.0 is a unified surgical video world model fine-tuned from NVIDIA Cosmos3-Nano. One checkpoint supports two workflows:

Video prediction: generate the next 92 surgical video frames from a starting image and a structured text description.
Simulation-to-real transfer: convert a blur, depth, segmentation, or edge control video into a photorealistic surgical video.

Prediction requires a starting image. Each transfer input specification activates exactly one control type. The four controls may be evaluated for the same source case as separate specifications.

This model is ready for commercial and non-commercial use.

The earlier Cosmos 2.5 release remains available from the cosmos-2.5 branch.

License

The v0.3.0 model weights and accompanying Cosmos-H-Surgical source code are provided under OpenMDW-1.1. The official license source is available from the OpenMDW repository. Third-party software and upstream model components remain subject to their own license terms.

Intended Use

Expected users include medical researchers, surgical robotics developers, AI developers, and healthcare institutions working on:

Synthetic surgical video generation for downstream robotics research
Simulation-to-real transfer for surgical scenes
Physical AI development and evaluation

This model is not intended for clinical diagnosis or autonomous clinical decision-making. Validate model behavior with use-case-specific data before deployment. Users must have appropriate rights and permissions for all input images and videos.

Model Architecture

Property	Value
Base model	NVIDIA Cosmos3-Nano
Architecture	Diffusion transformer with unified language and video pathways
Parameters	Approximately 16B
Precision	BF16
Resolution	480P, validated at 832 x 480
Frame rate	16 FPS
Output length	93 frames
Capabilities	Image-to-video prediction and controlled video-to-video transfer

Inputs

Inference requires an outer JSON or JSONL specification. Its prompt field is a serialized structured JSON object describing subjects, scene, actions, and temporal evolution. See the inference documentation for the complete schema.

Prediction

One starting JPG, PNG, JPEG, or WebP image
One structured text prompt
model_mode: "image2video"

Transfer

One control modality per JSON specification
Blur and edge controls use a source RGB video
Depth and segmentation controls use their corresponding control videos
model_mode: "video2video"

The validated release settings are 480P, 16 FPS, 93 frames, guidance 6 and shift 5 for prediction, and guidance 1, control guidance 1 and shift 5 for transfer.

Outputs

The model produces an MP4 surgical video at 832 x 480, 16 FPS, and 93 frames. Prediction includes the starting frame followed by 92 generated frames. Transfer preserves the temporal and spatial structure of its control input while synthesizing a photorealistic appearance.

Installation

Clone the Cosmos-H-Surgical v0.3.0 source release and install one CUDA group:

# CUDA 13, recommended
uv sync --group cu130

# CUDA 12.8
uv sync --group cu128

source .venv/bin/activate

Inference

The Cosmos-H-Surgical checkpoint is selected automatically when --checkpoint-path is omitted:

torchrun --nproc-per-node=8 \
  -m cosmos_h_surgical infer \
  --parallelism-preset=latency \
  --dp-shard-size=1 \
  -i inputs/predict/surgical_predict.jsonl \
  -o outputs/cosmos-h-surgical/predict \
  --no-guardrails \
  --seed=0

The explicit equivalent is:

--checkpoint-path Cosmos-H-Surgical

For transfer, pass the prepared specifications together so the model is loaded once:

torchrun --nproc-per-node=8 \
  -m cosmos_h_surgical infer \
  --parallelism-preset=latency \
  --dp-shard-size=1 \
  --no-use-torch-compile \
  -i "inputs/transfer/specs/*.json" \
  -o outputs/cosmos-h-surgical/transfer \
  --no-guardrails \
  --seed=0

The first inference downloads the checkpoint and tokenizer dependencies into the Hugging Face cache; later runs reuse the cache.

Training and Evaluation Data

Dataset Overview

Cosmos-H-Surgical was trained using GraSP real surgical video and synthetic surgical video data. The synthetic collection contains 12,600 laparoscopic cholecystectomy videos generated through the Cosmos-H-Surgical transfer pipeline. GraSP contributes 15,043 robot-assisted radical prostatectomy videos. The combined collection uses a 90% training and 10% validation split.

Dataset	Source	Role
Synthetic Data	Synthetic surgical video	Prediction and transfer adaptation
GraSP	Real robot-assisted surgery	Surgical-domain adaptation and validation

The training collection contains approximately 24,879 videos and the held-out validation collection approximately 2,764 videos. Total video duration is less than 10,000 hours.