PrimeSim: A Generalist World Model Simulator from Large-Scale Cross-Embodiment Demonstrations
Paper • 2611.04870 • Published • 10
Image-to-Video
Diffusers
English
robotics
video-generation
world-model
multiview
robot-manipulation
simulation
import torch
from diffusers import DiffusionPipeline
from diffusers.utils import load_image, export_to_video
# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("qiyuan-robot/PrimeSim", dtype=torch.bfloat16, device_map="cuda")
pipe.to("cuda")
prompt = "A man with short gray hair plays a red electric guitar."
image = load_image(
"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/guitar-man.png"
)
output = pipe(image=image, prompt=prompt).frames[0]
export_to_video(output, "output.mp4")PrimeSim
An action-conditioned, multiview world model that runs as a robot simulator. Given the first frame of each camera view, a prompt and an action trajectory, it generates what happens next.
- Paper: arXiv 2611.04870
- Project page: primebottech.github.io/PrimeSim-website
- Code: github.com/PrimebotTech/PrimeSim -- inference and training code
Trained on 30,110 hours of cross-embodiment data spanning 22 platforms. In simulation it leads four of five WorldSimProbe suites and posts the best overall EWMBench score.
Files
| file | use |
|---|---|
primesim_base.safetensors |
the midtrained model -- start here to fine-tune |
primesim_posttrain_g2.safetensors |
stage 4, causal few-step model -- use for inference |
Run primesim_posttrain_g2 on the demos in the
repository to see what posttraining adds.
Requirements
These weights are not self-contained. The text encoder, VAE and tokenizer come from Wan-AI/Wan2.2-TI2V-5B:
hf download Wan-AI/Wan2.2-TI2V-5B \
--include "models_t5_umt5-xxl-enc-bf16.pth" "Wan2.2_VAE.pth" "google/umt5-xxl/*" \
--local-dir checkpoints
Point config/model/primesim.yaml at the three resulting paths.
Inference
Three camera views are generated together at 192x256, running above 10 FPS. An A100 or H100 (80 GB) is recommended: the 5B transformer and the 11 GB text encoder are resident at the same time, which does not fit on a 24 GB card.
License
Apache 2.0. The training corpus is not distributed and is not covered by this licence -- the weights are released, the data is not.
- Downloads last month
- 120
Model tree for qiyuan-robot/PrimeSim
Base model
Wan-AI/Wan2.2-TI2V-5B