RoboSnap

One-Shot Real-to-Sim Scene Generation for Generalizable Robot Learning and Evaluation

Shujie Zhang1,4* Jingkun Yi1,3* Weipeng Zhong1,2 Zirui Zhou4 Yangkun Zhu1 Hanqing Wang1 Xudong Xu1† Weinan Zhang1,2 Chunhua Shen1,3
1Shanghai AI Laboratory 2Shanghai Jiao Tong University 3Zhejiang University 4Tsinghua University
*Equal contribution. †Corresponding author.
Paper Video Code Dataset Coming soon
564 DROID-Sim scenes
+30.0% success-rate gain
0.887 real-sim correlation

From One RGB Image to Sim-Ready Real-to-Sim 3D Scene

With interactive foreground geometry and visual background constructed separately,
RoboSnap scenes can be conveniently re-rendered, edited and reused for robot learning workflows.

Abstract

Real-to-Sim: From Digital Twins to Reusable Robot Learning Infrastructure.

RoboSnap overview: DROID-Sim, trajectory replay, data generation, robustness, and sim-real correlation

Recovering real-world scenes as interactive simulation environments can enable generalizable robot learning and reproducible policy evaluation. However, constructing scenes that are both physically stable and visually faithful remains slow and expensive.

RoboSnap builds a simulation-ready scene from one RGB image by separating collision-aware foreground assets from a re-renderable Gaussian-splat visual context.

Across DROID scenes and real-robot tasks, the recovered scenes support trajectory replay, synthetic data generation, and sim-real evaluation. We also introduce DROID-Sim, a companion dataset of 564 real-world scenes for real-to-sim robot learning.

Method

Construct a Simulation-Ready Scene in 20 Minutes.

RoboSnap method pipeline
1

Layered Scene Reconstruction

Starting from a single RGB image, RoboSnap separates the scene into what the robot can physically interact with and what should remain as visual context. Objects and support surfaces are recovered as a physical layer, while the surrounding environment is completed and kept as a re-renderable visual layer.

Both layers are registered into the same world frame, so foreground assets can be simulated, moved, and checked for contact while the background continues to preserve the real deployment context. For articulated objects, RoboSnap also supports splitting the reconstructed mesh into joint-aware parts.

01 / 10

01

Kitchen manipulation scene

Scene 01 original RGB image
Original RGB
Scene 01 inpainted background
Inpainted background
Scene 01 rendered simulation
Rendered simulation

02

Tabletop objects and bowl task

Scene 02 original RGB image
Original RGB
Scene 02 inpainted background
Inpainted background
Scene 02 rendered simulation
Rendered simulation

03

Striped-wall tabletop scene

Scene 03 original RGB image
Original RGB
Scene 03 inpainted background
Inpainted background
Scene 03 rendered simulation
Rendered simulation

04

Round-table office layout

Scene 05 original RGB image
Original RGB
Scene 05 inpainted background
Inpainted background
Scene 05 rendered simulation
Rendered simulation

05

Small tabletop mug scene

Scene 07 original RGB image
Original RGB
Scene 07 inpainted background
Inpainted background
Scene 07 rendered simulation
Rendered simulation

06

Cardboard-box workspace

Scene 09 original RGB image
Original RGB
Scene 09 inpainted background
Inpainted background
Scene 09 rendered simulation
Rendered simulation

07

Workshop layout

Scene 10 original RGB image
Original RGB
Scene 10 inpainted background
Inpainted background
Scene 10 rendered simulation
Rendered simulation

08

Herringbone-floor office table

Scene 12 original RGB image
Original RGB
Scene 12 inpainted background
Inpainted background
Scene 12 rendered simulation
Rendered simulation

09

Industrial workshop scene

Scene 14 original RGB image
Original RGB
Scene 14 inpainted background
Inpainted background
Scene 14 rendered simulation
Rendered simulation

10

Minimal room scene

Scene 15 original RGB image
Original RGB
Scene 15 inpainted background
Inpainted background
Scene 15 rendered simulation
Rendered simulation
2

Simulation-Ready Refinement

RoboSnap turns generated assets into a scene that can directly run in a simulator. Because independent object pose estimates can float, interpenetrate, or collapse under gravity, refinement should improve physical stability while changing the recovered layout as little as possible.

The process first builds a physical scene graph that records which objects support others and which objects should remain in contact. Support roots stay fixed, while non-root objects are refined with an alternating SDF-physics procedure: the SDF phase removes penetrations and restores contact constraints, and the physics phase settles the updated layout under gravity.

Repeating these two phases produces stable poses for simulation without washing away the original scene structure, so replay and interaction happen on a layout that still matches the recovered real environment.

Example-1
Example-2
3 Applications

Data Generation & Policy Evaluation

DROID-Sim

Preview of examples from DROID-Sim

Original RGB reference 1
1
1

Large interactive scene assets are loaded on demand.
The viewer may take 2-5 minutes to become ready.

RoboSnap scenes support real trajectory replay in the simulation.

By completing real trajectory replay, RoboSnap tries to turn robot datasets into reusable simulation scene datasets for scalable and generalizable data augmentation.

Ground Truth
RoboSnap Success
RoLA Success
Ground Truth
RoboSnap Success
RoLA Failure

Synthetic data generated in RoboSnap scenes improves real robot policy.

RoboSnap synthetic data improves real-world success rates most consistently at Ratio 2: +9.0 pts for pi0.5 and +13.4 pts for pi0 on average, with individual-task gains up to +30.0 pts.

Real-world robot task settings
Real-world task settings

Real-world task descriptions

1.1

Put the bread into the blue plate.

1.2

Put the spoon in the pot.

2.1

Put the carrot on the desk into the bowl.

2.2

Put the pumpkin into the blue bowl.

2.3

Put the small hamburger in the plate.

3.1

Open the white microwave.

3.2

Put the bowl into the microwave.

4.1

Put the can into the white bin.

4.2

Close the laptop.

4.3

Hang the mug on the wooden shelf.

Tasks 3.1 and 3.2 are consecutive stages of a long-horizon task.

Average real-world success rate across real-only and data mixture ratios
Average success rates across data-mixture ratios

Before co-training 50% success rate

Failure Case (5x speed)

After co-training 80% success rate

Success Case (5x speed)

Browse Data Generated in RoboSnap Scenes Here

01 / 06

Example 1

Exterior
Wrist

Example 2

Exterior
Wrist

Example 3

Exterior
Wrist

Example 4

Exterior
Wrist

Example 5

Exterior
Wrist

Example 6

Exterior
Wrist

Example 7

Exterior
Wrist

Example 8

Exterior
Wrist

Example 9

Exterior
Wrist

Example 10

Exterior
Wrist

Example 11

Exterior
Wrist

Example 12

Exterior
Wrist

Augmented synthetic data from RoboSnap scenes improves real-world robustness.

Across six real-world perturbations, sim-real co-training with RoboSnap scenes reduces average policy degradation from 13% to 8%.

Performance degradation under six real-world perturbations
Real-sim co-training reduces average degradation
from 13% to 8% across six perturbations.

Successful rollouts of Ratio 2 policy under real-world perturbations

01 / 03

Object Pose

Exterior
Wrist

Background

Exterior
Wrist

Lighting

Exterior
Wrist

Texture

Exterior
Wrist

Camera Pose

Exterior
Wrist

Initial State

Exterior
Wrist
5x speed

RoboSnap scenes can act as a policy evaluation harness.

RoboSnap reports Pearson correlation r = 0.887 and MMRV = 0.0066 between real-world and simulation success rates for real-only fine-tuned pi0.5 policies.

This suggests that simulation scenes recovered from a single image can capture meaningful visual, semantic, and task-difficulty dynamics.

Real-only fine-tuned policy rollouts in real-world and RoboSnap scenes

01 / 04
Real-world rollout
RoboSnap simulation rollout
Real-world rollout
RoboSnap simulation rollout
Real-world rollout
RoboSnap simulation rollout
Real-world rollout
RoboSnap simulation rollout
5x speed
Behavior-space sim-real comparison of end-effector displacement trajectories and joint increment distributions
Behavior-space diagnostics compare end-effector trajectories and
executed joint-increment distributions between real and simulated rollouts.

Thanks for reading here!

BibTeX

@misc{zhang2026robosnap,
  title={RoboSnap: One-Shot Real-to-Sim Scene Generation for Generalizable Robot Learning and Evaluation},
  author={Shujie Zhang and Jingkun Yi and Weipeng Zhong and Zirui Zhou and Yangkun Zhu and Hanqing Wang and Xudong Xu and Weinan Zhang and Chunhua Shen},
  year={2026}
}