Paper 2603.15612
HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human-Scene Interactions
- Published
- Mar 2026
- Research lab
- Independent
- Citations
- 2
- GitHub
- 43 stars
01 In brief
Summary
HSImul3R is a unified framework for reconstructing simulation-ready 3D human–scene interactions (HSI) from casual captures like sparse-view images and monocular videos.
It addresses the perception–simulation gap where visually plausible reconstructions fail in physics engines.
The method uses a physically-grounded bi-directional optimization pipeline with a physics simulator as an active supervisor.
Forward optimization refines human motion via scene-targeted reinforcement learning, while reverse optimization refines scene geometry using Direct Simulation Reward Optimization (DSRO) based on simulation feedback.
The authors also introduce HSIBench, a dataset with 19 objects, 50 motion sequences, and 300 interaction instances from three participants.
Experiments show HSImul3R outperforms baselines like HSfM in stability and motion quality, and it enables real-world deployment on Unitree G1 humanoid robots.
Limitations include lower success rates in complex multi-object scenarios and potential biases from training data.
The framework is the first to achieve stable, simulation-ready HSI reconstructions, bridging visual realism and embodied AI deployment.
The method extends to monocular videos for 4D reconstructions and demonstrates robust performance across diverse scenarios, significantly improving over existing techniques.
The pipeline includes alignment via explicit 3D structural priors from image-to-3D generative models, and the DSRO fine-tuning improves object geometry and stability.
The work is validated through extensive quantitative and qualitative evaluations, including ablation studies and analysis of input view counts.
The approach also supports training whole-body control policies for physical deployment, showcasing its utility in embodied AI applications.
The paper concludes by highlighting the framework's contributions and potential for scalable, cost-effective data augmentation for large-scale embodied AI models…
02 From the paper
Abstract
We present HSImul3R, a unified framework for simulation-ready 3D reconstruction of human-scene interactions (HSI) from casual captures, including sparse-view images and monocular videos. Existing methods suffer from a perception-simulation gap: visually plausible reconstructions often violate physical constraints, leading to instability in physics engines and failure in embodied AI applications. To bridge this gap, we introduce a physically-grounded bi-directional optimization pipeline that treats the physics simulator as an active supervisor to jointly refine human dynamics and scene geometry. In the forward direction, we employ Scene-targeted Reinforcement Learning to optimize human motion under dual supervision of motion fidelity and contact stability. In the reverse direction, we propose Direct Simulation Reward Optimization, which leverages simulation feedback on gravitational stability and interaction success to refine scene geometry. We further present HSIBench, a new benchmark with diverse objects and interaction scenarios. Extensive experiments demonstrate that HSImul3R produces the first stable, simulation-ready HSI reconstructions and can be directly deployed to real-world humanoid robots.