Download to read offline
Uploaded byRenaldas Zioma
Challenges in Sim 2 Real. Tutorial on Simulation Environments.
Tutorial Talk at Telluride Neuromorphic Cognition Workshop in 2025. Imperfections in dynamical systems, sensor simulation, imperfect data acquisition and real world noise hampers the usability of robotic simulations in the real world. The talk will introduce the common architecture behind the physical simulation engines and sensor simulation, as well as discuss the challenges balancing performance, speed and applicability to robotic simulations.
Challenges in Sim 2 Real. Tutorial on Simulation Environments.
- 1. ReJ aka RenaldasZioma @ Telluride’2025 Telluride Neuromorphic Cognition Workshop Tutorial on Simulation Environments Challenges in Sim 2 Real IT IS NOT JUST PHYSICS!
- 2. Sim 2 Real •Physics simulation imperfections • Sensors —//— • Imperfect data acquisition • “Reasoning” integration with the simulation engine • Real world noise in sensors & actuators
- 3. Physics Engines • MuJoCo- pretty precise, great for robotics! • ODE - old, meh • Bullet - continuous time, partial GPU acceleration, but generally very fast • PhysX / Havok - fast, supports GPU acceleration! • DART / Drake - slow, very precise • BRAX - di ff erentiable, supports GPU acceleration!
- 4. Complexity levels ofphysics engines source: wikipedia
- 5. What physics enginesactually do? • Body dynamics (motion) • Collision detection • Friction, joints, springs… • Constraint solver
- 6. Body Dynamics • Usuallydiscreet time steps!
- 7. Body Dynamics • Usuallydiscreet time steps! • Newton-Euler dynamics
- 8. Body Dynamics • Usuallydiscreet time steps! • Newton-Euler dynamics Center of the Mass Velocity, Torque
- 9. Collision Detection • Findif objects overlap
- 10. Spatial Hierarchical Representation CollisionDetection for N bodies • Reduces O(N·N) ~> O(N·logN)
- 11. Types of SpatialHierarchies
- 12. Contact Resolution The simplestcase: 2 spheres
- 13. Contact Resolution The simplestcase: 2 spheres
- 14. Contact Resolution The simplestcase: 2 spheres
- 15.
- 16.
- 17. Contact Resolution 3 spheres? Canbe solved iteratively
- 19. Drag, friction, springs,joints…
- 20. Collisions + drag,friction, springs, joints… Images from: Physics Tutorial - Part III: Constrained Rigid Body Simulation by Nilson Souto
- 21. Impulse Based Dynamics •Collision detection • Friction, joints, springs… • Constraints • Impulses • Solver Collisions + drag, friction, springs, joints…
- 22. • Collision detection •Friction, joints, springs… • Constraints • Impulses • Solver Images from: Physics Tutorial - Part III: Constrained Rigid Body Simulation by Nilson Souto Another Spatial Hierarchical Representation Collisions + drag, friction, springs, joints…
- 23.
- 24.
- 25.
- 26.
- 27. Constructive Solid Geometry ConservativeApproximation
- 28.
- 29.
- 30. Fast Moving Objects Asimilar problem to “hollow” geometry Δt Δx Δx
- 31. Fast Moving Objects Asimilar problem to “hollow” geometry Δx Δx
- 32. Fast Moving Objects Asimilar problem to “hollow” geometry Δx Δx
- 33. The Future isDifferentiable
- 34. The Future isDifferentiable • Dynamics • Geometric primitives • Geometric meshes non di ff erentiable • Collisions non di ff erentiable • SDF • Joints For Physical Simulation
- 35.
- 36.
- 37.
- 38.
- 39.
- 40.
- 41.
- 42.
- 43. 2 Hard Problems ManualAnnotation Near Accident Situations
- 45.
- 46. Scenes: static and dynamicelements constituting stages, parameters and actors for the simulation Episodes: scene variations - different placements and actions for actors, scene parameters (weather, light, textures)
- 47. Scenes: static and dynamicelements constituting stages, parameters and actors for the simulation Episodes: scene variations - different placements and actions for actors, scene parameters (weather, light, textures)
- 52. Summary • Sim2Real Challenges •Physics and sensor simulation imperfections • Imperfect and real world data acquisition • “Reasoning” integration with the simulation engine • Differentiable physics to close RL loop • Directions to Bridge Sim2Real Gap • Domain Randomization • Learnable Approximations for Complex Real World Dynamics
- 53.
- 54.