Workshop slides for the ICRA 2019 workshop on "Benchmarks for Robotic Manipulation"

1. www.data61.csiro.au
A Benchmark for Simulated
Manipulation
Jack Collins, David Howard, Juxi Leitner
Queensland University of Technology (QUT), Brisbane, Australia
Data61/CSIRO, Brisbane, Australia

2. • Currently we have protocols, benchmarks and object
data sets for manipulation.
• Benchmarks and data sets are all for real-world
manipulation but a lot of research is undertaken
primarily or initially in simulation.
• In the context of manipulation how can we validate our
simulation as transferable to the real world?
• Who will this be useful for?
• Manipulation Community
• Sim2Real
• Simulator/Physics Engine Developers
Motivation
(Image: Leitner et al., 2017)
(Image: Calli et al., 2015)
A Benchmark for Simulated Manipulation | Jack Collins2 |

3. Protocol for Real-World
Data
• Setup
• State Elements Monitored
• File Formats
Protocol for Simulation
• Setup
• State Elements Monitored
• File Formats
Benchmark
• Metrics
Planned Benchmarking Protocol
(Image: Collins et al., 2019)
(Image: Collins et al., 2019)
A Benchmark for Simulated Manipulation | Jack Collins3 |

4. Real-world Data
• Setup
- YCB Protocol Template
• Workspace
• Robot (Type, Control, etc.)
• Positions of Objects
• Coordinate Frame
• Frequency of Data Recording (100Hz)
• Repetitions (20)
• State Elements
• 6DOF Bodies (Wrist position and Additional objects)
• File Format
• CSV file with added header
Time X-Pos Y-Pos Z-Pos W-Quat X-Quat Y-Quat Z-Quat …
0.000
0.001
A Benchmark for Simulated Manipulation | Jack Collins4 |

5. Simulation
Time X-Pos Y-Pos Z-Pos W-Quat X-Quat Y-Quat Z-Quat …
0.00
0.05
• Setup
- YCB Protocol Template
• Workspace
• Robot (Type, Control, etc.)
• Positions of Objects
• Coordinate Frame
• Frequency of Data Recording (20Hz)
• Repetitions
• State Elements
• 6DOF Bodies (Wrist position and Additional objects)
• File Format
• CSV file with added header
A Benchmark for Simulated Manipulation | Jack Collins5 |

6. Benchmarking
• YCB Manipulation Benchmark Template
• Metrics
- Calculate throughout experimental run (20Hz) (sum/total, min, max,
mean)
• Euclidean Distance
• Rotation: Geodesic on the Unit Sphere (0-Pi)
• Delta: The change between time steps
• First Derivative: Velocity
• Second Derivative: Acceleration
• Compare real data to simulated
• Compare the CSV’s
• Using the provided script, generate metrics to benchmark
• Submit simulator data to display score publicly
𝑒𝑒 = 𝑠𝑠𝑥𝑥 − 𝑚𝑚𝑥𝑥
2 + 𝑠𝑠𝑦𝑦 − 𝑚𝑚𝑦𝑦
2
+ 𝑠𝑠𝑧𝑧 − 𝑚𝑚𝑧𝑧
2
A Benchmark for Simulated Manipulation | Jack Collins6 |

7. Tasks
Things that will expose flaws in a
simulator
• Heavy objects
• Dumbbells
• Robot Kinematics
• Interaction with objects
• Drop a sphere
• Roll a sphere
• Roll a cylinder
• Roll a cone
• Knock over a cuboid
• Multiple interactions
• Push multiple cuboids across a tabletop
• Push cylinders and cuboids
• Large Surface area
• Wave around a rigid sheet
• Long Levers
• Move a broom handle or something around
Tasks relevant to the community
• Kitchen Tasks
• Unscrew a jar lid
• Stir spoon in saucepan
• Cluttered Picking/Packing
• Pack/Unpack cuboids into box
• Stacking
• Peg-In-Hole
• Buttons/Switches/Handles
A Benchmark for Simulated Manipulation | Jack Collins7 |

8. A Benchmark for Simulated Manipulation | Jack Collins8 |

9. www.data61.csiro.au
Jack Collins
e jack.collins@data61.csiro.au
w jacktcollins.com
THANK YOU