Presented at the IROS workshop on "DEVELOPMENT OF BENCHMARKING PROTOCOLS FOR ROBOTIC MANIPULATION" http://ycbbenchmarks.org/IROS2017workshop.html The ACRV Picking Benchmark has been developed over the last year to facilitate comparison of robotic systems in pick and place settings! With the ABP we propose a physical benchmark for robotic picking: overall design, objects, configuration, and guidance on appropriate technologies to solve it. Challenges are an important way to drive progress but they occur only occasionally and the test conditions are difficult to replicate outside the challenge. This benchmark is motivated by experience in the recent Amazon Picking Challenge and contains a commonly-available shelf, 42 objects, a set of stencils and standardized task setups. A major focus through the design of this benchmark was to maximise reproducibility: a number of carefully chosen scenarios with precise instructions on how to place, orient, and align objects with the help of printable stencils are defined. To make the benchmark as accessible as possible to the research community, a white IKEA shelf is used for all picking tasks. Furthermore, we carefully curated a set of 42 objects to ensure global availability and reduced chance of import restrictions.

1. The ACRV Picking
Benchmark
j.leitner@qut.edu.au
Jürgen ‘Juxi’ Leitner
Juxi
#IROS2017
J, Leitner, AW Tow, JE Dean, N. Suenderhauf, JW Durham, M. Cooper, M. Eich, C.
Lehnert, R. Mangels, C. McCool, P. Kujala, L. Nicholson, T. Pham, J. Sergeant, L. Wu, F.
Zhang, B. UpcroI, Pl Corke
http://Juxi.net/acrv-picking-benchmark/

2. http://roboticvision.org/
http://Juxi.net/acrv-picking-benchmark/
https://arxiv.org/abs/1609.05258
The ACRV Picking Benchmark: A Robo6c Shelf Picking
Benchmark to Foster Reproducible Research
Leitner et al, ICRA 2017

3. http://roboticvision.org/
overcoming limita6ons of
robo6c systems comparison
reproducible research on
end-to-end TASKS
http://Juxi.net/acrv-picking-benchmark/

4. http://roboticvision.org/

5. http://roboticvision.org/
http://Juxi.net/acrv-picking-benchmark/
“manual”
standard protocol
overcome limits —> reproducible research on
PICKING TASKS
PICKING BENCHMARK
PICK AND PLACE TASKS FOCUS

6. http://roboticvision.org/
standard shelf
overcome limits —> reproducible research on
TASKS
http://Juxi.net/acrv-picking-benchmark/
PICKING BENCHMARK

7. http://roboticvision.org/
standard objects
http://Juxi.net/acrv-picking-benchmark/
PICKING BENCHMARK

8. http://roboticvision.org/
CREATE An APB slide!!
http://Juxi.net/acrv-picking-benchmark/
PICKING BENCHMARK

9. http://roboticvision.org/
task evaluaFon
repeatable
video + public ranking
open source
overcome limits —> reproducible research on
TASKS
http://Juxi.net/acrv-picking-benchmark/
PICKING BENCHMARK

10. http://roboticvision.org/
setups / stencils standard setups
overcome limits —> reproducible research on
TASKS
http://Juxi.net/acrv-picking-benchmark/
PICKING BENCHMARK

11. http://roboticvision.org/
repeat and run 3x
http://Juxi.net/acrv-picking-benchmark/
PICKING BENCHMARK
How do you evaluate your own
robo6c system on the APB?
http://Juxi.net/acrv-picking-benchmark/pdf/manual.pdf

12. http://roboticvision.org/
http://Juxi.net/acrv-picking-benchmark/
PICKING BENCHMARK
standard setups

13. roboticvision.orghttp://roboticvision.org/
picking benchmark
http://Juxi.net/acrv-picking-benchmark/

14. roboticvision.orghttp://roboticvision.org/
http://Juxi.net/acrv-picking-benchmark/
thank you!
http://Juxi.net/acrv-picking-benchmark/
PICKING BENCHMARK