An overview and some of the recent work at the ACRV in Australia. Presentation at the IST Lisbon robotics group (ISR)

1. Juxi Leitner
arc centre of excellence for robotic vision
queensland university of technology
<j.leitner@qut.edu.au> http://Juxi.net
robotic vision
an australian vision for robotics
ACRV @ UC Berkeley

3. http://roboticvision.org/
tinyurl.com/QUTRobotics
ICRA 2018
Brisbane,Australia

4. http://roboticvision.org/
tinyurl.com/QUTRobotics
roboticvision.org

5. http://roboticvision.org/

6. http://roboticvision.org/

7. http://roboticvision.org/
10 96%eye
designs
animals
haveeyes
Nature Reviews | Neuroscience
Protostomes
Bilateria
Ecdysozoa
Lophotrochozoa
~580 Mya
~550 Mya
~530 Mya
~500 Mya
~430 Mya
Deuterostomes
Chordates*
Craniates*
Vertebrates*
Gnathostomes*
Arthropods
Annelids
Molluscs
Hemichordates
Echinoderms
Cephalochordates*
Tunicates*
Myxiniformes*
Petromyzoniformes*
Last fossil jawless fish
Stages of interest in
vertebrate eye evolution Cambrian
Mya
600 550 500 450 400 0
21 3 4 5 6
Lampreys
Jawed vertebrates
Hagfish
Lancelets
Sea squirts
Ocellus
Eye patch
Figure1|The origin of vertebrates. Theevolutionofjawedvertebratesisillustratedagainstanapproximatetime-scaleof
millionsofyearsago(Mya).ThetaxaconsideredinthisReviewareindicatedwithanasteriskandareaccompaniedby
schematicsanddiagramsofthe‘eye’region.Theearliestchordates,representedbyextantcephalochordatesandtunicates,
arethoughttohaveappearedaround550Mya.Jawlesscraniates(agnathans)werepresentintheearlyCambrian,by525Mya,
andatimeof530Myahasbeenindicatedfortheirpresumedfirstappearance.AselaboratedoninBOX 1,thereis
considerablecontroversyastowhethermyxiniformes(solelyrepresentedbyextanthagfish)divergedbeforeorafterthe
separationoflampreysfromjawedvertebrates(shownasdashedblackandgreylines).Numerouslinesofjawlessfishevolved
between500and430Myaago,althoughnonehavesurvivedtothepresentday.Thefirstjawedvertebratearosearound430
Mya,andthislineisrepresentedtodaybycartilagenousfish,bonyfishandtetrapods.Six‘stagesofinterest’invertebrateeye
evolutioncorrespondtothetimeintervalsbetweenthedivergenceofimportantsurvivingtaxa.Thisdiagramdoesnot
includetheevolutionarychangesthathaveoccurredinthelast400millionyears.Thepresentedtimelineisbasedprimarily
onevidencefromthefossilrecord;seeREFS 2,13,15,17,18,144,160–163.Theschematicsaremodified,withpermission,from
REF. 11 (1996)OxfordUniversityPress(lancelet,seasquirt,hagfishandlamprey)andREF. 164 (2004)AcademicPress
(jawedvertebrate).Theeyeimagesarereproduced,withpermission,fromthefollowingreferences:lancelet,REF. 165
BIODIDAC(1996)UniversityofCaliforniaMuseumofPaleontology;seasquirt,REF. 63 (2006)BlackwellPublishing;hagfish,
REF. 166 (2006)AustralianMuseum.LampreyandjawedvertebrateeyeimagesarecourtesyofG.WesthoffandS.P.Collin).
CIENCE VOLUME 8 | DECEMBER 2007 | 961

8. VISION
ROBOTIC
thanks to G. Metta and IIT for this picture
Robust Vision
Seman/c Representa/ons
Algorithms & Architecture
Visual Learning
Vision and Ac/on

9. Dalle Molle Institute for AI (IDSIA)
Work
Juxi
Leitner
PhD Informatics / Intelligent Systems
MSc Space Robotics & Automation
BSc Information & Software Engineering
Intelligent (Space) Robots
European Space Agency (ESA)
Erasmus Intelligent Systems
Work (Humanoid) Robot Vision
Instituto Superior Técnico (IST)
Mobility Intelligent Space Systems Laboratory
About Me
Current Robotic Vision and Actions
h:p://Juxi.net
Queensland University of Technology (QUT)

10. coordination
eye-hand
move to see
see to move
Vision
and Ac2on
http://Juxi.net/projects/VisionAndAction/

11. http://Juxi.net/projects
coordinationeye-hand

12. http://roboticvision.org/

13. http://roboticvision.org/
http://Juxi.net/special-issue/ijrr-deep-learning-2016/
IJRR Special Issue
R:SS WS
(what) scep/cs?
deeper understanding
tool (how to use it?)
RV vs CV
data driven models

14. [Zhang et al, ACRA 2015]
deep learning visual control

15. understanding limita/ons of deep nets,
reinforcement learning & transfer of knowledge
deep learning visual control
Perception Module Control Module
Conv1 Conv2 Conv3 FC_c2 FC_c3FC_c1
Q-values
7×7conv+ReLU
stride2
4×4conv+ReLU
stride2
3×3conv+ReLU
stride1
64 lters 64 lters 64 lters
fullyconn.
300units
9units
84×84
400units
fullyconn.
fullyconn.+ReLU
fullyconn.+ReLU
I BN
5units
θ
Bottleneck
Or
Occlusion Occlusion Occlusion
[Zhang et al, RA:L (submitted)]

16. vision & action
http://Juxi.net/projects

17. manipulation
for improved perception

18. http://roboticvision.org/
propertiesdetectingvisually
[Dansereau, Singh, Leitner, ICRA2016]
http://Juxi.net/projects/squishy/

19. http://roboticvision.org/
http://Juxi.net/projects/AmazonPickingChallenge/

20. http://roboticvision.org/
http://Juxi.net/projects/AmazonPickingChallenge/
systemdesign

21. http://roboticvision.org/
http://Juxi.net/projects/AmazonPickingChallenge/
systemdesign

22. systemdesign
Object
Detection
Object
Localisation
Object
Identification
Shelf
Localisation
perception
Kinect2
(once)
RealSense
(every attempt)
KinectFusion
2D regions
(CNN)
point cloud
segmentation
grasp point
detection
http://Juxi.net/projects/AmazonPickingChallenge/

23. http://roboticvision.org/
score

24. http://roboticvision.org/

25. ARC Centre of Excellence for Robotic Vision roboticvision.org
fruit picking
https://www.youtube.com/watch?v=8rq4iSTsg68

26. ARC Centre of Excellence for Robotic Vision roboticvision.org
limita/ons of current robo/c systems
reproducible research on TASKS not datasets
picking benchmark
http://Juxi.net/dataset/acrv-picking-benchmark/

27. http://Juxi.net/dataset/acrv-picking-benchmark/arxiv number!

28. tinyurl.com/QUTLunaRoo
Australian Moonshot

29. tinyurl.com/QUTLunaRoo
Going
Higher
SKP-2SKP-1A

30. tinyurl.com/QUTLunaRoo TIM HOJNIK
Leaving Footprints

31. for listening
thank you j.leitner@roboticvision.org
http://Juxi.net/projects
ICRA 2018
Brisbane,Australia