Briefly explains about robotic vision.

1. Robot Vision System

2. Robotics in General
Industrial Robotics
Medical Robotics
Robot Vision
What can Computer Vision do for
Robotics?
Vision Sensors
Issues / Problems
Visual Servoing
Application Examples
Summary
CONTENTS

3.  What is a robot?
"A reprogrammable, multifunctional manipulator
designed to move material, parts, tools, or specialized
devices through various programmed motions for the
performance of a variety of tasks"
Robot Institute of America, 1979
 Industrial
 Mostly automatic manipulation of rigid parts with well-
known shape in a specially prepared environment.
 Medical
 Mostly semi-automatic manipulation of deformable
objects in a naturally created, space limited
environment.
 Field Robotics
 Autonomous control and navigation of a mobile vehicle
in an arbitrary environment.

4. Robot vs
Human
 Robot Advantages:
 Strength
 Accuracy
 Speed
 Does not tire
 Does repetitive tasks
 Can Measure
• Human advantages:
• Intelligence
• Flexibility
• Adaptability
• Skill
• Can Learn
• Can Estimate

5.  Requirements:
 Accuracy
 Tool Quality
 Robustness
 Strength
 Speed
 Price Production Cost
 Maintenance
Industrial
Robot
Production Quality

6. Medical (Surgical)
Robot
 Requirements
 Safety
 Accuracy
 Reliability
 Tool Quality
 Price
 Maintenance
 Man-Machine Interface

7. Vision for robots requires the ability to identify and
accurately determine the positions of all relevant three
dimensional objects within the robot work place.
Robot
Vision
Robot vision may be defined as the process of extracting,
characterizing, and interpreting information from images
of a three dimensional world

8. ROBOT VISION
2000 Jaskaran Singh
Purpose of A Machine Vision
System
 Analyzes images and produces descriptions of
what is being imaged.
 Input to the system- Image
 Output from the system- satisfy two criteria.

9. ROBOT VISION
2000 Jaskaran Singh
Robot Vision-Fundamental
Tasks
 -Image transformation
 -Image analysis
 -Image understanding

10. ROBOT VISION
2000 Jaskaran Singh
General Purpose Robot
Vision
 Important thing-
System should capture the relevant data and with the
motion of the object it should be able to update the
information.
Four steps to General Purpose Robot Vision
 Object verification and tracking
 Fast extraction of stable image features
 Object model acquisition
 Efficient indexing of the model database

11. What can Computer Vision do for Robotics?
 Accurate Robot-Object Positioning
 Keeping Relative Position under Movement
 Visualization / Teaching / Telerobotics
 Performing measurements
 Object Recognition
 Registration

12. Vision
Sensors
 Single Perspective Camera
 Multiple Perspective Cameras (e.g. Stereo Camera Pair)
 Laser Scanner
 Omnidirectional Camera
 Structured Light Sensor

13. Vision
Sensors
 Single Perspective Camera
XPx x43

14.  Multiple Perspective Cameras (e.g. Stereo Camera
Pair)
Vision
Sensors

15.  Multiple Perspective Cameras (e.g. Stereo Camera
Pair)
0Fxx'T
 Fxl'
Vision
Sensors

16.  Laser Scanner
Vision
Sensors

17.  Laser Scanner
Vision
Sensors

18.  Omnidirectional Camera
Vision
Sensors

19.  Omnidirectional Camera
Vision
Sensors

20.  Structured Light Sensor
Figures from PRIP, TU Vienna
Vision
Sensors

21. Issues/Problems of Vision Guided
Robotics
 Measurement Frequency
 Measurement Uncertainty
 Occlusion, Camera Positioning
 Sensor dimensions

22. Visual
Servoing
 Vision System operates in a closed control loop.
 Better Accuracy than „Look and Move“ systems
Figures from S.Hutchinson: A Tutorial on Visual Servo Control

23.  Example: Maintaining relative Object Position
Figures from P. Wunsch and G. Hirzinger. Real-Time Visual Tracking of 3-D Objects with Dynamic Handling of Occlusion
Visual
Servoing

24. Visual
Servoing
 Camera Configurations:
End-Effector Mounted Fixed
Figures from S.Hutchinson: A Tutorial on Visual Servo Control

25.  Servoing Architectures
Figures from S.Hutchinson: A Tutorial on Visual Servo Control
Visual
Servoing

26.  Position-based and Image Based control
 Position based:
 Alignment in target coordinate system
 The 3D structure of the target is rconstructed
 The end-effector is tracked
 Sensitive to calibration errors
 Sensitive to reconstruction errors
 Image based:
 Alignment in image coordinates
 No explicit reconstruction necessary
 Insensitive to calibration errors
 Only special problems solvable
 Depends on initial pose
 Depends on selected features
target
End-effector
Image of target
Image of end
effector
Visual
Servoing

27.  EOL and ECL control
 EOL: endpoint open-loop; only the target is
observed by the camera
 ECL: endpoint closed-loop; target as well as end-
effector are observed by the camera
EOL ECL
Visual
Servoing

28.  Position Based Algorithm:
1. Estimation of relative pose
2. Computation of error between current pose and
target pose
3. Movement of robot
 Example: point alignment
p1
p2
Visual
Servoing

29.  Position based point alignment
 Goal: bring e to 0 by moving p1
e = |p2m – p1m|
u = k*(p2m – p1m)
 pxm is subject to the following measurement errors: sensor
position, sensor calibration, sensor measurement error
 pxm is independent of the following errors: end effector
position, target position
p1m p2m
d
Visual
Servoing

30.  Image based point alignment
 Goal: bring e to 0 by moving p1
e = |u1m – v1m| + |u2m – v2m|
 uxm, vxm is subject only to sensor measurement error
 uxm, vxm is independent of the following measurement
errors: sensor position, end effector position, sensor
calibration, target position
p1 p2
c1
c2
u1
u2
v1 v2
d1
d2
Visual
Servoing

31.  Example Laparoscopy
Figures from A.Krupa: Autonomous 3-D Positioning of Surgical Instruments in Robotized Laparoscopic Surgery Using Visual Servoing
Visual
Servoing

32.  Example Laparoscopy
Figures from A.Krupa: Autonomous 3-D Positioning of Surgical Instruments in Robotized Laparoscopic Surgery Using Visual Servoing
Visual
Servoing

33. Registratio
n
 Registration of CAD models to scene features:
Figures from P.Wunsch: Registration of CAD-Models to Images by Iterative Inverse Perspective Matching

34.  Registration of CAD models to scene features:
Figures from P.Wunsch: Registration of CAD-Models to Images by Iterative Inverse Perspective
Matching
Registratio
n

35. Trackin
g
 Instrument tracking in laparoscopy
Figures from Wei: A Real-time Visual Servoing System for Laparoscopic Surgery

36. Summar
y
 Computer Vision provides accurate and versatile
measurements for robotic manipulators
 With current general purpose hardware, depth
and pose measurements can be performed in
real time
 In industrial robotics, vision systems are
deployed in a fully automated way.
 In medicine, computer vision can make more
intelligent „surgical assistants“ possible.