This document discusses computer vision and robot vision. It describes early work using artificial neural networks to allow a robot to steer a vehicle based on camera images (ALVINN system). The document outlines the two main stages of robot vision: image processing and scene analysis. Image processing transforms raw images, e.g. through averaging, edge enhancement, and region finding algorithms. Scene analysis extracts task-specific information by interpreting lines, curves, and applying model-based approaches to reconstruct scenes from primitive 3D objects. Stereo vision obtains depth information through triangulation using two camera images.

1. Robot Vision
Chapter 6.

2. 2
Introduction
 Computer vision
– Endowing machines with the means to “see”
 Create an image of a scene and extract features
– Very difficult problem for machines
 Several different scenes can produce identical images.
 Images can be noisy .
 Cannot directly „invert‟ the image to reconstruct the scene.

3. 3
Human Vision (1)

4. 4
Human Vision (2)

5. 5
Human Vision (3)

6. 6
Steering an Automobile
 ALVINN system [Pomerleau 1991,1993]
– Uses Artificial Neural Network
 Used 30*32 TV image as input (960 input node)
 5 Hidden node
 30 output node
– Training regime: modified “on-the-fly”
 A human driver drives the car, and his actual steering
angles are taken as correct labels for the corresponding
inputs.
 Shifted and rotated images were also used for training.
– ALVINN has driven for 120 consecutive kilometers at
speeds up to 100km/h.

7. 7
Steering an Automobile-ALVINN

8. 8
Two stages of Robot Vision (1)
 Finding out objects in the scene
– Looking for “edges” in the image
 Edge:a part of the image across which the image intensity or some other
property of the image changes abruptly.
– Attempting to segment the image into regions.
 Region:a part of the image in which the image intensity or some other
property of the image changes only gradually.

9. 9
Two stages of Robot Vision (2)
 Image processing stage
– Transform the original image into one that is more
amendable to the scene analysis stage.
– Involves various filtering operations that help reduce
noise, accentuate edges, and find regions.
 Scene analysis stage
– Attempt to create an iconic or a feature-based
description of the original scene, providing the task-
specific information.

10. 10
Two stages of Robot Vision (3)
 Scene analysis stage produces task-specific
information.
– If only the disposition of the blocks is important, appropriate
iconic model can be (C B A FLOOR)
– If it is important to determine whether there is another block on
top of the block labeled C, adequate description will include the
value of a feature, CLEAR_C.

11. 11
Averaging (1)
 Original image can be represented as an m*n array of
numbers. The numbers represent the light intensities
at corresponding points in the image.
 Certain irregularities in the image can be smoothed by
an averaging operation.
 Averaging operation involves sliding an averaging
widow all over the image array.

12. 12
Averaging (2)
 Smoothing operation thickens broad lines and eliminates thin lines and
small details.
 The averaging window is centered at each pixel, and the weighted sum
of all the pixel numbers within the averaging window is computed. This
sum then replaces the original value at that pixel.

13. 13
Averaging (3)
 Common function used for smoothing is a Gaussian of
two dimensions.
 Convolving an image with a Gaussian is equivalent to
finding the solution to a diffusion equation when the
initial condition is given by the image intensity field.

14. 14
Averaging (4)

15. 15
Edge enhancement (1)
 Edge: any boundary between parts of the image with
markedly different values of some property.
 Edges are often related to important object properties.
 Edges in the image occur at places where the second
derivative of the image intensity is zero.

16. 16
Edge enhancement (2)

17. 17
Combining Edge Enhancement
with Averaging (1)
 Edge enhancement alone would tend to emphasize
noise elements along with enhancing edges.
 To be less sensitive to noise, both operations are
needed. (First averaging and then edge enhancing)
 We can convolve the one-dimensional image with the
second derivative of a Gaussian curve to combine
both operation.

18. 18
Combining Edge Enhancement
with Averaging (2)
 Laplacian is second-derivate-type operation that enhances edges of any
orientation.
 Laplacian of the two-dimensional Gaussian function looks like an upside-
down hat, often called a sombrero function.
 Entire averaging/edge-finding operation can be achieved by convolving
the image with the sombrero function(Called Laplacian filtering)

19. 19
6.4.4 Finding Region
 Another method for processing image
 to find “regions”
 Finding regions  Finding outlines

20. 20
A region of the image
 A region is homogeneous.
– The difference in intensity values of pixels in the region is no
more than some 
– A polynomial surface of degree k can be fitted to the intensity
values of pixels in the region with largest error less than 
 For no two adjacent regions is it the case that the union of all the
pixels in these two regions satisfies the homogeneity property.
 Each region corresponds to a world object or a meaningful part of
one.

21. 21
Split-and-merge method
1. The algorithm begins with just one candidate region,
the whole image.
2. Until no more splits need be made.
1. For all candidate regions that do not satisfy the
homogeneity property, are each split into four equal-
sized candidate regions.
3. Adjacent candidate regions are merged if their pixels
satisfying homogeneity property.

22. 22

23. 23
Regions Found by Split Merge for a
Grid-World Scene (from Fig.6.12)

24. 24
“Cleaned Up” the regions found by
Split-and-merge method
 Eliminating very small regions (some of which are
transitions between larger regions).
 Straightening bounding lines.
 Taking into account the known shapes of objects likely
to be in the scene.

25. 25
6.4.5 Using Image Attributes Other
Than Intensity
 Image attributes other than the homogeneity
 Visual texture
 fine-grained variation of the surface reflectivity of
the objects
 Ex) a field of grass, a section of carpet, foliage in
tree, the fur of animals
 The reflectivity variations in objects cause
similar fine-grained structure in image intensity.

26. 26
Methods for analyzing texture
 Structural methods
– Represent regions in the image by a tessellation (花纹) of
primitive “texels” –small shapes comprising black and white
parts
 Statistical methods
– Based on the idea that image texture is best described by a
probability distribution for the intensity values over regions of
the image.
– Ex) an image of a grassy field in which the blades of grass are
oriented vertically
 a probability distribution that peaks for thin, vertically
oriented regions of high intensity, separated by regions of low
intensity

27. 27
Other attributes
 If we had a direct way to measure the range from the camera to
objects in the scene, we could produce a “range image” and look
for abrupt range differences.
– Range image : each pixel value represents the distance from the
corresponding point in the scene to the camera.
 Motion, color

28. 28
6.5 Scene Analysis (1)
 Scene Analysis
– Extracting from the image the needed information about the
scene
– Requires either additional images (for stereo vision) or general
information about the kinds of scenes, since the scene-to-image
transformation is many-to-one.
 The required knowledge
– very general or quite specific
– explicit or implicit

29. 29
6.5 Scene Analysis (2)
 Knowledge of surface reflectivity characteristics and
shading of intensity in the image
 give information about the shape of smooth objects
in the scene.
 Iconic scene analysis
– Build a model of the scene or parts of the scene
 Feature-based scene analysis
– Extracts features of the scene needed by task
– Task-oriented or purposive vision

30. 30
6.5.1 Interpreting Lines and
Curves in the Image
 Interpreting the line drawing
– Association between scene properties and the
components of a line drawing
Trihedral vertex polyhedra
The scene to contain only
planar surfaces such that no
more than three surfaces
intersect in a point

31. 31
Three kinds of edges in Trihedral
vertex polyhedra (1/2)
 There are only three kinds of ways in which two planes can
intersect in a scene edge.
– Occlude
 One kind of edge is formed by two planes, with
one of them occluding the other.
 labeled in Fig. 6.15 with arrows ().
 the arrowhead pointing along the edge such
that surface doing the occluding is to the right of
the arrow.

32. 32
Three kinds of edges in Trihedral
vertex polyhedra (2/2)
– Blade
 Two planes can intersect such that both planes
are visible in the scene.
 Two surfaces form a convex edge.
 Labeled with pluses (+).
– Ford
 Edge is concave.
 Labeled with minus ()

33. 33
Labels for Lines at Junctions

34. 34
Line-labeling scene analysis (1/2)
1. Labeling all of the junctions in the image as V, W, Y,
or T junctions according to the shape of the junctions
in the image

35. 35
Line-labeling scene analysis (2/2)
2. Assign +, , or  labels to the lines in the image.
 An image line that connects two junctions must have a
consistent labeling.
 If there is no consistent labeling,
 there must have been some error in converting the image
into a line drawing.
 the scene must no have been one of trihedral polyhedra.
 Constraint satisfaction problem

36. 36
6.5.2 Model-Based Vision (1/2)
 If, we knew that the scene contained a parallelepiped (in Figure
6.15), we could attempt to fit a projection of a parallelepiped to
components of an image of this scene.
 A generalized
cylinders as building
blocks for model
construction
 Each cylinder has
9 parameters.

37. 37
Model-Based Vision (2/2)
 An example rough scene
reconstruction of a human
figure
– Hierarchical representation
– Each cylinder in the model
can be articulated into a
set of smaller cylinders

38. 38
6.6 Stereo Vision and Depth
Information
 Depth information can be obtained using stereo vision,
which based on triangulation calculations using two (or
more) images.
 Some depth information can be extracted from a
single image.
– The analysis of texture in the image can indicate that
some elements in the scene are closer than are
others.
– More precise depth information; If we know that a
perceived object is on the floor and the camera height
above the floor, we can calculate the distance to the
object.

39. 39
Depth Calculation from a Single
Image

40. 40
Stereo Vision
 Stereo vision uses triangulation.
 Two lenses whose centers are separated by a baseline, b.
 The image point of a scene point, at distance d, created by these
lenses.
 The angles of these image points from the lens centers, , .
 The optical axes are parallel, the image planes are coplanar, and the
scene point is in the same plane as that formed by two parallel optical
axes.

41. 41
Triangulation in Stereo Vision

42. 42
The main complication
 In scenes containing more than one point, it must be
established which pair of points in the two images
correspond to the same scene point.
 We must be able to identify a corresponding pixel in
the other image.  correspondence problem

43. 43
Techniques for correspondence
problem
 Geometric analysis reveals that we need only search
along one dimension (epipolar line).
 One-dimensional searches can be implemented by
cross-correlation of two image intensity profiles along
corresponding epipolar lines.
 We do not have to find correspondences between
individual pairs of image points but can do so
between pairs of larger image components, such as
lines.

44. 44
Assignments
 Page 111~112
– Ex.6.2, Ex. 6.3