This document describes a memory-based object recognition method that combines an associative memory with a Hough-like evidence combination technique. This allows it to overcome limitations of traditional memory-based methods like sensitivity to clutter and occlusion. The method uses local, semi-invariant "keys" extracted from images to access an associative memory and retrieve potential object identities and configurations. These hypotheses are then used as keys in a second associative memory to accumulate evidence. Experiments were conducted using this two-stage method with polyhedral and curved objects, extracting either segment chains or curve patches as keys. The method was able to recognize objects over a full range of rotations in 3D space.

1. Memory-Based Recognition for 3-D Objects
Randal C. Nelson
Department of Computer Science
University of Rochester
Rochester, NY 14627
nelson@cs.rochester.edu
Abstract
Memory-based object recognition methods work by
comparing an object against many representations
stored in a memory, and nding the closest match.
However matches are generally made to representa-
tions of complete objects, hence such methods tend
to be sensitive to clutter and occlusion and require
good global segmentation for success. We describe a
method that combines an associative memory with a
Hough-like evidence combination technique, allowing
local segmentation to be used. This resolves the clut-
ter and occlusion sensitivity of traditional memory-
based methods, without encountering the space prob-
lems that plague voting methods for high DOF prob-
lems. The method is based on the two stage use
of a general purpose associative memory and semi-
invariant local objects called keys. Experiments using
keys based on a curve segmentation process are re-
ported, using both polyhedral and curved objects.
Key Words: Object recognition, Memory-based
representations, Visual learning.
1 Introduction
Object recognition is probably the most researched
area of computer vision. The most successful work
to date has been using model-based systems. Notable
recent examples are 11, 10, 9, 6]. The 3D geomet-
ric models on which these systems are based are both
their strength and their weakness. 7, 8]. On the one
hand, explicit models provide a framework that al-
lows powerful geometric constraints to be utilized to
good e ect. On the other, model schemas are gener-
ally severely limited in the sort of objects that they
can represent, and obtaining the models is typically
a di cult and time-consuming process. There has
been a fair amount of work on automatic acquisition
of geometric models, mostly with range sensors, e.g.,
17, 18, 2] but also visually, for various representations
19, 3, 1, 5]. However, these techniques are limited to
a particular geometric schema, and even within their
domain, especially with visual techniques, their per-
formance is often unsatisfactory.
Memory-based object recognition methods have
been proposed in order to make recognition systems
more general, and more easily trainable from visual
Support for this work was provided by ARPA UMD sub-
contract Z8440902,ONR grant N00014-93-I-0221,and NSF IIP
Grant CDA-94-01142
data. Most of them essentially operate by comparing
an image representation of object appearance against
many prototype representations stored in a memory,
and nding the closest match. They have the advan-
tage of being fairly general, and often easily trainable.
In recent work, Poggiohas recognized wire objects and
faces 15, 4]. Rao and Ballard 16] describe an ap-
proach based on the memorization of the responses of
a set of steerable lters. Mel 12] takes a somewhat
similar approach using a database of stored feature
vectors representing multiple low-level cues. Murase
and Nayar 13] nd the major principal components
of an image dataset, and uses the projections of un-
known images onto these as indices into a recognition
memory.
In general, meory-based methods have proven to
be a useful technique; however because matches are
generally made to representations of complete objects,
these methods tend to be more sensitive to clutter and
occlusion than is desirable, and require good global
segmentation for success. Hough transform methods
(and other voting techniques), on the other hand, al-
low evidence from disconnected parts to be e ectively
combined, but the size of the voting space increases
exponentially with the number of degrees of visual
freedom. Di culties deriving from the size of this
space make it di cult to apply such techniques di-
rectly when more than about 3 DOF are involved,
thus limiting the use of the technique for 3D object
recognition, which generally involves at least 6 DOF.
We describe a method that, by combining an asso-
ciative memory with a Hough-like evidence combina-
tion technique, resolves both the clutter and occlusion
sensitivity of traditional memory-based methods, and
the space problems of voting methods for high DOF
problems. The method is based on the two stage use
of a general purpose associative memory. This stores
both semi-invariant, local objects called keys associ-
ated with object hypothesis, and object con guration
hypotheses associated with evidence. Entry of objects
into the memory is an active, automatic procedure.
Experiments using keys based on perceptual groups
of line segments, and on extracted boundary curves
are reported. Results are reported for a set of poly-
hedral objects, and for a set of curved objects, with
databases covering the full rotation space. This is in
contrast to some recent results e.g. Murase and Nayar
13] where essentially only one of the two out-of-plane
rotational degrees of freedom is spanned.

2. 2 The Method
2.1 Overview
As mentioned above, the method based on two
stage use of a general purpose associative memory.
What we describe here is the application of the tech-
nique to principle views recognition of rigid 3-D ob-
jects, but the underlying principles are not dependent
on rigid geometry,
The approach makes use of semi-invariantlocal ob-
jects we call keys. A key is any robustly extractable
part or feature that has su cient informationcontent
to specify a con guration of an associated object plus
enough additional parameters to provide e cient in-
dexing and meaningfulveri cation. Con guration is a
general term for descriptors that provide information
aboutwhere inappearance space animageofanobject
is situated. For rigid objects, con guration generally
implies location and orientation, but more general in-
terpretations can be used for other object types. Semi-
invariant means that over all con gurations in which
the object of interest will be encountered, a matchable
form of the feature will be present a signi cant pro-
portion of the time. Robustly extractable means that
in any scene of interest containing the object, the fea-
ture will be in the N best features found a signi cant
proportion of the time.
The basic idea is to utilize an associative memory
organized so that access via a key feature evokes asso-
ciated hypotheses for the identity and con guration of
allobjects that could have produced it. These hypoth-
esis are fed into a second stage associative memory,
keyed by the con guration, which maintains a prob-
abilistic estimate of the likelihood of each hypothesis
based on statistics about the occurrence of the keys in
the primary database. The idea is similar to a multi-
dimensionalHough transform without the space prob-
lems. In our case, since 3-D objects are represented by
a set of views, the con gurations represent two dimen-
sional transforms. E cient access to the associative
memories is achieved using a hashing scheme.
The approach has several advantages. First, be-
cause it is based on a merged percept of local features
rather than global properties, the method is robust
to occlusion and background clutter, and does not re-
quire prior segmentation. This is an advantage over
systems based on principal components templateanal-
ysis, which are sensitive to occlusion and clutter. Sec-
ond, entry of objects into the memory is an active,
automatic procedure. Essentially, the system explores
the object visually from di erent viewpoints, accumu-
lating 2-D views, until it has seen enough not to mix
it up with any other object it knows about. Third, the
method lends itself naturally to multi-modal recogni-
tion. Because there is no single, global structure for
the model, evidence from di erent kinds of keys can
be combined as easily as evidence from multiple keys
of the same type. The only requirement is that the
con guration descriptions evoked by the di erent keys
have enough commonstructure to allowevidence com-
bination procedures to be used. This is an advantage
over conventional alignment techniques, which typi-
cally require a prior 3-D model of the object. Finally,
the probabilistic nature of the evidence combination
scheme, coupled with the formal de nitions for semi-
invariance and robustness allow quantitative predic-
tions of the reliability of the system to be made.
2.2 General associative memory
Our approach is based on an e cient associative
memory,The basic operation we need is partial match
associationover heterogeneous keys. More speci cally,
we want a structure into which we can store and access
(key, association) pairs where the key and association
objects may be any of a number of disparate types.
Associated with each object type employed as a key is
a distance metric. The ideal query operation takes a
reference key and returns all stored (key, association)
pairs where the key is of the correct type and within
a speci ed distance of the reference key in the appro-
priate metric. In practice, this ideal may have to be
modi ed somewhat for e ciency reasons.
In the current implementation, the memory is just
a large array of buckets each of which can hold a vari-
able number of (key, association) pairs. This allows a
number of di erent access schemes to coexist. In par-
ticular, hashing, array indexing and tree search can
all be implemented e ciently. Associated with each
key type are functions de ning a distance metric and
a search procedure for locating keys in the memory.
This allows a large amount of exibility in the system,
and permits new key types to be added in a modular
fashion.
2.3 Key Features
The recognition technique is based on the the as-
sumption that robustly extractable, semi-invariant
keys can be e ciently recovered from image data.
More speci cally, the keys must posses the following
characteristics. First, they must be complex enough
not only to specify the con guration the object, but to
have parameters left over that can be used for index-
ing. Second, the keys must have a substantial prob-
ability of detection if the object containing them oc-
cupies the region of interest (robustness). Third, the
index parameters must change relatively slowly as the
object con guration changes (semi-invariance). From
a computational standpoint, true invariance is desir-
able, and a lot of research has gone into looking for in-
variant features. Unfortunately, such features seem to
be hardtodesign, especially for2-Dprojections ofgen-
eral 3-Dobjects. Many classicalfeatures do not satisfy
these criteria. Line segments are not su ciently com-
plex, full object contours are not robustly extractable,
and simple templates are not semi-invariant.
A basic con ict that must be resolved is that be-
tween feature complexity and robust detectability. In
order to reduce multiplematches, key features mustbe
fairly complex. However, if we consider complex fea-
tures as arbitrary combinations of simpler ones, then
the number of potential high-level features undergoes
a combinatorial increase as the complexity increases.
This is clearly undesirable from the standpoint of ro-
bust detectability, as we do not wish to consider or
store exponentially many possibilities. The solution is
not to use arbitrary combinations, but to base the
higher level feature groups on structural heuristics
such as adjacency and good continuation. Such per-

3. ceptual grouping processes have been extensively re-
searched in the last few years.
Our current implementation is designed to recog-
nize 3-D objects on the basis of their shape, using a
set of 2-D views as the underlying representation. We
ran experiments with two type of key features: seg-
ment chains, and curve patches. The segment chains,
which were used onlyon the polyhedral database, con-
sisted of perceptual groups of three connected seg-
ments. The angles and length ratios provide matching
and indexing information. The curve patches consist
of curve orientation templates normalized by robust
curve fragments. Speci cally, a curve- nding algo-
rithmis run on animage,producing aset ofsegmented
contour fragments broken at points of high curvature.
The strongest curves are selected as normalizing base
curves, and a xed-size template constructed with the
endpoints ofthe base curve occupyingcanonicalpoints
in the template. All image curves that intersect the
normalized template are mapped into it with a code
specifying their orientation relative to the base curve.
Global curve properties such as total curvature and
compactness serve as index parameters. Final match-
ing of a candidate template involves taking the model
key curve points and verifying that a curve point with
similar orientation lies nearby in the candidate tem-
plate. Essentially this amounts to directional correla-
tion.
3 Recognition Algorithms
The basic recognition procedure consists of four
steps. First, potential key features are extracted from
the image using low and intermediate level visual rou-
tines. In the second step, these keys are used to ac-
cess the associative memory and retrieve information
about what objects could have produced them, and in
what relative con guration. The third step uses this
information, in conjunction with geometric parame-
ters factored out of the key features such as position,
orientation, and scale, to produce hypotheses about
the identity and con guration of potential objects. Fi-
nally, these hypotheses are themselves used as keys
into a second stage associative memory, which is used
to accumulate evidence for the various hypotheses.
In the nal step, an important issue is the method
of combining evidence. The simplest technique is to
use an elementary voting scheme - each piece of evi-
dence contributes equally to the total. This is clearly
not well founded, as a feature that occurs in many
di erent situations is not as good an indicator of the
presence of an object as one that is unique to it. An
evidence scheme that takes this into account would
probably display improved performance. The ques-
tion is how to evaluate the quality of various pieces of
evidence. An obvious approach in our case is to use
statistics computed over the information contained in
the associative memory to evaluate the quality of a
piece of information. Having said this, it is clear that
the optimal quality measure, which would rely on the
full joint probability distribution over keys, objects
and con gurations is infeasible to compute, and we
must use some approximation.
A simple example would be to use the rst or-
der feature frequency distribution over the entire
database, and this is what we do. The actual al-
gorithm is to accumulate evidence proportional to
log(1 + 1=(kx)) where x is the probability of making
the particular matching observation as approximated
from database statistics, and k is a proportionality
constant that attempts to estimate the actual geomet-
ric probabilityassociated with the prediction of a pose
froma key. The underlying modelis that the evidence
represents the log of the reciprocal of the probability
that the particular combination of features is due to
chance. The procedure used makes an independence
assumption which is unwarranted in the real world,
with the result that the evidence values actually ob-
tained are serious overestimates if interpreted as actu-
ally probabilities. However, the rank ordering of the
values, which is allthat is importantinthe recognition
system is robust to this distortion.
In the above discussion we have assumed that the
associative memory already existed in the requisite
form. However, one of the primary attractions of a
memory-based recognition system is that it can be
trained e ciently from image data. The basic pro-
cess of model acquisition is simply a matter of provid-
ing images of the object to the system, running the
key detection procedures on these images, and storing
the resulting (key, association) pairs. The number of
images needed may vary from one, for simple 2-D ap-
plications, to several tens for rigid object recognition,
and possibly more for complicated non-rigid objects.
The process is e cient, and essentially runs in time
proportional to the number of pairs stored in mem-
ory. This is in contrast to many learning algorithms
that scale poorly with the the number of stored items.
(Actually, indexed memory building process are apt
to scale as N log(N), for very large numbers of items.
However, since the processing for all databases run so
far is dominated by the key-feature extraction image
processing, the complexityhas essentially been linear).
4 Experiments
Using the principles described above, we imple-
mented a memory-based recognition system for poly-
hedral objects using rst segment chains, and then
curve patches as the basic keys. We then tested the
curve patch system on a set of complex curved ob-
jects. Component segments were extracted using a
stick-growing method developed recently at Rochester
14], and organized into chains. A modi ed version of
the same algorithm allowed us to extract curves as
well. For objects entered into the database, the best
10 key features were selected to represent the object.
The thresholds on the distance metrics between fea-
tures was adjusted so that it would tolerate approx-
imately 15-20 degrees deviation in the appearance of
a frontal plane (less for oblique ones). The practical
considerations leading to this selection were to allow
the system to discriminate pentagons from hexagons
without requiring more than about 100 views for an
object.
4.1 Experiments on Polyhedral Objects
We rst performed experiments using a set of 7
polyhedral objects from a child's toy. Some of these
are shown in Figure 1. Not shown is a hexagon.

4. Figure 1: Some polyhedral objects used in the test set
The objects are not simple prisms, but have an H-
shaped cross section which produces produces inter-
esting edges and shadow e ects.
We obtained a training database of approximately
150 views of these objects from di erent directions
ranging from 12 for the hexagon, to 60 for the trape-
zoid, and covering all viewing angles except straight-
on from the side, since from that point of view a num-
ber of the objects are indistinguishable without mea-
surements accurate to a few percent. The variation in
the number of views needed is due to varying degrees
of symmetry in the objects. All training images were
acquired under normal room illumination, with the
objects in isolation against a dark background. The
images were used to compile both segment-chain and
curve patch databases.
We then subjected the recognition system to a se-
ries of increasinglystringent tests. Recallthat the geo-
metricdesignofthe geometricindexingsystemensures
invariance to 2-D translation, rotation, and scale. In-
variance to out-of-plane rotations is provided by the
combination of slightly exible match criteria for the
key features coupled with multiple views. Robustness
against clutter and occlusion is provided by the rep-
resentation in terms of multiple local features. The
experiments were designed to test various aspects of
this design. A basic assumption made during these
tests is that some other process has isolated a region of
the image where a recognizable object may occur. We
do not assume prior segmentation, but we do assume
that only one, or at most a few objects of interest (as
opposed to tens or hundreds) will occur in a window
handed to the system. The system has a certain ca-
pability to provide a don't know" result and tends to
do this if no known objects are present. However, we
have not statistically grounded this ability and hence
the results reported here should be considered to be
forced choice experiments.
For the rst test, we acquired additional views of
the isolated objects, from viewpoints intermediate be-
tween the ones in the database. The idea here is to
test the 3-D rotation invariance. No errors were made
in these test cases, even between similar objects such
as the square and the trapezoid, or the pentagon and
Figure 2: View of a group of objects
Figure 3: A set of windows containing objects and mi-
nor clutter. The central object was correctly identi ed
in all these cases.
the hexagon. These results allow us to say that the
system is probably at least 95 percent accurate in sit-
uations of this type. Results from other tests lead
us to believe that the actual performance is, in fact,
somewhat better.
In the second test we took a number of images con-
taining multiple objects viewed from modest angles
(45 degrees or less from overhead) under normallight-
ing against a dark background. An example is shown
in Figure 2. We then supplied the system with win-
dows containing one object and parts of others. Since
the system performs no explicit segmentation of its
own, the intent of this experiment is to test robust-
ness against minor clutter. Examples of the sort of
windows passed to the system are shown in Figure 3.
In twenty plus tests, we observed no errors due to clut-
ter. We also tried examples with two objects in the
window. In this case, the system typically identi ed
one of the objects, and when asked what else was there
identi ed the second as well.
The third experiment was a more severe clutter
test. Here we took pictures of di erent objects held in
a robot hand at various angles. Examples are shown
inFigure 4. This was a harder problemforour system,
and we obtained recognition rates on the order of 75
to 90 percent with the better performance being ob-
tained with the curve-patch keys. These experiments
produced a signi cant number of failures. On analy-
sis, we found that the primary reason for failure was
not crosstalk in the memory caused by clutter, but
poor performance of the low-level feature identi ca-
tion process caused by the added complexity in the
image. Good segmentation is still critical, even when

5. Figure 4: A set of windows containing objects help by
a robot hand representing moderate clutter.
Figure 5: An image with serious clutter.
local features are used.
Finally, we tested the polyhedral recognition sys-
tem on windows from the image shown in Figure
5, which contains fairly serious clutter. The sys-
tem based on segment chains performed essentially at
chance level, while the curve patches worked at around
80 percent. In this case the system tended to confuse
the square and the trapezoid. These gures are so sim-
ple that losing a few of the main edges due to clutter
substantially reduced their discriminability.
In the various tests, the curve patch keys gener-
ally performed as well as or better than the segment
chains. Thisis asexpected, since the the curve patches
can represent allthe features representable by segment
chains, and some others as well. Also, since only a sin-
gle segmented boundary fragment is needed to key a
curve patch, features are less likely to be missed due
to failure in the low-level segmentation. On a typical
window containing one object plus clutter, the index-
ing process in the full database took less than a sec-
ond on a SPARC5 for chains. The low level processing
could take a few seconds, depending complexityof the
image. The matching look longer for the patches, 10
to 30 seconds, since they are more complex, but was
on the order of the low-level processing.
4.2 Experiments on Curved Objects
We have also carried out preliminary experiments
using the curve-patch based system to identify com-
plex curved objects. To carry out these experiments,
we compiled a database of curved objects, with im-
ages taken approximately every 20 degrees over the
Figure 6: Examples of curved objects learned by the
system
entire viewing sphere. For cases without symmetry,
this amounts to approximately 100 images per ob-
ject. The model pictures were taken against a black
background, using di use lighting in order to elimi-
nate specular highlights as much as possible. A full
3-D database was compiled for the six objects shown
in Figure 6, and tests were run as for the polyhedral
case.
In the rst test, as for the polyhedral objects, we
took pictures of isolated objects under the samecondi-
tions as the training images were taken, only at angles
in between the views contributing to the database.
The system performed fairly well in this situation,
with an accuracy somewhere upwards of 90-95 per-
cent.
The second test utilized groups of objects, still on
a dark background, but now under normal lighting
so that specularities and shadows become much more
evident, and at varying scales, which can a ect the
segmentation produced by the curve nder. An ex-
ample of such an image is shown in Figure 7. The
windows passed to the recognizer typically contained
clutter consisting of parts of other objects. As in the
polyhedral case, this sort of clutter, which does not
adversely a ect the performance of the curve extrac-
tor, had no noticeable e ect. For this test the perfor-
mance was slightlydegraded to about 90 percent, with
corruption of the segmentation process accounting for
most of the di erence.
5 Conclusions and Future Work
In this paper we have argued for a memory-access
interpretation of recognition, and proposed a general
framework for memory-based recognition using a 2-
stage association process. We have illustrated the
concept by implementinga memory-based recognition
system for 3-D polyhedral objects using chains of line
segments and curve patches as memorykeys. The sys-
tem actually performs quite well for a small database
of 3-D shapes, and exhibits a certain amount of ro-
bustness against clutter and occlusion. When the al-
gorithm fails, it is not due to crosstalk in the memory,
but to failure of the low-level processes to extract ro-
bust features. We are currently engaged in embedding
the system into a robotic manipulationsystem that we
will use for assembly tasks. We also plan to incorpo-

6. Figure 7: Curved objects with minor clutter.
rate multi-modalfeatures into the database, including
color and texture as well as shape information. We an-
ticipate that this will give us a capability to recognize
less well structured objects such as leaves or clothing
in addition to objects having a strictly de ned shape.
References
1] N. Ayache and O. Faugeras. Hyper: a new
approach for the recognition and positioning of
two-dimensional objects. IEEE Trans. PAMI,
8(1):44{54, January 1986.
2] A. F. Bobick and R. C. Bolles. Representation
space: An approach to the integration of visual
information. In Proc. CVPR, pages 492{499, San
Diego CA, June 1989.
3] R. C. Bolles and R. A. Cain. Recognizing and
localizing partially visible objects: The local-
features-focus method. International Journal of
Robotics Research, 1(3):57{82, Fall 1982.
4] R. Brunelli and T. Poggio. Face recognition:
Features versus templates. IEEE Trans. PAMI,
15(10):1042{1062, 1993.
5] F.Stein and G. Medioni. E cient 2-dimensional
object recgnition. In Proc. ICPR, pages 13{17,
Atlantic City NJ, June 1990.
6] W. E. L. Grimson. Object Recognition by Com-
puter: The role of geometric constraints. The
MIT Press, Cambridge, 1990.
7] W. E. L. Grimson and D. P. Huttenlocher. On
the sensitivity of geometric hashing. In 3rd Inter-
national Conference on Computer Vision, pages
334{338, 1990.
8] W. E. L. Grimson and D. P. Huttenlocher. On
the sensitivity of the hough transform for object
recognition. IEEE PAMI, 12(3):255{274, 1990.
9] D. P. Huttenlocher and S. Ullman. Recognizing
solid objects by alignment with an image. Inter-
national Journal of Computer Vision, 5(2):195{
212, 1990.
10] Y. Lamdan and H. J. Wolfson. Geometric hash-
ing: A general and e cient model-based recogni-
tion scheme. In Proc. International Conference
on Computer Vision, pages 238{249, Tampa FL,
December 1988.
11] D. G. Lowe. Three-dimensional object recogni-
tion from single two-dimensional images. Arti -
cial Intelligence, 31:355{395, 1987.
12] B. Mel. Object classi cation with high-
dimensional vectors. In Proc. Telluride Work-
shop on Neuromorphic Engineering, Telluride
CO, July 1994.
13] H. Murase and S. K. Nayar. Learning and recog-
nition of 3d objects from appearance. In Proc.
IEEE Workshop on Qualitative Vision, pages 39{
50, 1993.
14] R. C. Nelson. Finding line segments by stick
growing. IEEE Trans PAMI, 16(5):519{523,May
1994.
15] T. Poggioand S. Edelman. A network that learns
to recognize three-dimensional objects. Nature,
343:263{266, 1990.
16] R. P. Rao. Top-down gaze targeting for space-
variant active vision. In Proc. ARPA Image Un-
derstanding Workshop, pages 1049{1058, Mon-
terey CA, November 1994.
17] R. K. Ruud M. Bolle and D. Sabbah. Primitive
shape extraction from range data. In Proc. IEEE
Workshop on Computer Vision, pages 324{326,
Miami FL, Nov-Dec 1989.
18] F. Solina and R. Bajcsy. Recovery of parameteric
models from range images. IEEE Trans. PAMI,
12:131{147, February 1990.
19] S. Ullman and R. Basri. Recognition by linear
combinations of models. IEEE Trans. PAMI,
13(10), 1991.