Shape matching and object recognition using shape context belongie pami02
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 24, APRIL 2002 509 Shape Matching and Object Recognition Using Shape Contexts Serge Belongie, Member, IEEE, Jitendra Malik, Member, IEEE, and Jan Puzicha AbstractÐWe present a novel approach to measuring similarity between shapes and exploit it for object recognition. In our framework, the measurement of similarity is preceded by 1) solving for correspondences between points on the two shapes, 2) using the correspondences to estimate an aligning transform. In order to solve the correspondence problem, we attach a descriptor, the shape context, to each point. The shape context at a reference point captures the distribution of the remaining points relative to it, thus offering a globally discriminative characterization. Corresponding points on two similar shapes will have similar shape contexts, enabling us to solve for correspondences as an optimal assignment problem. Given the point correspondences, we estimate the transformation that best aligns the two shapes; regularized thin-plate splines provide a flexible class of transformation maps for this purpose. The dissimilarity between the two shapes is computed as a sum of matching errors between corresponding points, together with a term measuring the magnitude of the aligning transform. We treat recognition in a nearest-neighbor classification framework as the problem of finding the stored prototype shape that is maximally similar to that in the image. Results are presented for silhouettes, trademarks, handwritten digits, and the COIL data set. Index TermsÐShape, object recognition, digit recognition, correspondence problem, MPEG7, image registration, deformable templates. æ1 INTRODUCTIONC ONSIDER the two handwritten digits in Fig. 1. Regarded as vectors of pixel brightness values and comparedusing LP norms, they are very different. However, regarded coordinate transformations, as illustrated in Fig. 2. In the computer vision literature, Fischler and Elschlager  operationalized such an idea by means of energy mini-as shapes they appear rather similar to a human observer. mization in a mass-spring model. Grenander et al. Our objective in this paper is to operationalize this notion of developed these ideas in a probabilistic setting. Yuille shape similarity, with the ultimate goal of using it as a basis developed another variant of the deformable templatefor category-level recognition. We approach this as a three- concept by means of fitting hand-crafted parametrizedstage process: models, e.g., for eyes, in the image domain using gradient descent. Another well-known computational approach in 1. solve the correspondence problem between the two this vein was developed by Lades et al.  using elastic shapes, 2. use the correspondences to estimate an aligning graph matching. transform, and Our primary contribution in this work is a robust and 3. compute the distance between the two shapes as a simple algorithm for finding correspondences between sum of matching errors between corresponding shapes. Shapes are represented by a set of points sampled points, together with a term measuring the magni- from the shape contours (typically 100 or so pixel locations tude of the aligning transformation. sampled from the output of an edge detector are used). There is nothing special about the points. They are notAt the heart of our approach is a tradition of matching required to be landmarks or curvature extrema, etc.; as weshapes by deformation that can be traced at least as far back use more samples, we obtain better approximations to theas DArcy Thompson. In his classic work, On Growth and underlying shape. We introduce a shape descriptor, theForm , Thompson observed that related but not identical shape context, to describe the coarse distribution of the rest ofshapes can often be deformed into alignment using simple the shape with respect to a given point on the shape. Finding correspondences between two shapes is then equivalent to finding for each sample point on one shape. S. Belongie is with the Department of Computer Science and Engineering, AP&M Building, Room 4832, University of California, San Diego, La the sample point on the other shape that has the most Jolla, CA 92093-0114. E-mail: firstname.lastname@example.org. similar shape context. Maximizing similarities and enfor-. J. Malik is with the Computer Science Division, University of California at cing uniqueness naturally leads to a setup as a bipartite Berkeley, 725 Soda Hall, Berkeley, CA 94720-1776. E-mail: email@example.com. graph matching (equivalently, optimal assignment) pro-. J. Puzicha is with RecomMind, Inc., 1001 Camelia St., Berkeley, CA blem. As desired, we can incorporate other sources of 94710. E-mail: firstname.lastname@example.org. matching information readily, e.g., similarity of localManuscript received 09 Apr. 2001; revised 13 Aug. 2001; accepted 14 Aug. appearance at corresponding points.2001. Given the correspondences at sample points, we extendRecommended for acceptance by J. Weng.For information on obtaining reprints of this article, please send e-mail to: the correspondence to the complete shape by estimating email@example.com, and reference IEEECS Log Number 113957. aligning transformation that maps one shape onto the other. 0162-8828/02/$17.00 ß 2002 IEEE
510 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 24, APRIL 2002 databases including handwritten digits and pictures of 3D objects in Section 6. We conclude in Section 7. 2 PRIOR WORK ON SHAPE MATCHINGFig. 1. Examples of two handwritten digits. In terms of pixel-to-pixel Mathematicians typically define shape as an equivalencecomparisons, these two images are quite different, but to the human class under a group of transformations. This definition isobserver, the shapes appear to be similar. incomplete in the context of visual analysis. This only tells us when two shapes are exactly the same. We need moreA classic illustration of this idea is provided in Fig. 2. The than that for a theory of shape similarity or shape distance.transformations can be picked from any of a number of The statisticians definition of shape, e.g., Bookstein  orfamiliesÐwe have used Euclidean, affine, and regularized Kendall , addresses the problem of shape distance, butthin plate splines in various applications. Aligning shapes assumes that correspondences are known. Other statisticalenables us to define a simple, yet general, measure of shape approaches to shape comparison do not require correspon-similarity. The dissimilarity between the two shapes can dencesÐe.g., one could compare feature vectors containingnow be computed as a sum of matching errors between descriptors such as area or momentsÐbut such techniquescorresponding points, together with a term measuring the often discard detailed shape information in the process.magnitude of the aligning transform. Shape similarity has also been studied in the psychology Given such a dissimilarity measure, we can use literature, an early example being Goldmeier .nearest-neighbor techniques for object recognition. Philo- An extensive survey of shape matching in computersophically, nearest-neighbor techniques can be related to vision can be found in , . Broadly speaking, there areprototype-based recognition as developed by Rosch  two approaches: 1) feature-based, which involve the use ofand Rosch et al. . They have the advantage that a spatial arrangements of extracted features such as edgevector space structure is not requiredÐonly a pairwise elements or junctions, and 2) brightness-based, which makedissimilarity measure. more direct use of pixel brightnesses. We demonstrate object recognition in a wide variety ofsettings. We deal with 2D objects, e.g., the MNIST data set 2.1 Feature-Based Methodsof handwritten digits (Fig. 8), silhouettes (Figs. 11 and 13) A great deal of research on shape similarity has been doneand trademarks (Fig. 12), as well as 3D objects from the using the boundaries of silhouette images. Since silhouettesColumbia COIL data set, modeled using multiple views do not have holes or internal markings, the associated(Fig. 10). These are widely used benchmarks and our boundaries are conveniently represented by a single-closedapproach turns out to be the leading performer on all the curve which can be parametrized by arclength. Early workproblems for which there is comparative data. used Fourier descriptors, e.g., , . Blums medial axis We have also developed a technique for selecting the transform has led to attempts to capture the part structure ofnumber of stored views for each object category based on its the shape in the graph structure of the skeleton by Kimia,visual complexity. As an illustration, we show that for the Zucker and collaborators, e.g., Sharvit et al. . The3D objects in the COIL-20 data set, one can obtain as low as 1D nature of silhouette curves leads naturally to dynamic2.5 percent misclassification error using only an average of programming approaches for matching, e.g., , which usesfour views per object (see Figs. 9 and 10). the edit distance between curves. This algorithm is fast and The structure of this paper is as follows: We discuss invariant to several kinds of transformation including somerelated work in Section 2. In Section 3, we describe our articulation and occlusion. A comprehensive comparison ofshape-matching method in detail. Our transformation different shape descriptors for comparing silhouettes wasmodel is presented in Section 4. We then discuss the done as part of the MPEG-7 standard activity , with theproblem of measuring shape similarity in Section 5 and leading approaches being those due to Latecki et al.  anddemonstrate our proposed measure on a variety of Mokhtarian et al. .Fig. 2. Example of coordinate transformations relating two fish, from DArcy Thompsons On Growth and Form . Thompson observed that similarbiological forms could be related by means of simple mathematical transformations between homologous (i.e., corresponding) features. Examples ofhomologous features include center of eye, tip of dorsal fin, etc.
BELONGIE ET AL.: SHAPE MATCHING AND OBJECT RECOGNITION USING SHAPE CONTEXTS 511 Silhouettes are fundamentally limited as shape descrip- when used in a probabilistic framework . Murase andtors for general objects; they ignore internal contours and Nayar applied these ideas to 3D object recognition .are difficult to extract from real images. More promising are Several authors have applied discriminative classificationapproaches that treat the shape as a set of points in the methods in the appearance-based shape matching frame-2D image. Extracting these from an image is less of a work. Some examples are the LeNet classifier , aproblemÐe.g., one can just use an edge detector. Hutten- convolutional neural network for handwritten digit recogni-locher et al. developed methods in this category based on tion, and the Support Vector Machine (SVM)-based methodsthe Hausdorff distance ; this can be extended to deal of  (for discriminating between templates of pedestrianswith partial matching and clutter. A drawback for our based on 2D wavelet coefficients) and ,  (for hand-purposes is that the method does not return correspon- written digit recognition). The MNIST database of hand-dences. Methods based on Distance Transforms, such as, are similar in spirit and behavior in practice. written digits is a particularly important data set as many The work of Sclaroff and Pentland  is representative different pattern recognition algorithms have been tested onof the eigenvector- or modal-matching based approaches; it. We will show our results on MNIST in Section 6.1.see also , , . In this approach, sample points inthe image are cast into a finite element spring-mass model 3 MATCHING WITH SHAPE CONTEXTSand correspondences are found by comparing modes ofvibration. Most closely related to our approach is the work In our approach, we treat an object as a (possibly infinite)of Gold et al.  and Chui and Rangarajan , which is point set and we assume that the shape of an object isdiscussed in Section 3.4. essentially captured by a finite subset of its points. More There have been several approaches to shape recognition practically, a shape is represented by a discrete set of pointsbased on spatial configurations of a small number of sampled from the internal or external contours on thekeypoints or landmarks. In geometric hashing , these object. These can be obtained as locations of edge pixels asconfigurations are used to vote for a model without found by an edge detector, giving us a set fpI ; F F F ; pn g,explicitly solving for correspondences. Amit et al.  train pi P s P , of n points. They need not, and typically will not, decision trees for recognition by learning discriminative correspond to key-points such as maxima of curvature orspatial configurations of keypoints. Leung et al. , inflection points. We prefer to sample the shape withSchmid and Mohr , and Lowe  additionally use roughly uniform spacing, though this is also not critical.1gray-level information at the keypoints to provide greater Figs. 3a and 3b show sample points for two shapes.discriminative power. It should be noted that not all objects Assuming contours are piecewise smooth, we can obtainhave distinguished key points (think of a circle for as good an approximation to the underlying continuousinstance), and using key points alone sacrifices the shape shapes as desired by picking n to be sufficiently large.information available in smooth portions of object contours. 3.1 Shape Context2.2 Brightness-Based Methods For each point pi on the first shape, we want to find theBrightness-based (or appearance-based) methods offer a ªbestº matching point qj on the second shape. This is acomplementary view to feature-based methods. Instead of correspondence problem similar to that in stereopsis.focusing on the shape of the occluding contour or other Experience there suggests that matching is easier if oneextracted features, these approaches make direct use of the uses a rich local descriptor, e.g., a gray-scale window or agray values within the visible portion of the object. One can vector of filter outputs , instead of just the brightness atuse brightness information in one of two frameworks. a single pixel or edge location. Rich descriptors reduce the In the first category, we have the methods that explicitly ambiguity in matching. As a key contribution, we propose a novel descriptor, thefind correspondences/alignment using grayscale values. shape context, that could play such a role in shape matching.Yuille  presents a very flexible approach in that Consider the set of vectors originating from a point to allinvariance to certain kinds of transformations can be built other sample points on a shape. These vectors express theinto the measure of model similarity, but it suffers from the configuration of the entire shape relative to the referenceneed for human-designed templates and the sensitivity to point. Obviously, this set of n À I vectors is a richinitialization when searching via gradient descent. Lades et description, since as n gets large, the representation of theal.  use elastic graph matching, an approach that shape becomes exact.involves both geometry and photometric features in the The full set of vectors as a shape descriptor is much tooform of local descriptors based on Gaussian derivative jets. detailed since shapes and their sampled representation mayVetter et al.  and Cootes et al.  compare brightness vary from one instance to another in a category. We identifyvalues but first attempt to warp the images onto one the distribution over relative positions as a more robust andanother using a dense correspondence field. compact, yet highly discriminative descriptor. For a point pi The second category includes those methods that build on the shape, we compute a coarse histogram hi of theclassifiers without explicitly finding correspondences. In relative coordinates of the remaining n À I points,such approaches, one relies on a learning algorithm havingenough examples to acquire the appropriate invariances. In hi k 5fq T pi X q À pi P in kg: Ithe area of face recognition, good results were obtained usingprincipal components analysis (PCA) ,  particularly 1. Sampling considerations are discussed in Appendix B.
512 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 24, APRIL 2002Fig. 3. Shape context computation and matching. (a) and (b) Sampled edge points of two shapes. (c) Diagram of log-polar histogram bins used incomputing the shape contexts. We use five bins for log r and 12 bins for . (d), (e), and (f) Example shape contexts for reference samples marked by; Å; / in (a) and (b). Each shape context is a log-polar histogram of the coordinates of the rest of the point set measured using the reference point asthe origin. (Dark=large value.) Note the visual similarity of the shape contexts for and Å, which were computed for relatively similar points on the twoshapes. By contrast, the shape context for / is quite different. (g) Correspondences found using bipartite matching, with costs defined by the Pdistance between histograms.This histogram is defined to be the shape context of pi . We 3.2 Bipartite Graph Matchinguse bins that are uniform in log-polar2 space, making the Given the set of costs Cij between all pairs of points pi ondescriptor more sensitive to positions of nearby sample the first shape and qj on the second shape, we want topoints than to those of points farther away. An example is minimize the total cost of matching,shown in Fig. 3c. À Á Consider a point pi on the first shape and a point qj on H C pi ; q i Pthe second shape. Let Cij C pi ; qj denote the cost of imatching these two points. As shape contexts are subject to the constraint that the matching be one-to-one, i.e., distributions represented as histograms, it is natural to is a permutation. This is an instance of the square assignmentuse the P test statistic: (or weighted bipartite matching) problem, which can be solved in O N Q time using the Hungarian method . In our I hi k À hj kP K Cij C pi ; qj ; experiments, we use the more efficient algorithm of . The P kI hi k hj k input to the assignment problem is a square cost matrix withwhere hi k and hj k denote the KEin normalized entries Cij . The result is a permutation i such that (2) ishistogram at pi and qj , respectively.3 minimized. The cost Cij for matching points can include an In order to have robust handling of outliers, one can addadditional term based on the local appearance similarity at ªdummyº nodes to each point set with a constant matchingpoints pi and qj . This is particularly useful when we are cost of d . In this case, a point will be matched to acomparing shapes derived from gray-level images instead ªdummyº whenever there is no real match available atof line drawings. For example, one can add a cost based on smaller cost than d . Thus, d can be regarded as a thresholdnormalized correlation scores between small gray-scale parameter for outlier detection. Similarly, when the numberpatches centered at pi and qj , distances between vectors of of sample points on two shapes is not equal, the cost matrixfilter outputs at pi and qj , tangent orientation difference can be made square by adding dummy nodes to the smallerbetween pi and qj , and so on. The choice of this appearance point set.similarity term is application dependent, and is driven by 3.3 Invariance and Robustnessthe necessary invariance and robustness requirements, e.g., A matching approach should be 1) invariant under scalingvarying lighting conditions make reliance on gray-scale and translation, and 2) robust under small geometricalbrightness values risky. distortions, occlusion and presence of outliers. In certain 2. This choice corresponds to a linearly increasing positional uncertainty applications, one may want complete invariance underwith distance from pi , a reasonable result if the transformation between the rotation, or perhaps even the full group of affine transfor-shapes around pi can be locally approximated as affine. 3. Alternatives include Bickels generalization of the Kolmogorov- mations. We now evaluate shape context matching by theseSmirnov test for 2D distributions , which does not require binning. criteria.
BELONGIE ET AL.: SHAPE MATCHING AND OBJECT RECOGNITION USING SHAPE CONTEXTS 513 Invariance to translation is intrinsic to the shape context 3D points called the ªspin image.º A spin image is adefinition since all measurements are taken with respect to 2D histogram formed by spinning a plane around a normalpoints on the object. To achieve scale invariance we vector on the surface of the object and counting the pointsnormalize all radial distances by the mean distance that fall inside bins in the plane. As the size of this plane isbetween the nP point pairs in the shape. relatively small, the resulting signature is not as informative Since shape contexts are extremely rich descriptors, they as a shape context for purposes of recovering correspon-are inherently insensitive to small perturbations of parts of dences. This characteristic, however, might have the trade-the shape. While we have no theoretical guarantees here, off of additional robustness to occlusion. In another relatedrobustness to small nonlinear transformations, occlusions work, Carlsson  has exploited the concept of orderand presence of outliers is evaluated experimentally in structure for characterizing local shape configurations. InSection 4.2. this work, the relationships between points and tangent In the shape context framework, we can provide for lines in a shape are used for recovering correspondences.complete rotation invariance, if this is desirable for anapplication. Instead of using the absolute frame forcomputing the shape context at each point, one can use a 4 MODELING TRANSFORMATIONSrelative frame, based on treating the tangent vector at each Given a finite set of correspondences between points on twopoint as the positive xExis. In this way, the reference frame shapes, one can proceed to estimate a plane transformationturns with the tangent angle, and the result is a completely T X s P À P that may be used to map arbitrary points from 3srotation invariant descriptor. In Appendix A, we demon- one shape to the other. This idea is illustrated by thestrate this experimentally. It should be emphasized though warped gridlines in Fig. 2, wherein the specified corre-that, in many applications, complete invariance impedes spondences consisted of a small number of landmark pointsrecognition performance, e.g., when distinguishing 6 from 9 such as the centers of the eyes, the tips of the dorsal fins,rotation invariance would be completely inappropriate. etc., and T extends the correspondences to arbitrary points.Another drawback is that many points will not have well- We need to choose T from a suitable family ofdefined or reliable tangents. Moreover, many local appear- transformations. A standard choice is the affine model, i.e.,ance features lose their discriminative power if they are notmeasured in the same coordinate system. T x Ax o Q Additional robustness to outliers can be obtained by with some matrix A and a translational offset vector oexcluding the estimated outliers from the shape context parameterizing the set of all allowed transformations. Then,computation. More specifically, consider a set of points that have been labeled as outliers on a given iteration. We the least squares solution T A; o is obtained byrender these points ªinvisibleº by not allowing them to I À n Ácontribute to any histogram. However, we still assign them o pi À q i ; R n iIshape contexts, taking into account only the surroundinginlier points, so that at a later iteration they have a chance of A Q P t ; Sreemerging as an inlier. where P and Q contain the homogeneous coordinates of 3.4 Related Work and , respectively, i.e.,The most comprehensive body of work on shape corre- H I I pII pIPspondence in this general setting is the work of Gold et al. fF F F g and Chui and Rangarajan . They developed an P dF F F F F e: F Titerative optimization algorithm to determine point corre- I pnI pnPspondences and underlying image transformations jointly,where typically some generic transformation class is Here, Q denotes the pseudoinverse of Q.assumed, e.g., affine or thin plate splines. The cost function In this work, we mostly use the thin plate spline (TPS)that is being minimized is the sum of Euclidean distances model , , which is commonly used for representingbetween a point on the first shape and the transformed flexible coordinate transformations. Bookstein  found itsecond shape. This sets up a chicken-and-egg problem: The to be highly effective for modeling changes in biologicaldistances make sense only when there is at least a rough forms. Powell applied the TPS model to recover transfor-alignment of shape. Joint estimation of correspondences mations between curves . The thin plate spline is theand shape transformation leads to a difficult, highly non- 2D generalization of the cubic spline. In its regularizedconvex optimization problem, which is solved using form, which is discussed below, the TPS model includes thedeterministic annealing . The shape context is a very affine model as a special case. We will now provide somediscriminative point descriptor, facilitating easy and robust background information on the TPS model.correspondence recovery by incorporating global shape We start with the 1D interpolation problem. Let vi denoteinformation into a local descriptor. the target function values at corresponding locations pi As far as we are aware, the shape context descriptor and xi ; yi in the plane, with i I; P; F F F ; n. In particular, weits use for matching 2D shapes is novel. The most closely will set vi equal to xHi and yHi in turn to obtain one continuousrelated idea in past work is that due to Johnson and Hebert transformation for each coordinate. We assume that the in their work on range images. They introduced a locations xi ; yi are all different and are not collinear. Therepresentation for matching dense clouds of oriented TPS interpolant f x; y minimizes the bending energy
514 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 24, APRIL 2002 P P P P P @Pf @ f @ f We use two separate TPS functions to model a coordinate If P dxdy s P @xP @x@y @yP transformation, À Áand has the form: T x; y fx x; y; fy x; y II n which yields a displacement field that maps any position in f x; y aI ax x ay y wi U k xi ; yi À x; yk; iI the first image to its interpolated location in the second image.where the kernel function U r is defined by U r rP log rP In many cases, the initial estimate of the correspondencesand U H H as usual. In order for f x; y to have square contains some errors which could degrade the quality of theintegrable second derivatives, we require that transformation estimate. The steps of recovering correspon- n n n dences and estimating transformations can be iterated to wi H nd wi xi wi yi H: U overcome this problem. We usually use a fixed number of iI iI iI iterations, typically three in large-scale experiments, butTogether with the interpolation conditions, f xi ; yi vi , more refined schemes are possible. However, experimental experiences show that the algorithmic performance isthis yields a linear system for the TPS coefficients: independent of the details. An example of the iterative algorithm is illustrated in Fig. 4. 4.2 Empirical Robustness Evaluation In order to study the robustness of our proposed method, we performed the synthetic point set matching experiments described in . The experiments are broken into threewhere Kij U k xi ; yi À xj ; yj k, the ith row of P is parts designed to measure robustness to deformation, noise, I; xi ; yi , w and v are column vectors formed from wi and vi , and outliers. (The latter tests each include a ªmoderateºrespectively, and a is the column vector with elements amount of deformation.) In each test, we subjected theaI ; ax ; ay . We will denote the n Q Â n Q matrix of this model point set to one of the above distortions to create asystem by L. As discussed, e.g., in , L is nonsingular and ªtargetº point set; see Fig. 5. We then ran our algorithm towe can find the solution by inverting L. If we denote the find the best warping between the model and the target.upper left n Â n block of LÀI by A, then it can be shown that Finally, the performance is quantified by computing the average distance between the coordinates of the warped If G vT Av wT Kw: W model and those of the target. The results are shown in Fig. 6. In the most challenging part of the testÐthe outlier4.1 Regularization and Scaling Behavior experimentÐour approach shows robustness even up to aWhen there is noise in the specified values vi , one may wish level of 100 percent outlier-to-data ratio.to relax the exact interpolation requirement by means of In practice, we will need robustness to occlusion and segmentation errors which can be explored only in theregularization. This is accomplished by minimizing context of a complete recognition system, though these n experiments provide at least some guidelines. Hf vi À f xi ; yi P If : IH iI 4.3 Computational DemandsThe regularization parameter , a positive scalar, controls In our implementation on a regular Pentium III /500 MHzthe amount of smoothing; the limiting case of H workstation, a single comparison including computation ofreduces to exact interpolation. As demonstrated in , shape context for 100 sample points, set-up of the full, we can solve for the TPS coefficients in the matching matrix, bipartite graph matching, computation ofregularized case by replacing the matrix K by K I, the TPS coefficients, and image warping for three cycles takes roughly 200ms. The runtime is dominated by thewhere I is the n Â n identity matrix. It is interesting to number of sample points for each shape, with mostnote that the highly regularized TPS model degenerates to components of the algorithm exhibiting between quadraticthe least-squares affine model. and cubic scaling behavior. Using a sparse representation To address the dependence of on the data scale, throughout, once the shapes are roughly aligned, thesuppose xi ; yi and xHi ; yHi are replaced by xi ; yi and complexity could be made close to linear. xHi ; yHi , respectively, for some positive constant . Then,it can be shown that the parameters w; a; If of the optimalthin plate spline are unaffected if is replaced by P . This 5 OBJECT RECOGNITION AND PROTOTYPEsimple scaling behavior suggests a normalized definition of SELECTIONthe regularization parameter. Let again represent the scale Given a measure of dissimilarity between shapes, which weof the point set as estimated by the mean edge length will make precise shortly, we can proceed to apply it to thebetween two points in the set. Then, we can define in task of object recognition. Our approach falls into the categoryterms of and o , a scale-independent regularization of prototype-based recognition. In this framework, pioneeredparameter, via the simple relation P o . by Rosch et al. , categories are represented by ideal
BELONGIE ET AL.: SHAPE MATCHING AND OBJECT RECOGNITION USING SHAPE CONTEXTS 515Fig. 4. Illustration of the matching process applied to the example of Fig. 1. Top row: 1st iteration. Bottom row: 5th iteration. Left column: estimatedcorrespondences shown relative to the transformed model, with tangent vectors shown. Middle column: estimated correspondences shown relative tothe untransformed model. Right column: result of transforming the model based on the current correspondences; this is the input to the next iteration.The grid points illustrate the interpolated transformation over s P . Here, we have used a regularized TPS model with o I. examples rather than a set of formal logical rules. As an I and K=n 3 H, the error 3 E Ã ). This is interestingexample, a sparrow is a likely prototype for the category of because it shows that the humble nearest-neighbor classifierbirds; a less likely choice might be an penguin. The idea of is asymptotically optimal, a property not possessed byprototypes allows for soft category membership, meaning several considerably more complicated techniques. Ofthat as one moves farther away from the ideal example in course, what matters in practice is the performance forsome suitably defined similarity space, ones association with small n, and this gives us a way to compare differentthat prototype falls off. When one is sufficiently far away from similarity/distance measures.that prototype, the distance becomes meaningless, but bythen one is most likely near a different prototype. As an 5.1 Shape Distanceexample, one can talk about good or so-so examples of the In this section, we make precise our definition of shapecolor red, but when the color becomes sufficiently different, distance and apply it to several practical problems. We usedthe level of dissimilarity saturates at some maximum level a regularized TPS transformation model and three iterationsrather than continuing on indefinitely. of shape context matching and TPS reestimation. After Prototype-based recognition translates readily into the matching, we estimated shape distances as the weightedcomputational framework of nearest-neighbor methods sum of three terms: shape context distance, image appear-using multiple stored views. Nearest-neighbor classifiers ance distance, and bending energy.have the property  that as the number of examples n in We measure shape context distance between shapes the training set goes to infinity, the IExx error converges to and as the symmetric sum of shape context matchinga value PE Ã , where E Ã is the Bayes Risk (for KExx, K 3 costs over best matching points, i.e.,Fig. 5. Testing data for empirical robustness evaluation, following Chui and Rangarajan . The model pointsets are shown in the first column.Columns 2-4 show examples of target point sets for the deformation, noise, and outlier tests, respectively.
516 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 24, APRIL 2002Fig. 6. Comparison of our results (u) to Chui and Rangarajan (Ã) and iterated closest point () for the fish and Chinese character, respectively. The error tbars indicate the standard deviation of the error over 100 random trials. Here, we have used 5 iterations with o I:H. In the deformation and noisetests no dummy nodes were added. In the outlier test, dummy nodes were added to the model point set such that the total number of nodes was equalto that of the target. In this case, the value of d does not affect the solution. I 5.2 Choosing Prototypes hs ; rg min C p; T q n pP qP In a prototype-based approach, the key question is: what IP I examples shall we store? Different categories need different rg min C p; T q; m qP pP numbers of views. For example, certain handwritten digits have more variability than others, e.g., one typically seeswhere T Á denotes the estimated TPS shape transformation. more variations in fours than in zeros. In the category of In many applications there is additional appearance 3D objects, a sphere needs only one view, for example,information available that is not captured by our notion of while a telephone needs several views to capture the varietyshape, e.g., the texture and color information in the of visual appearance. This idea is related to the ªaspectºgrayscale image patches surrounding corresponding points. concept as discussed in . We will now discuss how weThe reliability of appearance information often suffers approach the problem of prototype selection.substantially from geometric image distortions. However, In the nearest-neighbor classifier literature, the problemafter establishing image correspondences and recovery of of selecting exemplars is called editing. Extensive reviews ofunderlying 2D image transformation the distorted image nearest-neighbor editing methods can be found in Ripleycan be warped back into a normal form, thus correcting for  and Dasarathy . We have developed a novel editingdistortions of the image appearance. algorithm based on shape distance and KEmedoids cluster- We used a term h ; for appearance cost, defined as ing. KEmedoids can be seen as a variant of KEmens thatthe sum of squared brightness differences in Gaussian restricts prototype positions to data points. First a matrix ofwindows around corresponding image points, pairwise similarities between all possible prototypes is computed. For a given number of K prototypes the I n Â À À Á Á ÃP KEmedoids algorithm then iterates two steps: 1) For a givenh ; G Á I pi ÁÀI T q i Á ; assignment of points to (abstract) clusters a new prototype n iI ÁPZ P is selected for that cluster by minimizing the average IQ distance of the prototype to all elements in the cluster, andwhere I and I are the gray-level images corresponding to 2) given the set of prototypes, points are then reassigned to and , respectively. Á denotes some differential vector clusters according to the nearest prototype. More formally,offset and G is a windowing function typically chosen to be denote by c the (abstract) cluster of shape , e.g., represented by some number fI; F F F ; kg and denote by p ca Gaussian, thus putting emphasis to pixels nearby. We the associated prototype. Thus, we have a class mapthus sum over squared differences in windows aroundcorresponding points, scoring the weighted gray-level c X I À fI; F F F ; kg 3 IRsimilarity. This score is computed after the thin plate spline and a prototype maptransformation T has been applied to best warp the images p X fI; F F F ; kg À P : 3 ISinto alignment. The third term he ; corresponds to the ªamountº of Here, I and P are some subsets of the set of all potentialtransformation necessary to align the shapes. In the TPS case shapes . Often, I P . KEmedoids proceeds bythe bending energy (9) is a natural measure (see ). iterating two steps:
BELONGIE ET AL.: SHAPE MATCHING AND OBJECT RECOGNITION USING SHAPE CONTEXTS 517Fig. 7. Handwritten digit recognition on the MNIST data set. Left: Test set errors of a 1-NN classifier using SSD and Shape Distance (SD) measures.Right: Detail of performance curve for Shape Distance, including results with training set sizes of 15,000 and 20,000. Results are shown on asemilog-x scale for K I; Q; S nearest-neighbors. 1. group I into classes given the class prototypes p c, tan Cij H:S I À os i À j measures tangent angle dissim- and ilarity, and
H:I. For recognition, we used a KExx 2. identify a representative prototype for each class classifier with a distance function given the elements in the cluster.Basically, item 1 is solved by assigning each shape P I to h I:Th hs H:Qhe : IWthe nearest prototype, thus The weights in (19) have been optimized by a leave-one-out procedure on a Q; HHH Â Q; HHH subset of the training data. c rg min h ; p k: IT k On the MNIST data set nearly 30 algorithms have beenFor given classes, in item 2 new prototypes are selected compared (http://www.research.att.com/~yann/exdb/based on minimal mean dissimilarity, i.e., mnist/index.html). The lowest test set error rate published at this time is 0.7 percent for a boosted LeNet-4 with a p k rg min h ; p: IU training set of size TH; HHH Â IH synthetic distortions per pP P Xc shapek training digit. Our error rate using 20,000 training examplesSince both steps minimize the same cost function and QExx is 0.63 percent. The 63 errors are shown in Fig. 8.4 As mentioned earlier, what matters in practical applica- H c; p h ; p c ; IV tions of nearest-neighbor methods is the performance for P I small n, and this gives us a way to compare differentthe algorithm necessarily converges to a (local) minimum. similarity/distance measures. In Fig. 7 (left), our shape As with most clustering methods, with kEmedoids one distance is compared to SSD (sum of squared differencesmust have a strategy for choosing k. We select the number between pixel brightness values). In Fig. 7 (right), we compareof prototypes using a greedy splitting strategy starting with the classification rates for different K.one prototype per category. We choose the cluster to splitbased on the associated overall misclassification error. This 6.2 3D Object Recognitioncontinues until the overall misclassification error has Our next experiment involves the 20 common householddropped below a criterion level. Thus, the prototypes are objects from the COIL-20 database . Each object wasautomatically allocated to the different object classes, thus placed on a turntable and photographed every S for a totaloptimally using available resources. The application of this of 72 views per object. We prepared our training sets byprocedure to a set of views of 3D objects is explored in selecting a number of equally spaced views for each objectSection 6.2 and illustrated in Fig. 10. and using the remaining views for testing. The matching algorithm is exactly the same as for digits. Recall, that the6 CASE STUDIES Canny edge detector responds both to external and internal6.1 Digit Recognition contours, so the 100 sample points are not restricted to theHere, we present results on the MNIST data set of hand- external boundary of the silhouette.written digits, which consists of 60,000 training and 10,000 test Fig. 9 shows the performance using IExx with thedigits . In the experiments, we used 100 points sampled distance function D as given in (19) compared to afrom the Canny edges to represent each digit. When 4. DeCoste and Scho Èlkopf  report an error rate of 0.56 percent on thecomputing the Cij s for the bipartite matching, we included same database using Virtual Support Vectors (VSV) with the full traininga term representing the dissimilarity of local tangent set of 60,000. VSVs are found as follows: 1) obtain SVs from the original training set using a standard SVM, 2) subject the SVs to a set of desiredangles. Specifically, we defined the matching cost as transformations (e.g., translation), 3) train another SVM on the generated sc tan scCij I À
Cij , where Cij is the shape context cost, examples.
518 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 24, APRIL 2002Fig. 8. All of the misclassified MNIST test digits using our method (63 out of 10,000). The text above each digit indicates the example numberfollowed by the true label and the assigned label.straightforward sum of squared differences (SSD). SSD error rate with an average of only four two-dimensionalperforms very well on this easy database due to the lack of views for each three-dimensional object, thanks to thevariation in lighting  (PCA just makes it faster). flexibility provided by the matching algorithm. The prototype selection algorithm is illustrated in Fig. 10.As seen, views are allocated mainly for more complex 6.3 MPEG-7 Shape Silhouette Databasecategories with high within class variability. The curve Our next experiment involves the MPEG-7 shape silhouettemarked SC-proto in Fig. 9 shows the improved classification database, specifically Core Experiment CE-Shape-1 part B,performance using this prototype selection strategy instead which measures performance of similarity-based retrievalof equally-spaced views. Note that we obtain a 2.4 percent . The database consists of 1,400 images: 70 shape categories, 20 images per category. The performance is measured using the so-called ªbullseye test,º in which eachFig. 9. 3D object recognition using the COIL-20 data set. Comparison oftest set error for SSD, Shape Distance (SD), and Shape Distance withkEmedoids prototypes (SD-proto) versus number of prototype views. For Fig. 10. Prototype views selected for two different 3D objects from theSSD and SD, we varied the number of prototypes uniformly for all COIL data set using the algorithm described in Section 5.2. With thisobjects. For SD-proto, the number of prototypes per object depended on approach, views are allocated adaptively depending on the visualthe within-object variation as well as the between-object similarity. complexity of an object with respect to viewing angle.
BELONGIE ET AL.: SHAPE MATCHING AND OBJECT RECOGNITION USING SHAPE CONTEXTS 519Fig. 11. Examples of shapes in the MPEG7 database for three differentcategories.image is used as a query and one counts the number ofcorrect images in the top 40 matches. As this experiment involves intricate shapes we in-creased the number of samples from 100 to 300. In somecategories, the shapes appear rotated and flipped, which weaddress using a modified distance function. The distancedist R; Q between a reference shape R and a query shapeQ is defined as dist Q; R minfdist Q; Ra ; dist Q; Rb ; dist Q; Rc g;where Ra ; Rb , and Rc denote three versions of R: un-changed, vertically flipped, and horizontally flipped. With these changes in place but otherwise using thesame approach as in the MNIST digit experiments, weobtain a retrieval rate of 76.51 percent. Currently the bestpublished performance is achieved by Latecki et al. ,with a retrieval rate of 76.45 percent, followed by Mokhtar-ian et al. at 75.44 percent.6.4 Trademark RetrievalTrademarks are visually often best described by their shapeinformation, and, in many cases, shape provides the onlysource of information. The automatic identification oftrademark infringement has interesting industrial applica-tions, since with the current state of the art trademarks arebroadly categorized according to the Vienna code, and thenmanually classified according to their perceptual similarity.Even though shape context matching does not provide a fullsolution to the trademark similarity problem (other poten-tial cues are text and texture), it still serves well to illustrate Fig. 12. Trademark retrieval results based on a database of 300 differentthe capability of our approach to capture the essence of real-world trademarks. We used an affine transformation model and ashape similarity. In Fig. 12, we depict retrieval results for a weighted combination of shape context similarity Ds and the sum overdatabase of 300 trademarks. In this experiment, we relied on local tangent orientation differences.an affine transformation model as given by (3), and as in theprevious case, we used 300 sample points. 7 CONCLUSION We experimented with eight different query trademarks We have presented a new approach to shape matching. A keyfor each of which the database contained at least one potential characteristic of our approach is the estimation of shapeinfringement. We depict the top four hits as well as their similarity and correspondences based on a novel descriptor,similarity score. It is clearly seen that the potential infringe- the shape context. Our approach is simple and easy to apply,ments are easily detected and appear as most similar in thetop ranks despite substantial variation of the actual shapes. It yet provides a rich descriptor for point sets that greatlyhas been manually verified that no visually similar trademark improves point set registration, shape matching and shapehas been missed by the algorithm. recognition. In our experiments, we have demonstrated
520 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 24, APRIL 2002 gray-scale image and selects a subset of the edge pixels found. The selection could be uniformly at random, but we have found it to be advantageous to ensure that the sample points have a certain minimum distance between them as this makes sure that the sampling along the contours is somewhat uniform. (This corresponds to sampling from a point process which is a hard-core model .) Since the sample points are drawn randomly and independently from the two shapes, there is inevitably jitter noise in the output of the matching algorithm which finds correspondences between these two sets of sample points. However, when the transformation between the shapes is estimated as a regularized thin plate spline, the effect of this jitter is smoothed away. ACKNOWLEDGMENTS This research is supported by the Army Research Office (ARO) DAAH04-96-1-0341, the Digital Library Grant IRI-Fig. 13. Kimia data set: each row shows instances of a different object 9411334, a US National Science Foundation Graduatecategory. Performance is measured by the number of closest matches Fellowship for S. Belongie, and the German Researchwith the correct category label. Note that several of the categories requirerotation invariant matching for effective recognition. All of the 1st ranked Foundation by DFG grant PU-165/1. Parts of this workclosest matches were correct using our method. Of the 2nd ranked have appeared in , . The authors wish to thank H. Chuimatches, one error occurred in 1 versus 8. In the 3rd ranked matches, and A. Rangarajan for providing the synthetic testing dataconfusions arose from 2 versus 8, 8 versus 1, and 15 versus 17. used in Section 4.2. We would also like to thank them and various members of the Berkeley computer vision group,invariance to several common image transformations, in- particularly A. Berg, A. Efros, D. Forsyth, T. Leung, J. Shi,cluding significant 3D rotations of real-world objects. and Y. Weiss, for useful discussions. This work was carried out while the authors were with the Department ofAPPENDIX A Electrical Engineering and Computer Science Division,COMPLETE ROTATION INVARIANT RECOGNITION University of California at Berkeley.In this appendix, we demonstrate the use of the relativeframe in our approach as a means of obtaining complete REFERENCESrotation invariance. To demonstrate this idea, we have used  Y. Amit, D. Geman, and K. Wilder, ªJoint Induction of Shapethe database provided by Sharvit et al.  shown in Fig. 13. Features and Tree Classifiers,º IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 11, pp. 1300-1305, Nov. 1997.In this experiment, we used n IHH sample points and, as  S. Belongie, J. Malik, and J. Puzicha, ªMatching Shapes,º Proc.mentioned above, we used the relative frame (Section 3.3) Eighth Intl. Conf. Computer Vision, pp. 454-461, July 2001.when computing the shape contexts. We used five bins for  S. Belongie, J. Malik, and J. Puzicha, ªShape Context: A New Descriptor for Shape Matching and Object Recognition,º Advanceslog r over the range H:IPS to P and 12 equally spaced in Neural Information Processing Systems 13: Proc. 2000 Conf., T.K.radial bins in these and all other experiments in this paper. Leen, T.G. Dietterich, and V. Tresp, eds.. pp. 831-837, 2001.No transformation model at all was used. As a similarity  P.J. Bickel, ªA Distribution Free Version of the Smirnov Two- Sample Test in the Multivariate Case,º Annals of Math. Statistics,score, we used the matching cost function i Ci; i after vol. 40, pp. 1-23, 1969.one iteration with no transformation step. Thus, this  F.L. Bookstein, ªPrincipal Warps: Thin-Plate Splines and Decom-experiment is specifically designed solely to evaluate the position of Deformations,º IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 11, no. 6, pp. 567-585, June 1989.power of the shape descriptor in the face of rotation.  F.L. Bookstein, Morphometric Tools for Landmark Data: Geometry and In  and , the authors summarize their results on Biology. Cambridge Univ. Press, 1991.this data set by stating the number of 1st, 2nd, and 3rd È  C. Burges and B. Scholkopf, ªImproving the Accuracy and Speed of Support Vector Machines,º Advances in Neural Informationnearest-neighbors that fall into the correct category. Our Processing Systems 9: Proc. 1996 Conf., D.S. Touretzky, M.C. Mozer,results are 25/25, 24/25, 22/25. In  and , the results and M. E. Hasselmo, eds. pp. 375-381, 1997.quoted are 23/25, 21/25, 20/25 and 25/25, 21/25, 19/25,  S. Carlsson, ªOrder Structure, Correspondence and Shape Based Categories,º Intl Workshop Shape, Contour and Grouping, May 1999.respectively.  H. Chui and A. Rangarajan, ªA New Algorithm for Non-Rigid Point Matching,º Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 44-51, June 2000.APPENDIX B  T. Cootes, D. Cooper, C. Taylor, and J. Graham, ªActive Shape ModelsÐTheir Training and Application,º Computer Vision andSAMPLING CONSIDERATIONS Image Understanding (CVIU), vol. 61, no. 1, pp. 38-59, Jan. 1995.In our approach, a shape is represented by a set of sample  C. Cortes and V. Vapnik, ªSupport Vector Networks,º Machine Learning, vol. 20, pp. 273-297, 1995.points drawn from the internal and external contours of an  Nearest Neighbor (NN) Norms: (NN) Pattern Classification Techniques.object. Operationally, one runs an edge detector on the B.V. Dasarathy, ed., IEEE Computer Soc., 1991.
BELONGIE ET AL.: SHAPE MATCHING AND OBJECT RECOGNITION USING SHAPE CONTEXTS 521 È D. DeCoste and B. Scholkopf, ªTraining Invariant Support Vector  B. Moghaddam, T. Jebara, and A. Pentland, ªBayesian Face Machines,º Machine Learning, to appear in 2002. Recognition,º Pattern Recognition, vol. 33, no. 11, pp. 1771-1782, J. Duchon, ªSplines Minimizing Rotation-Invariant Semi-Norms in Nov, 2000. Sobolev Spaces,º Constructive Theory of Functions of Several  F. Mokhtarian, S. Abbasi, and J. Kittler, ªEfficient and Robust Variables, W. Schempp and K. Zeller, eds., pp. 85-100, Berlin: Retrieval by Shape Content Through Curvature Scale Space,º Springer-Verlag, 1977. Image Databases and Multi-Media Search, A.W.M. Smeulders and M. Fischler and R. Elschlager, ªThe Representation and Matching R. Jain, eds., pp. 51-58, World Scientific, 1997. of Pictorial Structures,º IEEE Trans. Computers, vol. 22, no. 1,  H. Murase and S. Nayar, ªVisual Learning and Recognition of 3-D pp. 67-92, 1973. Objects from Oppearance,º Intl. J. Computer Vision, vol. 14, no. 1, D. Gavrila and V. Philomin, ªReal-Time Object Detection for pp. 5-24, Jan. 1995. Smart Vehicles,º Proc. Seventh Intl. Conf. Computer Vision, pp. 87-  M. Oren, C. Papageorgiou, P. Sinha, E. Osuna, and T. Poggio, 93, 1999. ªPedestrian Detection Using Wavelet Templates,º Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 193-199, June 1997. Y. Gdalyahu and D. Weinshall, ªFlexible Syntactic Matching of  C. Papadimitriou and K. Stieglitz, Combinatorial Optimization: Curves and its Application to Automatic Hierarchical Classifica- Algorithms and Complexity. Prentice Hall, 1982. tion of Silhouettes,º IEEE Trans. Pattern Analysis and Machine  E. Persoon and K. Fu, ªShape Discrimination Using Fourier Intelligence, vol. 21, no. 12, pp. 1312-1328, Dec. 1999. Descriptors,º IEEE Trans. Systems, Man and Cybernetics, vol. 7, no. 3, F. Girosi, M. Jones, and T. Poggio, ªRegularization Theory and pp. 170-179, Mar. 1977. Neural Networks Architectures,º Neural Computation, vol. 7, no. 2,  M.J.D. Powell, ªA Thin Plate Spline Method for Mapping Curves pp. 219-269, 1995. into Curves in Two Dimensions,º Computational Techniques and S. Gold, A. Rangarajan, C.-P. Lu, S. Pappu, and E. Mjolsness, Applications (CTAC 95), 1995. ªNew Algorithms for 2D and 3D Point Matching: Pose Estimation  B.D. Ripley, ªModelling Spatial Patterns,º J. Royal Statistical and Correspondence,º Pattern Recognition, vol. 31, no. 8, 1998. Society, Series B, vol. 39, pp. 172-212, 1977. E. Goldmeier, ªSimilarity in Visually Perceived Forms,º Psycho-  B.D. Ripley, Pattern Recognition and Neural Networks. Cambridge logical Issues, vol. 8, no. 1, pp. 1-135, 1936/1972. Univ. Press, 1996. U. Grenander, Y. Chow, and D. Keenan, HANDS: A Pattern  E. Rosch, ªNatural Categories,º Cognitive Psychology, vol. 4, no. 3, Theoretic Study Of Biological Shapes. Springer, 1991. pp. 328-350, 1973. M. Hagedoorn, ªPattern Matching Using Similarity Measures,º  E. Rosch, C.B. Mervis, W.D. Gray, D.M. Johnson, and P. Boyes- PhD thesis, Universiteit Utrecht, 2000. Braem, ªBasic Objects in Natural Categories,º Cognitive Psychol- D. Huttenlocher, G. Klanderman, and W. Rucklidge, ªComparing ogy, vol. 8, no. 3, pp. 382-439, 1976. Images Using the Hausdorff Distance,º IEEE Trans. Pattern  C. Schmid and R. Mohr, ªLocal Grayvalue Invariants for Image Analysis and Machine Intelligence, vol. 15, no. 9, pp. 850-863, Sept. Retrieval,º IEEE Trans. Pattern Analysis and Machine Intelligence, 1993. vol. 19, no. 5, pp. 530-535, May 1997. D. Huttenlocher, R. Lilien, and C. Olson, ªView-Based Recogni-  S. Sclaroff and A. Pentland, ªModal Matching for Correspondence tion Using an Eigenspace Approximation to the Hausdorff and Recognition,º IEEE Trans. Pattern Analysis and Machine Measure,º IEEE Trans. Pattern Analysis and Machine Intelligence, Intelligence, vol. 17, no. 6, pp. 545-561, June 1995. vol. 21, no. 9, pp. 951-955, Sept. 1999.  G. Scott and H. Longuet-Higgins, ªAn Algorithm for Associating the Features of Two Images,º Proc. Royal Soc. London, vol. 244, S. Jeannin and M. Bober, ªDescription of Core Experiments for pp. 21-26, 1991. MPEG-7 Motion/Shape,º Technical Report ISO/IEC JTC 1/SC  L.S. Shapiro and J. M. Brady, ªFeature-Based Correspondence: An 29/WG 11 MPEG99/N2690, MPEG-7, Seoul, Mar. 1999. Eigenvector Approach,º Image and Vision Computing, vol. 10, no. 5, A.E. Johnson and M. Hebert, ªRecognizing Objects by Matching pp. 283-288, June 1992. Oriented Points,º Proc. IEEE Conf. Computer Vision and Pattern  D. Sharvit, J. Chan, H. Tek, and B. Kimia, ªSymmetry-Based Recognition, pp. 684-689, 1997. Indexing of Image Databases,º J. Visual Comm. and Image D. Jones and J. Malik, ªComputational Framework to Determining Representation, vol. 9, no. 4, pp. 366-380, Dec. 1998. Stereo Correspondence from a Set of Linear Spatial Filters,º Image  L. Sirovich and M. Kirby, ªLow Dimensional Procedure for the and Vision Computing, vol. 10, no. 10, pp. 699-708, Dec. 1992. Characterization of Human Faces,º J. Optical Soc. Am. A, vol. 4, R. Jonker and A. Volgenant, ªA Shortest Augmenting Path no. 3, pp. 519-524, 1987. Algorithm for Dense and Sparse Linear Assignment Problems,º  D.W. Thompson, On Growth and Form, Cambridge Univ. Press, Computing, vol. 38, pp. 325-340, 1987. 1917. D. Kendall, ªShape Manifolds, Procrustean Metrics and Complex  M. Turk and A. Pentland, ªEigenfaces for Recognition,º J. Cognitive Projective Spaces,º Bull. London Math. Soc., vol. 16, pp. 81-121, Neuroscience, vol. 3, no. 1, pp. 71-96, 1991. 1984.  S. Umeyama, ªAn Eigen Decomposition Approach to Weighted J.J. Koenderink and A.J. van Doorn, ªThe Internal Representation Graph Matching Problems,º IEEE Trans. Pattern Analysis and of Solid Shape with Respect to Vision,º Biological Cybernetics, Machine Intelligence, vol. 10, no. 5, pp. 695-703, Sept. 1988. vol. 32, pp. 211-216, 1979.  R.C. Veltkamp and M. Hagedoorn, ªState of the Art in Shape È M. Lades, C. Vorbuggen, J. Buhmann, J. Lange, C. von der Matching,º Technical Report UU-CS-1999-27, Utrecht, 1999. Malsburg, R. Wurtz, and W. Konen, ªDistortion Invariant Object  T. Vetter, M.J. Jones, and T. Poggio, ªA Bootstrapping Algorithm Recognition in the Dynamic Link Architecture,º IEEE Trans. for Learning Linear Models of Object Classes,º Proc. IEEE Conf. Computers, vol. 42, no. 3, pp. 300-311, Mar. 1993. Computer Vision and Pattern Recognition, pp. 40-46, 1997. Y. Lamdan, J. Schwartz, and H. Wolfson, ªAffine Invariant Model-  G. Wahba, Spline Models for Observational Data. Soc. Industrial and Based Object Recognition,º IEEE Trans. Robotics and Automation, Applied Math., 1990. vol. 6, pp. 578-589, 1990.  A. Yuille, ªDeformable Templates for Face Recognition,º È L.J. Latecki, R. Lakamper, and U. Eckhardt, ªShape Descriptors for J. Cognitive Neuroscience, vol. 3, no. 1, pp. 59-71, 1991. Non-Rigid Shapes with a Single Closed Contour,º Proc. IEEE Conf.  C. Zahn and R. Roskies, ªFourier Descriptors for Plane Closed Computer Vision and Pattern Recognition, pp. 424-429, 2000. Curves,º IEEE Trans. Computers, vol. 21, no. 3, pp. 269-281, Mar. 1972. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, ªGradient-Based Learning Applied to Document Recognition,º Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998. T.K. Leung, M.C. Burl, and P. Perona, ªFinding Faces in Cluttered Scenes Using Random Labelled Graph Matching,º Proc. Fifth Intl. Conf. Computer Vision, pp. 637-644, 1995. D.G. Lowe, ªObject Recognition from Local Scale-Invariant Features,º Proc. Seventh Intl. Conf. Computer Vision, pp. 1150- 1157, Sept. 1999. J. Meinguet, ªMultivariate Interpolation at Arbitrary Points Made Simple,º J. Applied Math. Physics (ZAMP), vol. 5, pp. 439-468, 1979.
522 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 24, APRIL 2002 Serge Belongie received the BS degree (with Jan Puzicha received the Diploma degree in honor) in electrical engineering from the California 1995 and the PhD degree in computer science in Institute of Technology, Pasadena, California, in 1999, both from the University of Bonn, Bonn, 1995, and the MS and PhD degrees in electrical Germany. He was with the Computer Vision and engineering and computer sciences (EECS) at Pattern Recognition Group, University of Bonn, the University of California at Berkeley, in 1997 from 1995 to 1999. In September 1999, he and 2000, respectively. While at Berkeley, his joined the Computer Science Department, Uni- research was supported by a National Science versity of California, Berkeley, as an Emmy Foundation Graduate Research Fellowship and Noether Fellow of the German Science Founda- the Chancellors Opportunity Predoctoral Fellow- tion, where he is currently working on optimiza-ship. He is also a cofounder of Digital Persona, Inc., and the principal tion methods for perceptual grouping and image segmentation. Hisarchitect of the Digital Persona fingerprint recognition algorithm. He is research interests include computer vision, image processing, unsu-currently an assistant professor in the Computer Science and Engineering pervised learning, data analysis, and data mining.Department at the University of California at San Diego. His researchinterests include computer vision, pattern recognition, and digital signalprocessing. He is a member of the IEEE. Jitendra Malik received the BTech degree in electrical engineering from Indian Institute of . For more information on this or any other computing topic, Technology, Kanpur in 1980 and the PhD please visit our Digital Library at http://computer.org/publications/dilb. degree in computer science from Stanford University, Stanford, California, in 1986. In January 1986, he joined the faculty of the Computer Science Division, Department of EECS, University of California at Berkeley, where he is currently a professor. During 1995- 1998, he also served as vice-chair for GraduateMatters. He is a member of the Cognitive Science and Vision Sciencegroups at UC Berkeley. His research interests are in computer visionand computational modeling of human vision. His work spans a range oftopics in vision including image segmentation and grouping, texture,stereopsis, object recognition, image-based modeling and rendering,content-based image querying, and intelligent vehicle highway systems.He has authored or coauthored more than 100 research papers on thesetopics. He received the gold medal for the best graduating student inelectrical engineering from IIT Kanpur in 1980, a Presidential YoungInvestigator Award in 1989, and the Rosenbaum fellowship for theComputer Vision Programme at the Newton Institute of MathematicalSciences, University of Cambridge in 1993. He received the Diane S.McEntyre Award for Excellence in Teaching from the Computer ScienceDivision, University of California at Berkeley, in 2000. He is an Editor-in-Chief of the International Journal of Computer Vision. He is a member ofthe IEEE.