Andalo wifs2010


Published on

Detecting vanishing points by segment clustering on the projective plane for single-view photogrametry

Published in: Technology, Art & Photos
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Andalo wifs2010

  1. 1. Detecting vanishing points by segment clustering onthe projective plane for single-view photogrammetry Fernanda A. Andal´ 1 , Gabriel Taubin 2 , Siome Goldenstein 1 o 1 Institute of Computing, University of Campinas (Unicamp) CEP 13083-852, Campinas, SP, Brazil {feandalo,siome} 2 Division of Engineering, Brown University Providence, RI 02912 USA Abstract—In this paper, we describe how an effective vanishing The early methods rely on knowing the internal parameterspoint detector can be applied to photogrammetry when only a of the camera and its position with respect to the viewedsingle view of an architectural environment is available. Our scene. Even when this information is not available, cameramethod performs automatic segment clustering in projectivespace – a direct transformation from the image space – instead calibration can be performed if the depth of some key points isof the traditional bounded accumulator space. Experiments on available [3] or using self-calibration techniques [4]. However,real images show the effectiveness of the proposed detector the necessary data is not always available or involves a muchin finding all vanishing points, as well as its application in a bigger and ill-posed problem, the 3D reconstruction.photogrammetry algorithm, by recovering the vertical direction In this scenario, previous works show that vanishing linesof the scene and the vanishing line for the ground plane. and points are useful features [1], [5]. A vanishing point is defined as the convergence point of a set of lines in the image I. I NTRODUCTION plane that is produced by the projection of parallel lines in Images provide strong cues about the three-dimensional real space, under the assumption of perspective projection, e.g.(3D) geometry of the scenes depicted. These cues can be with a pin-hole camera. Because they are invariant features,exploited in a variety of computer vision applications, such the analysis of vanishing points provides strong cues to makeas forensics, image-based rendering, and surveillance. The inferences about the 3D structures of a scene, such as depthadvantages of using images for such applications, instead of and object dimension.other specific devices (laser scanners, ultrasonic devices, etc.), In this work, we propose a novel and automated method foris that images can be applied to measure the distance between vanishing point detection based on a geometrical approach,the camera and the objects, as well as two other points in in which all finite and infinite vanishing points are estimatedspace, areas of surfaces and angles [1]. Furthermore, since in a single image of a man-made environment. We use theimages are the only input data, the depth cues can be analysed vanishing points to detect the vertical direction of the sceneeven if the scene is no longer accessible, using archived images and the vanishing line of the ground plane. For estimatingor old footage. heights of objects in the scene, we insert our method in the The problem of taking world measurements using images – algorithm proposed by [1].photogrammetry – has one of its main application in forensic This paper is organized as follows. Section II presents thescience, where image measurements are being used to provide problem of measuring heights in an image using vanishinguseful information about the course of events and the size of points and lines. Section III describes our method for vanishingitems and persons in a crime scene [2]. The two most com- point estimation. Section IV describes how to estimate themonly reported forensic activities that involve photogrammetry vertical direction in an image and also shows how to detectare human height analysis and accident analysis of motor the vanishing line associated with the ground plane. Section Vvehicles [2]. In a reliable system, the estimated height of discusses experimental results and Section VI concludes thepeople can be used as a corroborative evidence, for example. paper. However, when dealing with images, the essential problemis how to recover the third dimension or take measures of the II. BACKGROUNDscene, knowing that the required information was not capturedand stored in the process of projecting the 3D space to the In [1], Criminisi addresses the problem of measuring dis-planar image. tances of points from planes in an uncalibrated framework. The aim is to compute the height of an object relative to a reference. The author assumes that the vertical direction andWIFS’2010, December 12-15, 2010, Seattle, WA, USA. the vanishing line for the ground plane have been computed978-1-4244-4652-0/09/$25.00 c 2009 IEEE. by some method.
  2. 2. Selection of seeds based If v is the vanishing point for the vertical direction, m is the Input image 2.1 on a segment quality valuevanishing line of the ground plane, tr and br are the top andbase points of the reference, respectively, and tx and bx arethe top and base points of the object to be measured, then [1] bi × t i αZi = − , ∀i = r, x, (1) (m · bi ) v × tiwhere Zx is the height of the object to be measured, Zr isthe reference height and α is a scalar quantity herein referredto as metric factor. If α is known, then a metric value for theheight Z is obtained. If the height Z is known, then α can be until convergencecomputed. The method outline [1] is presented in Algorithm 1. Segment detection based on 1 the Helmholtz principle 2.2 Segment clusteringAlgorithm 1 Measuring heights in an image. 1) Estimate the vanishing point v for the vertical direction; 2) Estimate the vanishing line m of the reference plane; 3) Select top and base points of the reference segment (points tr and br ); 4) Compute the metric factor α by applying Equation 1 with tr and br ; loop (a) Select top and base of the object to be measured (points tx and bx ); Detection of a vanishing point (b) Compute the metric factor α by applying Equation 1 3 for each cluster with tx and bx . end loop According to Criminisi [1], the key to the success of thiscalculation is an accurate estimation of the vertical vanishingpoint v and the vanishing line m of the reference plane. The following sections describe our technique for the au-tomatic computation of vanishing points and how to obtain,from them, the vanishing point for the vertical direction and Fig. 1. Stages of the proposed vanishing point detector.the vanishing line for the ground plane. III. E STIMATING THE VANISHING POINTS A. Line segment detection Our method for vanishing point detection is presented in Our vanishing point detector considers line segments asAlgorithm 2. primitives. To detect these primitives, we use a method based on the Helmholtz Principle [6]. This principle states that if theAlgorithm 2 Vanishing point estimation expectation of an observed configuration in an image is very 1) Extraction of line segments on the image plane; small, then the grouping of the objects in this configuration is 2) Clustering of line segments to groups of lines converging valid. to the same vanishing point; Aside from detecting line segments, this method provides loop an important value – the number of false alarms for a segment (a) determination of seeds based on a computed quality – that is useful in the next steps to compute a quality value value for each segment; for each segment. The number of false alarms of a segment (b) grouping of the line segments based on the distance represents an upper-bound of the expectation, in an image, of among lines and intersection points in projective space; the number of segments of probability less than the one of the end loop considered segment [6]. 3) Vanishing point estimation for the extracted segment In the following, we denote the set of the detected line clusters. segments by S = {s1 , . . . , s|S| }, the number of false alarms for the segment si ∈ S by N F i , and the intersection point The outline of the proposed method is illustrated in Figure 1. between the lines corresponding to segments si and sj byThe following subsections describe each stage. w(li ,lj ) .
  3. 3. B. Line segment clustering The goal of the line segment clustering is to assign a clusterfor each one of the line segments in S. To achieve this goal,the clustering process is subdivided in three steps: selectionof the first seeds, assignment step, and update step. The number M of clusters to be detected is informed bythe user. However, this parameter does not need to be tunned,since the method works independently of the chosen number,detecting the corresponding number of vanishing points. 1) First seeds: The first seeds are chosen based on the (a) (b)quality value of the segments. The quality value qi of segmentsi ∈ S is defined as Fig. 2. Example of possible clusters. The dashed line represents the mean line of the clusters. N Fi − (max(N Fj ) + min(N Fj )) qi = , j ∈ S. (2) max(N Fj ) n We select the 2M lines with highest corresponding segment C= qj ∗ cos(2θj ). (7)quality and randomly choose pairs of these lines as seeds for j=1the M clusters. We denote d1i and d2i the two seeds of clusteri. In Equations 6 and 7, n is the number of segments assigned 2) Assignment step: Each segment s ∈ S has to be assigned to cluster i, and qj and θj are, respectively, the quality valueto a cluster. The assignment step is based on the distance and orientation of the j-th segment assigned to the cluster i.between the lines corresponding to the segments and the The seed d2i is the line corresponding to a segment assignedpseudo-centroids ci = w(d1i ,d2i ) , i = 1, . . . , M . to cluster i and which intersection point with seed d1i , The distance between a line l and a pseudo-centroid c is w(d1i ,d2i ) , minimizes the sum of the distances to all otherdefined as intersection points w(d1i ,lj ) , where sj is assigned to cluster i. |l · c| DLP (l, c) = . (3) l c C. Vanishing point detectionDLP is symmetric, but it is not a full metric. However, it is a The vanishing point vi for the cluster i is the intersectionrobust way to measure the amount of symmetry between lines point that is the closest one to all lines in the cluster:and points. The segment si is assigned to the cluster with the closest vi = argmin DLP (lj , p). (8) ppseudo-centroid. Formally, sj ∈ Cluster i cluster(si ) = j | cj = argmin DLP (li , ck ), (4) IV. E STIMATING THE VANISHING POINT FOR THE ck ,k∈[1,M ] VERTICAL DIRECTION AND THE VANISHING LINE FOR THEwhere li is the line corresponding to the segment si . GROUND PLANE 3) Update step: The update step comprises the selection of As shown in [8], the vanishing points for the vertical direc-the new seeds d1i and d2i , for each cluster i = 1, . . . , M . tion are constrained to fall on the image y-axis, under normal The choice of the new seeds is such that they minimize camera positions. Furthermore, they tend to be well separatedthe error to the lines that would pass through the real corre- from the vanishing points for the horizontal directions [8].sponding vanishing point. The seed d1i minimizes the distance Considering this constraint, the first condition to detect theto the mean line of cluster i and d2i is chosen so that the cluster associated to the vertical direction is for the cluster tonew pseudo-centroid ci minimizes the distance to some key have a little to no offset between the mean orientation of itsintersection points. segments (Equation 5) and the image y-axis. But this condition The seed d1i is the line corresponding to a segment assigned is not sufficient. Observe, for example, the clusters illustratedto cluster i, that minimizes the angular distance to the weighted in Figure 2.circular mean orientation of the cluster. The angular distance isthe smallest angle between two orientations; and the weighted Both mean orientations, represented by the dashed lines incircular mean orientation θi for cluster i, considering the Figure 2, have the same orientation and have no offset toquality values as the weights, is computed as [7] the image y-axis. However, if they occur in the same image, we want to choose the cluster in Figure 2(a) to be the one S associated with the vertical direction. θi = arctan , (5) C To accomplish this, we have to consider the distribution ofwhere S and C correspond to the lines’ orientation in the same cluster. In Figure 2(a), the n standard deviation of the lines’ orientation is lower than in S= qj ∗ sin(2θj ), (6) Figure 2(b). Considering this observation, the second condition j=1 to detect the cluster associated to the vertical direction is for
  4. 4. the cluster to have low circular standard deviation, definedas [7]: −2ln R σ= , (9) 2 ¯where R corresponds to √ S2 + C2 R= n . (10) j=1 qj The two conditions together are sufficient to select thecluster associated to the vertical direction. Formally, we wantto select the cluster i that minimizes π Dang θi , + σi , (11) 2where Dang is the smallest angle between the orientations,θi is the circular mean of the lines’ orientation in cluster i Fig. 3. The first column shows the input image and all detected segments.(Equation 5) and σi is the circular standard deviation in cluster The second column shows the line clustering result and the detected finitei (Equation 9). vanishing points. Parallel lines with the same color are associated with a vanishing point at infinity; the other lines are associated with finite vanishing The vanishing point v associated with the cluster that points.minimizes Equation 11 is the vertical vanishing point. Following the same idea, to select the vanishing line for theground plane, first we consider all lines passing through alldetected vanishing points. We select the line that minimizes Given the camera pixel dimension, the principal point andthe angular distance to the image x-axis. If only two vanishing the skew factor, provided in the York Urban Database, wepoints are available, the vanishing line for the ground plane can construct the intrinsic parameters matrix K of the the line that passes through the non-vertical vanishing Given K, the Image of the Absolute Conic (IAC) ω is givenpoint and which orientation is the same as the circular mean byorientation of the respective cluster. ω = K−T K−1 . (12) V. E XPERIMENTS Let vi , i = 1, . . . , M , be the estimated vanishing points. The more orthogonal triplet minimizes the orthogonality error The method was implemented in C++ and we conductedthe experiments in the York Urban database [9]. It consists of ep,q,r = (vp ωvq )2 + (vq ωvr )2 + (vr ωvp )2 . (13)102 indoor and outdoor images of man-made environments.The York Urban database also provides the camera intrinsic A perfectly orthogonal triplet leads to a zero orthogonalityparameters as well as vanishing points computed with hand- error. To select the more orthogonal vanishing points detecteddetected segments. by our method, we choose the triplet that minimizes Equa- Figure 3 illustrates a few obtained results for the detection tion 13.of vanishing points. The first column shows input images with Figure 4 shows the cumulative orthogonality error histogramthe segments and the second column shows the line clustering for our method and for the method provided in York Urbanresults and the location of the finite vanishing points. For database, called here as Ground Truth. In the database, theexperimental purposes, the number M of vanishing points was segments are detected by hand and the vanishing points areset for each image. computed by a Gaussian Sphere method. To show the effectiveness of the method, we executed two Considering the triplet of vanishing points that minimizesexperiments in the York Urban database: the orthogonality error, we can compute the focal length and compare it to the estimated focal length provided in the York • Computation of the orthogonality error of the detected Urban database. To compute the focal length, we recovered vanishing points. This experiment shows how much the the camera intrinsic matrix K by decomposing the IAC matrix more orthogonal vanishing points1 deviate from the real with unknown focal length. orthogonality on the image plane. We compared our method with three other vanishing point • Computation of the focal length error based on the detectors by comparing the focal length error generated by detected vanishing points and the intrinsic parameters each one. The method Almansa 2003 [10] detects line matrix for the camera. This experiment compares the fo- and vanishing regions based on the Helmholtz principle. For cal length computed from the more orthogonal vanishing comparison purposes, we have selected the center of the points with the expected focal length. detected regions as the vanishing points location. We called 1 The term more orthogonal vanishing points refers here to the detected this extension as Almansa 2003 + vpe. The method Tardifvanishing points that minimizes the orthogonality error (Equation 13). 2009 [11] uses the Canny edge detector and flood fill for
  5. 5. line detection and an algorithm called J-Linkage for vanishingpoint estimation. Figure 5 shows the cumulative focal length error histogramfor our method and for the other methods, using the YorkUrban database. We can see that for the critical part of thehistogram, where the focal length error is low, our methodprovides significant superior results. 1 0.8 (a) 0.6 0.4 0.2 Ground Truth Our method 0 0 1 2 3 4 5 Cumulative Orthogonality ErrorFig. 4. Cumulative histogram for the estimated errors on York Urban (b)Database. A point (x, y) represents the fraction y of images in the databasethat have error e < x. 1 0.8 0.6 0.4 (c) 0.2 Ground Truth Our method Tardif 2009 Almansa 2003 + vpe 0 0 50 100 150 200 250 300 Cumulative Focal Length Error (pixels)Fig. 5. Cumulative histogram for the focal length errors on York UrbanDatabase. A point (x, y) represents the fraction y of images in the databasethat have focal length error less than x. The two experiments shows the effectiveness of the pro- (d)posed method for vanishing point estimation. To measureheights in an image, we need to find the vertical direction Fig. 6. Measuring heights in single images. (a) and (c) input images and theand the vanishing line for the ground plane. This is done line clustering result. (b) and (d) the vanishing line, the reference object andas described in Section IV. If only one view of the scene the object available, we also need the height of a reference object tocompute an absolute value.
  6. 6. Figure 6 shows some examples of heights measured in [9] P. Denis, J. Elder, and F. Estrada, “Efficient edge-based methods forsingle images. The aim is to compute the height of a person estimating manhattan frames in urban imagery,” in European Conference on Computer Vision, 2008, pp. 197–210.relative to the height of a reference object. The mean error in [10] A. Almansa, A. Desolneux, and S. Vamech, “Vanishing point detectionestimating the height in Figures 6(b) and 6(d) is 0.18 cm. without any a priori information,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, no. 4, pp. 502–507, 2003. VI. C ONCLUSION [11] J.-P. Tardif, “Non-iterative approach for fast and accurate vanishing point detection,” International Conference on Computer Vision, pp. 1250– In this work, we presented a vanishing point detector and its 1257, 2009.application to photogrammetry. It is a useful tool for forensic [12] H.-T. T. G. Wang and Q. Wu, “What can we learn about the scene structure from three orthogonal vanishing points in images,” Patternscience, since it allows the measurement of heights in images Recognition Letters, vol. 30, no. 3, pp. 192–202, 2009.of man-made environments. Our method detects finite and infinite vanishing points,without any prior camera calibration, and since it is performedon an unbounded space – the projective plane – all vanishingpoints can be accurately estimated with no loss of geometricalinformation from the original image. The method is effective when applied to images of architec-tural environments, where there is a predominance of straightlines corresponding to different 3D orientations. It is important to notice that when only one view of thescene is available, the scale factor α cannot be recovereduniquely. So to compute heights of objects in the scene, suchas people, we need an additional reference in the scene. If anreference height is not provided, the estimated height will notbe absolute. However, if multiple views are available, it is possible tocompute α uniquely by observing one arbitrary point acrossthe multiple views [12]. The experiments show the effectiveness of the vanishingpoints estimation and its application to photogrammetry, wherethe height of a person was estimated within a small error range. ACKNOWLEDGMENT This work is primarily supported by CNPq grant141248/2009-2, with additional funding from NSF (grantsCNS-0729126, CNS-0721703, IIS-0808718, and CCF-0915661), CNPq (grants 309254/2007-8, 551007/2007-9,551623/2009-8 and 200717/2010-3), FAPESP, and CAPES. R EFERENCES [1] A. Criminisi, “Single-view metrology: Algorithms and applications,” in Proceedings of the 24th DAGM Symposium on Pattern Recognition, 2002, pp. 224–239. [2] S. Bramble, D. Compton, and L. Klasn, “Forensic image analysis,” in 13th INTERPOL Forensic Science Symposium, 2001. [3] O. Faugeras, Three-dimensional computer vision: a geometric viewpoint. Cambridge, MA, USA: MIT Press, 1993. [4] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed. Cambridge University Press, ISBN: 0521540518, 2004. [5] B. Caprile and V. Torre, “Using vanishing points for camera calibration,” International Journal of Computer Vision, vol. 4, no. 2, pp. 127–140, 1990. [6] A. Desolneux, L. Moisan, and J.-M. Morel, “Edge detection by helmholtz principle,” Journal of Mathematical Imaging and Vision, vol. 14, no. 3, pp. 271–284, 2001. [7] K. Mardia and P. Jupp, Directional statistics, ser. Wiley Series in Probability and Statistics. John Wiley and Sons, Nov. 1999. [8] A. Gallagher, “Using vanishing points to correct camera rotation in images,” in Proceedings of the 2nd Canadian conference on Computer and Robot Vision, 2005, pp. 460–467.