Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Like this document? Why not share!

- An analysis of contrast agent flow ... by ieeepondy 175 views
- Optical Flow на GPU by MSU GML VideoGroup 1942 views
- "Embedded Lucas-Kanade Tracking: Ho... by Embedded Vision A... 531 views
- Introduction to Optial Flow by Sylvain_Lobry 473 views
- Video Inpainting detection using in... by Cybersecurity Edu... 998 views
- MOTION FLOW by Yusuf Uzun 228 views

No Downloads

Total views

14,396

On SlideShare

0

From Embeds

0

Number of Embeds

34

Shares

0

Downloads

103

Comments

0

Likes

1

No embeds

No notes for slide

- 1. Improving Horn and Schunck’s Optical Flow Algorithm Sylvain Lobry Technical Report no 1202, JULY 2012 revision 2325Computing the optical ﬂow of a video has many applications including motion estimation, or as a ﬁrst steptowards video inpainting. The optical ﬂow equation can not be solved as is due to the aperture problem(two unknowns for one single equation). One set of algorithms tries to solve this equation based on aglobal strategy, that is, minimizing variations in the ﬂow. The ﬁrst of them is Horn-Schunck’s method.This algorithm, even though it globally works, has several drawbacks, including being slow and unableto ﬁnd large displacements. In order to solve those problems, many strategies have been developed. Wepresent these strategies and analyze the beneﬁts of a multi-layer strategy applied to Horn-Schunck.Calculer le ﬂux optique a de nombreuses applications telles que l’estimation de mouvement ou un pre-mier pas vers l’inpainting vidéo. L’équation du ﬂux optique ne peut pas être résolue telle quelle à causedu problème de fenêtrage (deux inconnues pour une seule équation). Un ensemble d’algorithmes essayede résoudre cette équation en se basant sur une stratégie globale, c’est-à-dire en essayant d’avoir des pe-tites variations dans le ﬂux. Le premier d’entre eux est l’algorithme d’Horn-Schunck. Celui-ci, même si ilmarche dans de nombreux cas, a plusieurs inconvénients, notamment celui d’être lent et de ne pas pou-voir trouver les grands déplacements. Dans le but de résoudre ces problèmes, plusieurs stratégies ont étédéveloppées. Dans ce rapport, nous présenterons les méthodes et analyserons les bénéﬁces apportés parune stratégie multi-échelle à l’algorithme d’Horn-Schunck.KeywordsOptical ﬂow, Horn-Schunck’s algorithm, multi-layer approach, multiscale Laboratoire de Recherche et Développement de l’Epita 14-16, rue Voltaire – F-94276 Le Kremlin-Bicêtre cedex – France Tél. +33 1 53 14 59 47 – Fax. +33 1 53 14 59 22 lrde@lrde.epita.fr – http://www.lrde.epita.fr/
- 2. 2Copying this documentCopyright c 2012 LRDE. Permission is granted to copy, distribute and/or modify this document under the terms ofthe GNU Free Documentation License, Version 1.2 or any later version published by the FreeSoftware Foundation; with the Invariant Sections being just “Copying this document”, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is provided in the ﬁle COPYING.DOC.
- 3. Contents1 Deﬁnition of the optical ﬂow 6 1.1 Basic equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2 The aperture problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Overview of existing methods 8 2.1 Local methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2 Global methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2.1 Basic idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2.2 Further improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Implementation of a method 10 3.1 Horn-Schunck’s method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.2 Coarse-to-ﬁne version . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.2.1 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.2.2 A generic way to implement pyramidal representation . . . . . . . . . . . 134 Results 14 4.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 4.1.1 Representation of the optical ﬂow . . . . . . . . . . . . . . . . . . . . . . . 14 4.1.2 Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.3 Analysis of the results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 Bibliography 21
- 4. AcknowledgmentsThe author thanks Thierry Géraud, Guillaume Lazzara, Coddy Levi and Roland Levillain.
- 5. IntroductionThe optical ﬂow is deﬁned as the apparent motion of brightness patterns in an image. Comput-ing it leads to many applications, including video compression and motion estimation. It is alsothe ﬁrst step of many video inpainting algorithms which is the reason why we want to work onit. The LRDE1 has taken part into the “Terra Rush” project which has been chosen to be one ofthe 18 projects from the “Investissement d’avenir”2 program sponsored by the Government ofFrance. In this project, one of our goal is the to remove subtitles embeded in videos. This can bedone using video inpainting, a subject developed by Levi (2012). Having the optical ﬂow com-puted over the image to be inpainted can be useful for an inpainting algorithm if well exploited. Even though the ﬁrst attempts of computing the optical ﬂow has been made in 1981 by Lucasand Kanade (1981), the subject is the object of many recent studies leading to several improve-ments. This report aims to give an overview of the state of the art in this domain and to providea fast implementation of one of these methods. In this technical report, we will ﬁrst give a formal deﬁnition of the problem. The existingmethods are then overviewed and classiﬁed. We review one of those algorithms in its basic ver-sion and in a coarse-to-ﬁne adaptation. Finally an evaluation method is proposed and appliedto our implementation. 1 http://lrde.epita.fr 2 http://investissement-avenir.gouvernement.fr/
- 6. Chapter 1Deﬁnition of the optical ﬂowThe ﬁrst deﬁnition of the optical ﬂow given in the introduction can be easily formalized intomathematical equations. Yet, we will see in this chapter the reasons that make this problemill-posed.1.1 Basic equationsIn the following, we will only handle the 2D case with 1D values (i.e. gray-level images). Let I(x, y, t) be the value of the (x, y) pixel at frame t and (u(x, y, t), v(x, y, t)), also denoted(u, v), the ﬂow at the (x, y) pixel at frame t. The main idea of most algorithms for computing optical ﬂow is to use the Brightness Con-stancy: from a frame to an other, the brightness of a pixel does not change according to the ﬂow.This can be written as : I(x, y, t) = I(x + u, y + v, t + 1) (1.1)Equation 1.1 can be approximated with a Taylor series as: ∂I ∂I ∂I I(x, y, t) ≈ I(x, y, t) + u +v + (1.2) ∂x ∂y ∂t ∂I ∂I ∂IIntroducing the abbreviations Ex = ∂x , Ey = ∂y and Et = ∂t , the following holds : uEx + vEy + Et ≈ 0 (1.3)1.2 The aperture problemEquation 1.3 is often denoted as the Optical Flow Contraint. As we can see, we have a singleequation with two unknowns. This leads to an ill-posed problem which is often called the Aper-ture Problem as illustrated by Figure 1.1. To compute the optical ﬂow from Equation 1.3, we must add additional constraints to obtaina system with more equations. As illustrated by Figure 1.1 , the optical ﬂow cannot always be
- 7. 7 Deﬁnition of the optical ﬂow (a) First frame. (b) Second frame.Figure 1.1: The Aperture Problem : we can not tell in which way the line is moving with onlythose 2 frames.determined by a human being for certain. However, most of the time, humans may have a goodintuition of the optical ﬂow in a real scene by using additional information such as the context. Finding the constraints that are the most likely replicating this behaviour is the main chal-lenge handled by articles about optical ﬂow. chapter 2 presents some of these methods.
- 8. Chapter 2Overview of existing methodsMethods for computing the optical ﬂow can be classiﬁed in two categories. The ﬁrst one isknown as local methods whereas the other takes a global approach to the problem.2.1 Local methodsLocal methods are based on the approximation that the ﬂow does not vary inside small regionsof an image. The ﬁrst method using this assumption is from Lucas and Kanade (1981). Following this simple idea, we can obtain the following equations, given a window of sizen × n and by denoting pk the k th pixel within this window: Ex (p1 )u + Ey (p1 )v + Et (p1 ) = 0 Ex (p2 )u + Ey (p2 )v + Et (p2 ) = 0 ... Ex (pn2 )u + Ey (pn2 )v + Et (pn2 ) = 0 Thus, this assumption allows us to have n2 equations for 2 unknowns, giving a simple systemto solve. Further developments of this method includes a coarse-to-ﬁne implementation by pyramidalrepresentation of the image. This have two advantages : • We can take into account larger displacements. • It can speed-up computations. This coarse-to-ﬁne implementations seem to be the most used optical ﬂow computation al-gorithm into video inpainting solutions. On the opposite, it has not been the object of manyimprovements during the recent years for optical ﬂow algorithms. Indeed global methods seemto have more potential since they do not rely on simplistic assumptions.
- 9. 9 Overview of existing methods2.2 Global methods2.2.1 Basic ideaGlobal methods generally work by favoring small ﬁrst-order derivates of the ﬂow ﬁeld. Theﬁrst method to follow this idea is from Horn and Schunck (1981) who consider the optical ﬂowcomputation by deﬁning the measure of departure from smoothness in the ﬂow using an L2norm : 2 2 2 2 2 ∂u ∂u ∂v ∂v Es = + + + (2.1) ∂x ∂y ∂x ∂y Using Equation 1.3 (denoted Ec = uEx + vEy + Et ) and Equation 2.1 and introducing a ﬁxedconstant α2 to balance both terms, we can deﬁne the total error to be minimized : Err2 = (α2 Es + Ec ) dx dy 2 2 (2.2)We can note that having a high value for α2 will lead to smoother ﬂows. This factor is application-dependent.2.2.2 Further improvementsOther approaches include weighting Es along the magnitude of the gradient (| I|) to give lessimportance to areas having a high value for | I|. This is denoted by a function w. Indeed,it is more likely to ﬁnd ﬂow discontinuities at an edge than elsewhere. The equation to beminimized then becomes : Err2 = (α2 w(| I|)Es + Ec ) dx dy 2 2 (2.3)Equation 2.3 can be improved by adopting an anisotropic approach such as the one proposedby Werlberger et al. (2009). Nir et al. (2008) proposes to model the ﬂow ﬁeld with piecewise afﬁne regions and to penal-ize departure from it. This leads us to an over-parametrized approach which recognizes betterafﬁne regions. Trobin et al. (2008) takes the same idea and simplify it by using second order 2 2 ∂2u ∂2 ∂2 ∂2vderivates ( ∂ u , ∂ u , ∂x∂y , ∂xv , ∂yv , ∂x∂y ) for the prior term. ∂x2 ∂y 2 2 2 Another method which might have a lot of potential is proposed by Brox and Malik (2011). Itconsiders several energies including ones given by high-level descriptors such as SIFT. There-fore, the energy it tries to minimize is : Err2 = Ecolor + γEgradient + αEsmooth + βEmatch + Edesc (2.4)with γ, α and β being application-speciﬁc. As we can see, this method takes into account manyinformation in addition to the simple pixel value for a basic method such as the one proposed byHorn and Schunck (1981), with the drawback of being more expensive in terms of computation.
- 10. Chapter 3Implementation of a methodIn this chapter, we will ﬁrst explain which methods seen in chapter 2 could ﬁt our application.That is a method which is : • Fast, since we need real-time computation. • Robust enough to give good results, even in the presence of “holes” (the inpainting zone) in the image.Because it has to be fast, we made the choice to try a simple method, that is, with a low compu-tational effort. In addition to that criterion, we wanted a method which can evolve quite easily.The approach which seemed to ﬁt the best our needs is the one Horn and Schunck (1981). In-deed, since it uses the global approach, it is quite easy and straightforward to add constraintsso it can be more precise. Also, it is simple enough to be computed in real time.3.1 Horn-Schunck’s methodAs we explained in section 2.2, the method proposed by Horn and Schunck (1981) is a straightforward application of global methods principle to compute the optical ﬂow. In addition tothe basic equation of the optical ﬂow (Equation 1.3), it adds the measure of departure fromsmoothness (Equation 2.1). Equation 2.2 can be seen as a problem leading to the followingEuler-Lagrange equations : ∂(α2 Es + Ec ) 2 2 ∂ ∂(α2 Es + Ec ) 2 2 ∂ ∂(α2 Es + Ec ) 2 2 − − =0 (3.1) ∂u ∂x ∂ux ∂y ∂uy ∂(α2 Es + Ec ) 2 2 ∂ ∂(α2 Es + Ec ) 2 2 ∂ ∂(α2 Es + Ec ) 2 2 − − =0 (3.2) ∂v ∂x ∂vx ∂y ∂vy ∂2 ∂2 Which leads to the following equations (with ∆ = ∂x2 + ∂y 2 , the Laplacian operator) : Ex (Ex u + Ey v + Et ) − α2 ∆u = 0 (3.3) Ey (Ex u + Ey v + Et ) − α2 ∆v = 0 (3.4)
- 11. 11 Implementation of a method Using a ﬁnite distance, we can approximate ∆u by ∆u = u − u where u is the local average ¯ ¯of u. Therefore, Equation 3.3 and Equation 3.4 become : (α2 + Ex + Ey )(u − u) = −Ex (Ex u + Ey v + Et ) 2 2 ¯ ¯ ¯ (3.5) (α2 + Ex + Ey )(v − v ) = −Ey (Ex u + Ey v + Et ) 2 2 ¯ ¯ ¯ (3.6) From two consecutive frames, we can ﬁnd Ex , Ey and Et . Also, we deﬁne u and v to be the ¯ ¯weighted mean from u and v its 8-connectivity neighborhood. Having those terms, we can ﬁndan iterative solution to Equation 3.5 and Equation 3.6 : Ex un + Ey v n + Et ¯ ¯ un+1 = un − Ex ¯ (3.7) α2 + Ex + Ey 2 2 Ex un + Ey v n + Et ¯ ¯ v n+1 = v n − Ey ¯ 2 + E2 + E2 (3.8) α x y This iterative scheme is stopped when the mean evolution of (u, v) between two consecutiveiteration is too small. That is : 1 (un+1 − un )2 + (vi,j − vi,j )2 < ε2 i,j i,j n+1 n (3.9) Nsites i,j We also give an upper bound to the number of iterations, in order to ensure that the algorithmﬁnishes. We implemented this algorithm using Milena1 , leading to the results described in chapter 4.3.2 Coarse-to-ﬁne versionIn chapter 3, we have seen how to implement the method described by Horn and Schunck (1981)in its original version. In addition, we also implemented a multi-scale version of the algorithm.3.2.1 ObjectivesOne of our main objectives, described in 3 is to have a fast computation of the optical ﬂow.Horn-Schunck method being iterative, a classical way to speed-up computations is to use acoarse-to-ﬁne strategy. We downsample the original images (the two consecutive frames) repetitively a certain num-ber of time to obtain a pyramidal image illustrated by ﬁgure 3.1. We then run the algorithm to the lowest resolution image and ﬁnd our iterative solution. Thisﬂow is then propagated to the image below it so we can start from this solution and not havingto iterate from scratch. Another advantage of adapting this algorithm so it can be multiscale is the fact that we alsowant to recover large displacements. The original method from Horn and Schunck (1981) islikely to converge to a local minimum, leading to a wrong solution. With a multiscale approach, 1 http://lrde.epita.fr/cgi-bin/twiki/view/Olena/WebHome
- 12. 3.2 Coarse-to-ﬁne version 12Figure 3.1: A pyramid of images. We start by computing the optical ﬂow at the coarsest scale,then we propagate the results to the next scale before running the algorithm again. We keepdoing so until we have reached the ﬁnest scale.
- 13. 13 Implementation of a methodaccording to Fleet and Yair (2005), we avoid those local minima by guiding our convergence toa global minimum from scale to scale. So there are two main objectives that can be achieved by going multiscale : • Our solution is faster to compute. • We can recover large displacement by avoiding local minima.3.2.2 A generic way to implement pyramidal representationUsing the genericity model provided by C++ and Olena, we can implement a pyramidal imagethat can work for any source or destination value type. Therefore, we provide a simple classwhere the images can be accessed and modiﬁed before being downsampled or propagated. Thanks to this genericity model, coarse-to-ﬁne versions of classical iterative algorithms canbe implemented easily. More speciﬁcally, having a generic pyramidal image is very useful in our case, since most ofthe methods using the global approach provides an iterative solution. Since we only have genericity on value types, this generic model could be improved in twoways : • We could add the genericity on structures (for now, it only works for images in 2D). • We could even embed the algorithm inside the structure, so you would only have to give a functor to the class.The last two points were not useful for our application, but it could be an improvement toimplement them in a future work.
- 14. Chapter 4Results4.1 Methodology4.1.1 Representation of the optical ﬂowIn our algorithms, the optical ﬂow is stored as an image of 2D vectors. Such an image can notbe displayed easily and needs its own format to be stored in. We used the ideas provided byBaker et al. (2011).The evaluation database Baker et al. (2011) provides a database of consecutive frames withassociated ground truth on which we can bench the results found with our 2 versions of Horn-Schunck. We used the 8 sequences of images with public ground truth.The .ﬂo format We also used their format in order to store our results. This has three advan-tages: • We store our results with the same format as the ground truth. • We use the very same format as in many other papers which allows us to compare. • We can submit our results in order to be evaluated by the authors.This format is described by Baker1 and has been implemented in Olena.Image visualization Baker et al. (2011) introduce a new way of visualizing the optical ﬂow ina classical 2D color image. Most of the optical ﬂow visualization algorithms available use theway colors are encoded (RGB, CIE...) in order to deﬁne a mapping from a vector to a color. Thiscan give a wrong perception of the distance since the natural distance between two colors is notrelated to how they are encoded. Based on this observation, they use the work of Savard 2 inorder to map the vectors to colors based on actual human perception. 1 http://vision.middlebury.edu/flow/code/flow-code/README.txt 2 http://members.shaw.ca/quadibloc/other/colint.htm
- 15. 15 Results Figure 4.1: The map from vectors to colors. (a) First frame (b) Second frame Figure 4.2: Two consecutive frames
- 16. 4.1 Methodology 16Figure 4.3: A sample image of a computed optical ﬂow for frames 4.2 with ﬂow vectors mappedto the wheel. The ﬁgure 4.1 shows the actual map from vectors to colors and ﬁgure 4.3 shows a sampleresult of this visualization method. In Olena, this is a simple fun::v2v that can be applied with data::transform.4.1.2 MeasuresThere are 2 commonly used measures for the evaluation of an optical ﬂow algorithm. The ﬁrstof them is the angular error, the second, the error in ﬂow endpoint.Angular error (AE) The angular error has been introduced by Fleet and Jepson (1990) and isdeﬁned as the arc cos of the dot product of the two vectors over the product of their lengths : 1 + u × uGT + v × vGT AE = cos−1 ( √ ) (4.1) 1 + u2 + v 2 1 + u2 + vGT GT 2Error in ﬂow endpoint (EE) We also compute the error in ﬂow endpoint deﬁned by Otte andNagel (1994) as : EE = (u − uGT )2 + (v − vGT )2 (4.2)Baker et al. (2011) argues that this second measure should be favored since AE penalizes moreerrors in regions of zero motion than errors in smooth non-zero motion regions. In the following, we report both.
- 17. 17 Results (7, 0.1) (7, 0.01) (7, 0.001) (10, 0.1) (10, 0.01) (10, 0.001) (100, 0.1) (100, 0.01) (100, 0.001) Dimetrodon (1.94, 57) (1.73, 48) (1.58, 42) (1.96, 59) (1.78, 51) (1.56, 42) (2.06, 63) (2.04, 63) (1.84, 53.6) Grove2 (3.02, 66) (2.98, 63) (2.95, 62) (2.98, 66) (2.91, 63) (2.88, 61) (3.08, 71) (3.02, 68) (2.78, 57) Grove3 (3.78, 59) (3.70, 55) (3.68, 53) (3.72, 59) (3.61, 54) (3.57, 52) (3.90, 69) (3.77, 63) (3.48, 50) Hydrangea (3.48, 63) (3.27, 56) (3.17, 52) (3.52, 66) (3.27, 56) (3.13, 51) (3.73, 74) (3.69, 72) (3.40, 61) RubberWhale (0.97, 37) (0.53, 18) (0.38, 12) (1.04, 40) (0.59, 20) (0.39, 13) (1.25, 50) (1.23, 49) (0.98, 36) Urban2 (8.22, 61) (7.98, 53) (7.90, 50) (8.26, 63) (8.02, 54) (7.89, 50) (8.39, 69) (8.38, 69) (8.26, 62) Urban3 (7.16, 71) (7.00, 65) (6.92, 62) (7.18, 73) (6.99, 66) (6.88, 61) (7.30, 79) (7.29, 78) (7.13, 70) Venus (3.70, 63) (3.57, 57) (3.51, 55) (3.69, 65) (3.54, 58) (3.45, 54) (3.80, 71) (3.74, 68) (3.52, 58) Mean Time (s) 2.10 13.22 47.00 1.90 12.68 53.64 0.40 3.98 70.18Table 4.1: The results in the form (Endpoint Error, Angular Error) compared to the ground truthfor the original version (7, 0.1) (7, 0.01) (7, 0.001) (10, 0.1) (10, 0.01) (10, 0.001) (100, 0.1) (100, 0.01) (100, 0.001) Dimetrodon (1.68, 45) (1.62, 43) (1.56, 41) (1.67, 46) (1.62, 43) (1.53, 40) (1.74, 47) (1.73, 47) (1.60, 42) Grove2 (2.77, 55) (2.90, 60) (2.92, 61) (2.65, 51) (2.80, 58) (2.84, 60) (2.53, 43) (2.51, 43) (2.47, 43) Grove3 (3.56, 49) (3.64, 52) (3.66, 52) (3.43, 47) (3.52, 50) (3.54, 51) (3.26, 40) (3.25, 39) (3.24, 39) Hydrangea (2.92, 42) (3.04, 47) (3.13, 51) (2.86, 39) (2.94, 43) (3.05, 48) (2.88, 38) (2.87, 37) (2.74, 34) RubberWhale (0.82, 29) (0.51, 17) (0.37, 12) (0.91, 33) (0.55, 19) (0.38, 12) (1.08, 41) (1.08, 41) (0.84, 29) Urban2 (7.86, 49) (7.89, 50) (7.89, 50) (7.83, 47) (7.86, 49) (7.86, 49) (7.78, 44) (7.80, 43) (7.85, 44) Urban3 (6.73, 54) (6.81, 58) (6.84, 58) (6.69, 53) (6.74, 56) (6.76, 57) (6.68, 50) (6.68, 50) (6.67, 49) Venus (3.25, 45) (3.39, 49) (3.48, 53) (3.19, 43) (3.28, 47) (3.39, 51) (3.14, 41) (3.13, 40) (3.09, 39) Mean Time (s) 2.23 12.23 49.64 2.02 11.28 53.85 0.67 2.93 58.32Table 4.2: The results in the form (Endpoint Error, Angular Error) compared to the ground truthfor our multi-layer version4.2 ResultsFor each of the 8 frames, we compute 9 outputs by varying on two terms : • α takes the values 7, 10 and 100; • takes the values 0.1, 0.01 and 0.001.Tables 4.1 and 4.2 show the full results and ﬁgures 4.4, 4.5 and 4.6 show detailed results forDimetrodon.4.3 Analysis of the resultsWith the results on the 8 images with 9 parameters showed in tables 4.1 and 4.2, we can makeseveral observations : • The results are always better with the multiscale version of Horn and Schunck (1981). • Sometimes, the single layer version is faster. • Only one tuning of parameters among the 9 introduced could be used in our real-time application : α = 100 and = 0.1. • Both errors give the same order and seem equivalent.As expected in section 3.2, the results are better with the multiscale version of the algorithm. On the opposite, the fact that this second version is sometimes slower was not expected. Wecan see 2 reasons for that to happen :
- 18. 4.3 Analysis of the results 18 Figure 4.4: The Angular Errors for Dimetrodon. Figure 4.5: The Endpoint Errors for Dimetrodon.
- 19. 19 Results Figure 4.6: The computing times for Dimetrodon. • The time taken for the initialization of the pyramid is long. Indeed, we have to downsam- ple the images. • The number of scales might be application dependant and need to be tuned in an auto- matic way. Finally, the tuning proposed (α = 100, = 0.1) with seven scales gives pretty good results fora small cost in term of time. This could be used in an inpainting application.
- 20. ConclusionOptical ﬂow algorithms have been the subject of many studies during the past years, leadingto many improvements as seen in chapter 2. We have chosen to implement one of the simplestmethod, Horn and Schunck (1981), and to improve it with a coarse-to-ﬁne version. We showed how this method could be suitable for a real-time application with appropriateparameters, and how a multi-scale version of it could improve the qualitative results by givingless errors than the original version.Future work Even though this method could be good enough for a prior to video inpainting,we still have some leads to explore including : • Reducing the domain on which we compute the optical ﬂow. Indeed, since we want to use it for video inpainting, we do not need to compute it for areas too far from the region to inpaint. • We could improve the performances of the pyramidal algorithm so it could be faster. We should also try to be as much generic as possible so it would be reusable in other projects. • Since the proposed method is fast enough, we can improve its qualitative performance by adopting a more sophisticated approach as described in chapter 2.
- 21. Chapter 5BibliographyBaker, Scharstein, Lewis, Roth, Black, and Szeliski (2011). A database and evaluation method-ology for optical ﬂow. International Journal of Computer Vision.Brox, T. and Malik, J. (2011). Large displacement optical ﬂow: Descriptor matching in varia-tional motion estimation. IEEE Trans. Pattern Anal. Mach. Intell., 33(3):500–513.Fleet, D. J. and Jepson, A. D. (1990). Computation of component image velocity from localphase information. International Journal of Computer Vision, 5:77–104. 10.1007/BF00056772.Fleet, D. J. and Yair, W. (2005). Optical ﬂow estimation. In Mathematical models for ComputerVision: The Handbook.Horn, B. K. P. and Schunck, B. G. (1981). Determining optical ﬂow. ARTIFICAL INTELLI-GENCE, 17:185–203.Levi (2012). Fast structure preserving inpainting.Lucas, B. D. and Kanade, T. (1981). An iterative image registration technique with an applica-tion to stereo vision. In Proceedings of the 7th international joint conference on Artiﬁcial intelligence- Volume 2, pages 674–679, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.Nir, T., Bruckstein, A. M., and Kimmel, R. (2008). Over-parameterized variational optical ﬂow.Int. J. Comput. Vision, 76(2):205–216.Otte, M. and Nagel, H. (1994). Optical ﬂow estimation: Advances and comparisons. In Ek-lundh, J.-O., editor, Computer Vision ECCV ’94, volume 800 of Lecture Notes in Computer Science,pages 49–60. Springer Berlin / Heidelberg.Trobin, W., Pock, T., Cremers, D., and Bischof, H. (2008). An unbiased second-order priorfor high-accuracy motion estimation. In Rigoll, G., editor, Pattern Recognition, volume 5096 ofLecture Notes in Computer Science, pages 396–405. Springer Berlin / Heidelberg.Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., and Bischof, H. (2009). Anisotropichuber-l1 optical ﬂow. In Proceedings of the British Machine Vision Conference (BMVC), London,UK. to appear.

No public clipboards found for this slide

×
### Save the most important slides with Clipping

Clipping is a handy way to collect and organize the most important slides from a presentation. You can keep your great finds in clipboards organized around topics.

Be the first to comment