Fast Global Stereo Matching Via Energy Pyramid Minimization
•
1 like•754 views
We define a global matching framework based on energy pyramid, the Global Matching via Energy Pyramid (GM-EP) algorithm, which estimates the disparity map from a single stereo-pair by solving an energy minimization problem. We efficiently address this minimization by globally optimizing a coarse to fine sequence of sparse Conditional Random Fields (CRF) directly defined on the energy. This global discrete optimization approach guarantees that at each scale we obtain a near optimal solution, and we demonstrate its superiority over state of the art image pyramid approaches through application to real stereo-pairs. We conclude that multiscale approaches should be build on energy pyramids rather than on image pyramids.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Fast Global Stereo Matching Via Energy Pyramid Minimization
Similar to Fast Global Stereo Matching Via Energy Pyramid Minimization (20)
Recently uploaded
Recently uploaded (20)
Fast Global Stereo Matching Via Energy Pyramid Minimization
- 1. ISPRS – PCV 2014 Fast global matching via energy pyramid (disparity esAmaAon) Zurich, 9/5/2014 Bruno Conejo, Phd student (bconejo@caltech.edu) with S. Leprince, F. Ayoub & JP. Avouac (GPS, Caltech)
- 2. B.Conejo -‐ Fast global Matching via Energy Pyramid 2 Disparity is inversely propor?onal to depth! Epipolar geometry stereo-‐imaging setup Introduc?on: Reference Image Target Image Disparity map
- 3. Given a stereo-‐pair of images (Ir ,It) how to retrieve the most probable disparity map d*? Regulariza?on: priors on disparity Matching: encourages similarity In term of probability, we need to es?mate the Maximum A Posteriori (MAP) of: B.Conejo -‐ Fast global Matching via Energy Pyramid 3 Modeling: a bayesian approach Gibbs measure relates probability density func?on to energy: Energy of configura?on x Normalizing constant Reference Image: Ir Target Image: It
- 4. From the Gibbs measure we relates probabili?es to the energies (EM , ER , E): Matching: Similarity criteria (L1, L2, ZNCC, ...) Regulariza?on: Piecewise constant prior: Modulated by radiometric discon?nuity: B.Conejo -‐ Fast global Matching via Energy Pyramid 4 Modeling: con?nuous Condi?onal Random Field (CRF) First order Condi?onal Random Field (CRF): p q Associated graph Reference Image Set of nodes Set of edges
- 5. We need to globally op?mize a con?nuous CRF over all possible disparity maps (D): However, this is a non-‐convex problem: varia?onal approaches can not work! B.Conejo -‐ Fast global Matching via Energy Pyramid 5 Modeling: non-‐convexity Many local minima!
- 6. Solu%on: Restrict d to take value in a finite discrete set, i.e., the “search space” encoded by a label space. This leads to globally op?mize a first order discrete CRF (s?ll NP-‐Hard) : -‐ Message passing (quadra?c w.r.t search space): Loopy BP, TRW-‐S, DD-‐MRF, … -‐ Making move (linear w.r.t search space) : α-‐exp, β-‐swap, Fast-‐PD, … B.Conejo -‐ Fast global Matching via Energy Pyramid Discrete op?miza?on 6 Pairwise term: Encodes prior (regulariza?on) Unary term: Encodes similarity (matching) Label space: Encodes for each node all poten?al disparity to evaluate
- 7. Mul?-‐scale approaches We work with large images (30,000 by 30,000) and we have large disparity range (-‐300,300). A direct approach is inefficient (even impossible) and unnecessary! Locally the disparity range is “small”. We can use a mul?-‐scale approach: -‐ Coarsest scales: “large” dispari?es with low spa?al frequencies (natural topography). -‐ Finest scales: “small” dispari?es with high spa?al frequencies (man made objects). Two mul?-‐scale schemes are possible: -‐ Image pyramid (classic, GM-‐IP algorithm). -‐ Energy pyramid (ours, GM-‐EP algorithm). B.Conejo -‐ Fast global Matching via Energy Pyramid 7 Reference Image Associated disparity
- 8. B.Conejo -‐ Fast global Matching via Energy Pyramid 8 Mul?-‐scale: GM-‐IP algorithm (Image Pyramid) Build & Opt. CRF Build & Opt. CRF Ir It Algorithm 1) Build pyramid of image for each image by itera?ve downsampling 2) Compute and op?mize CRF at coarsest scale 3) Define new search space around current solu?on 4) Repeat (2-‐3) un?l finest scale
- 9. B.Conejo -‐ Fast global Matching via Energy Pyramid 9 Mul?-‐scale: GM-‐EP algorithm (Energy Pyramid) Op?mize CRF Op?mize CRF Energy of CRF ◊©◊©◊ ◊©◊©◊ Algorithm: 1) Compute CRF at finest scale 2) Build energy pyramid by itera?ve downsampling 3) Op?mize CRF at coarsest scale 4) Define new search space around current solu?on 5) Repeat (3-‐4) un?l finest scale
- 10. B.Conejo -‐ Fast global Matching via Energy Pyramid 10 Mul?-‐scale: CRF sparsifica?on Label before op?m. Label amer op?m. Label space to explore Removed label range
- 11. B.Conejo -‐ Fast global Matching via Energy Pyramid 11 Mul?-‐scale: GM-‐IP(Image Pyramid) vs GM-‐EP (Energy Pyramid) The image pyramid yields a smoothed representa?on of the energy and destroys local minimums, especially at coarse scale: Different minima! Energy pyramid Image pyramid
- 12. B.Conejo -‐ Fast global Matching via Energy Pyramid 12 Stereo-‐imaging in urban context: (Reference Image)
- 13. B.Conejo -‐ Fast global Matching via Energy Pyramid 13 Stereo-‐imaging in urban context: (Target Image)
- 14. B.Conejo -‐ Fast global Matching via Energy Pyramid 14 Stereo-‐imaging in urban context: (Reference Image)
- 15. B.Conejo -‐ Fast global Matching via Energy Pyramid 15 Stereo-‐imaging in urban context: Area of interest in reference image Baseline = no pyramid (direct op@miza@on) Quan?ta?ve comparison between algorithms: Unary terms: ZNCC with 5x5 windows 4 scales, 1 itera?on per scale.
- 16. B.Conejo -‐ Fast global Matching via Energy Pyramid 16 Coarse scale: GM-‐EP (Energy Pyramid) vs GM-‐IP(Image Pyramid)
- 17. B.Conejo -‐ Fast global Matching via Energy Pyramid 17 Coarse scale: GM-‐EP (Energy Pyramid) vs GM-‐IP(Image Pyramid)
- 18. B.Conejo -‐ Fast global Matching via Energy Pyramid 18 Finest scale: GM-‐EP (Energy Pyramid) vs GM-‐IP(Image Pyramid)
- 19. B.Conejo -‐ Fast global Matching via Energy Pyramid 19 Finest scale: GM-‐EP (Energy Pyramid) vs GM-‐IP(Image Pyramid)
- 20. B.Conejo -‐ Fast global Matching via Energy Pyramid 20 GM-‐EP (Energy Pyramid) vs MicMac (Image Pyramid)
- 21. B.Conejo -‐ Fast global Matching via Energy Pyramid 21 GM-‐EP (Energy Pyramid) vs MicMac (Image Pyramid)
- 22. Key points: • A versa?le matching model efficiently op?mized with state of the art discrete op?miza?on technique. • Energy pyramid yields a beper representa?on of the energy. Future work: • Modeling: • Impact of images’ noise, • Symmetry w.r.t. the images, • Occlusions, • CRF parameters (unary terms, weights of CRF, distance func?on). • Op?miza?on • Auto defini?on of the search-‐space, • Mul?grid instead of mul?scale, • Paralleliza?on for shared and distributed memory architectures. B.Conejo -‐ Fast global Matching via Energy Pyramid 22 Conclusion & Future work:
- 23. B.Conejo -‐ Fast global Matching via Energy Pyramid 23
- 24. B.Conejo -‐ Fast global Matching via Energy Pyramid 24 Stereo-‐imaging in urban context: (Reference Image)
- 25. B.Conejo -‐ Fast global Matching via Energy Pyramid 25 Stereo-‐imaging in urban context: (Target Image)
- 26. B.Conejo -‐ Fast global Matching via Energy Pyramid 26 Finest scale: GM-‐EP (Energy Pyramid) vs MicMac (Image Pyramid) Micmac GM-‐EP (Energy Pyramid)
- 27. B.Conejo -‐ Fast global Matching via Energy Pyramid 27 Finest scale: GM-‐EP (Energy Pyramid) vs MicMac (Image Pyramid)
- 28. B.Conejo -‐ Fast global Matching via Energy Pyramid 28 Mul?-‐scale: GM-‐IP algorithm (Image Pyramid)
- 29. B.Conejo -‐ Fast global Matching via Energy Pyramid 29 Mul?-‐scale: GM-‐EP algorithm (Energy Pyramid)