SlideShare a Scribd company logo
Pixel Matching From Stereo Images
          Outline



•    Part One: Pixel Matching in 3D Reconstruction
    • Introduction
    • Basic local algorithm
    • Propagation-based algorithm
    • Reliability measure for propagation-based stereo matching
    • Conclusion




•    Part Two: Pixel Matching in Urban Environment
    • Introduction
    • Viewpoint normalization
    • Pixel matching in viewpoint normalized space
    • Conclusion



                                                                  2
Introduction
Acquisition Calibration Pixel Matching Reconstruction




                                                        3
Pixel Matching in 3D Reconstruction
Acquisition Calibration Pixel Matching Reconstruction




                                                        4
Pixel Matching in 3D Reconstruction
Acquisition Calibration Pixel Matching Reconstruction




                                                        5
6
Introduction
        Acquisition Calibration Pixel Matching Reconstruction



                                  Y


                              Z         X
    j       Left image                      j   Right image
                                  P i
                                  •
i

                         pl                       pr




                                                                7
Introduction
        Acquisition Calibration Pixel Matching Reconstruction



                                  Y


                              Z         X
    j       Left image                      j   Right image
                                  P i
                                  •
i

                         pl                       pr




                                                                8
Introduction
Acquisition Calibration Pixel Matching Reconstruction




                                                        9
Introduction
Acquisition Calibration Pixel Matching Reconstruction




                                                        10
Introduction
        Acquisition Calibration Pixel Matching Reconstruction



                                  Y


                              Z         X
    j       Left image                      j    Right image
                                  P i
                                  •
i



                         pl                     pr




                                                                11
Introduction
Acquisition Calibration Pixel Matching Reconstruction



                            Y


                     Z             X
              j      Right image
                      Left image
         i                   P
         i
                             •
                    pr                      pl
                         disparity vector




                   pr = pl + dl
                             
                               0
                      = pl +
                               d

                                                        12
Introduction
             Acquisition Calibration Pixel Matching Reconstruction




Left image                         Right image                       Disparity map




                                                                                     13
Introduction
Acquisition Calibration Pixel Matching Reconstruction




                                                        14
Pixel Matching From Stereo Images
          Outline



•    Part One: Pixel Matching in 3D Reconstruction
    • Introduction
    • Basic local algorithm
    • Propagation-based algorithm
    • Reliability measure for propagation-based stereo matching
    • Conclusion




•    Part Two: Pixel Matching in Urban Environment
    • Introduction
    • Viewpoint normalization
    • Pixel matching in viewpoint normalized space
    • Conclusion



                                                                  15
Basic Local Algorithm
Example Difficulties




Left image              Right image




                                      16
Basic Local Algorithm
Example Difficulties




Left image              Right image




                                      17
Basic Local Algorithm
Example Difficulties




Left image              Right image




                                      18
Basic Local Algorithm
Example Difficulties




Left image              Right image




                                      19
Basic Local Algorithm
Example Difficulties




Left image              Right image




                                      20
Basic Local Algorithm
Example Difficulties




                                      Search area

Left image              Right image




                                                21
Basic Local Algorithm
Example Difficulties




Left image                 Right image


         correlation




                       Candidates
                                         22
Basic Local Algorithm
Example Difficulties




Left image                 Right image


         correlation




                       Candidates
                                         23
Basic Local Algorithm
Example Difficulties




                        24
Basic Local Algorithm
Example Difficulties




                        25
Basic Local Algorithm
Example Difficulties




                        26
Basic Local Algorithm
Example Difficulties




                        27
Basic Local Algorithm
Example Difficulties




                        28
Basic Local Algorithm
Example Difficulties




                        29
Pixel Matching From Stereo Images
          Outline



•    Part One: Pixel Matching in 3D Reconstruction
    • Introduction
    • Basic local algorithm
    • Propagation-based algorithm
    • Reliability measure for propagation-based stereo matching
    • Conclusion




•    Part Two: Pixel Matching in Urban Environment
    • Introduction
    • Viewpoint normalization
    • Pixel matching in viewpoint normalized space
    • Conclusion



                                                                  30
Propagation-Based Algorithm
               Idea




•   Hypothesis: almost everywhere, two
    neighboring pixels are the projections of
    two neighboring scene points
    •   almost everywhere, two neighboring pixels
        have almost the same disparity
    •   reduction of the search area to the
        neighborhood of reliable matches (seeds)
        •   reduction of ambiguities and
                                                    The hypothesis is not valid
            computation time                         at depth discontinuities




                                                                                  31
Propagation-Based Algorithm
Example




Left image           Right image




                                   32
Propagation-Based Algorithm
Example




Left image           Right image




                                   33
Propagation-Based Algorithm
Example




                                   Search area

Left image           Right image




                                             34
Propagation-Based Algorithm
Example




Left image           Right image




                                   35
Propagation-Based Algorithm
Example




Left image           Right image




                                   36
Propagation-Based Algorithm
Example




Left image           Right image




                                   37
Propagation-Based Algorithm
         Example



•   Sequential approach: best-first strategy




                                              38
Pixel Matching From Stereo Images
         Outline



•    Part One: Pixel Matching in 3D Reconstruction
    • Introduction
    • Basic local algorithm
    • Propagation-based algorithm
    • Reliability measure for propagation-based stereo matching
    • Conclusion




•    Part Two: Pixel Matching in Urban Environment
    • Introduction
    • Viewpoint normalization
    • Pixel matching in viewpoint normalized space
    • Conclusion



                                                                  39
Reliability Measure




•   Correlation can be ambiguous
•   Reliability measure
    •   Unambiguity term
    •   Continuity term
    •   Color consistency term




                                   40
Reliability Measure
                    Unambiguity term Continuity term Color consistency term




•         The lower other candidates are unlikely to be matches, the higher the
          confidence is




                                                 correlation
correlation




                                   Candidates                                 Candidates
                 Ambiguity                                     No ambiguity




                                                                                           41
Reliability Measure
                    Unambiguity term Continuity term Color consistency term




•         The lower other candidates are unlikely to be matches, the higher the
          confidence is




                                                 Unambiguity
Unambiguity




                                   Candidates                                 Candidates
                 Ambiguity                                     No ambiguity




                                                                                           42
Reliability Measure
            Unambiguity term Continuity term Color consistency term

•   The better a candidate satisfies the hypothesis that two neighbors should
    have almost the same disparity, the higher the confidence is
                                                                          Search area
             Left image                                 Right image



                      Continuity




                                                   Distance of the candidate
                                   0               to the disparity given by
                                                           the seed                43
Reliability Measure
            Unambiguity term Continuity term Color consistency term

•   The better a candidate satisfies the hypothesis that two neighbors should
    have almost the same disparity, the higher the confidence is
                                                                          Search area
             Left image                                 Right image



                      Continuity




                                                   Distance of the candidate
                                   0               to the disparity given by
                                                           the seed                44
Reliability Measure
             Unambiguity term Continuity term Color consistency term


•   Two neighboring pixels are more likely to have almost the same color, the
    smallest the color difference between the left pixel of a seed and the current
    pixel is, the highest the confidence is


                   Color consistency




                                                 Color difference between
                                                  the pixel to match and
                                       0         the left pixel of the seed
                                                                                     45
Reliability Measure
             Unambiguity term Continuity term Color consistency term


•   Two neighboring pixels are more likely to have almost the same color, the
    smallest the color difference between the left pixel of a seed and the current
    pixel is, the highest the confidence is


                   Color consistency




                                                 Color difference between
                                                  the pixel to match and
                                       0         the left pixel of the seed
                                                                                     46
Reliability Measure
             Unambiguity term Continuity term Color consistency term


•   Two neighboring pixels are more likely to have almost the same color, the
    smallest the color difference between the left pixel of a seed and the current
    pixel is, the highest the confidence is


                   Color consistency




                                                 Color difference between
                                                  the pixel to match and
                                       0         the left pixel of the seed
                                                                                     47
Reliability Measure
             Unambiguity term Continuity term Color consistency term


•   Two neighboring pixels are more likely to have almost the same color, the
    smallest the color difference between the left pixel of a seed and the current
    pixel is, the highest the confidence is


                   Color consistency




                                                 Color difference between
                                                  the pixel to match and
                                       0         the left pixel of the seed
                                                                                     48
Reliability Measure
             Results

Left image             PRM   Pcorr   B.L.A.3⇥3   B.L.A.9⇥9




                                                             49
Reliability Measure
                 Evaluation

               100

               80             PRM
       C (%)




                                                                                              C(%)
                              Pcorr
               60
                              B.L.A. 3x3
               40             B.L.A. 9x9                      50%  D < 75%

                     0   5       10       15    20           25       30        35
                                           Images
      1.Cloth1 2.Aloe 3.Cloth3 4.Rocks2 5.Cloth4 6.Rocks1 7.Cloth2 8.Cones 9.Sawtooth 10.Barn1 11.D
           100
      19.Tsukuba 20.Baby1 21.Venus 22.Bowling2 23-24.Baby2-3 25.Laundry 26.Reindeer 27-28.Midd1-

     Fig. 2.80
             Comparison of different methods of matching. These graphs sho
       C(%)




     obtained with a density 50%  D < 75% (left) and the ones obtained w
            60

               40                                                     75%  D
      M ETHOD                    50%  D < 75%                    D 75%                    able s
      PRM ( s , c , t) 5                           6
                                (1, 750, 4.7 ⇥ 10 20)                          6
                                                            (1, 1000, 30 ⇥ 1035 )
                                                                      2.8
35             0
      Pcorr (s, t)
                                 10       15
                                       (2, 0.8)
                                                             25
                                                                   (2, 0.65)
                                                                                           This
                                           Images
      B.L.A. (t3⇥3 ; t9⇥9 )         (0.825; 0.825)               (0.725; 0.65)             ation
                                                                                               50
Conclusion




•  Pixel matching is difficult in a general context
•  Constraints help to simplify the problem
  • Geometrical constraint: epipolar geometry
• Reliability measure for propagation-based matching: unambiguity,
   continuity and color consistency constraints




                                                                     51
Pixel Matching From Stereo Images
          Outline



•    Part One: Pixel Matching in 3D Reconstruction
    • Introduction
    • Basic local algorithm
    • Propagation-based algorithm
    • Reliability measure for propagation-based stereo matching
    • Conclusion




•   Part Two: Pixel Matching in Urban Environment
    •   Introduction
    •   Viewpoint normalization
    •   Pixel matching in viewpoint normalized space
    •   Conclusion



                                                                  52
Introduction




•   Vision-based geotechnologies
    •   Knowing where things are
    •   Knowing where things are in relation to other things
    •   Interacting and making more informed decisions
•   The problem of vision can be constrained by the fact that in an urban
    environment, the scene is composed of facades that can be
    approximated by planes




                                                                            53
Pixel Matching From Stereo Images
          Outline



•    Part One: Pixel Matching in 3D Reconstruction
    • Introduction
    • Basic local algorithm
    • Propagation-based algorithm
    • Reliability measure for propagation-based stereo matching
    • Conclusion




•    Part Two: Pixel Matching in Urban Environment
    • Introduction
    • Viewpoint normalization
    • Pixel matching in viewpoint normalized space
    • Conclusion



                                                                  54
Viewpoint Normalization
Tilt Rectification




                          55
Viewpoint Normalization
Line grouping




                          56
Viewpoint Normalization
Line grouping




                          57
Viewpoint Normalization
Layout extraction




                          58
Viewpoint Normalization
Layout extraction




                          59
Viewpoint Normalization
Plane extraction




                          60
Viewpoint Normalization
Plane extraction




                          61
Pixel Matching From Stereo Images
          Outline



•    Part One: Pixel Matching in 3D Reconstruction
    • Introduction
    • Basic local algorithm
    • Propagation-based algorithm
    • Reliability measure for propagation-based stereo matching
    • Conclusion




•    Part Two: Pixel Matching in Urban Environment
    • Introduction
    • Viewpoint normalization
    • Pixel matching in viewpoint normalized space
    • Conclusion



                                                                  62
Pixel Matching
Affine constraint




                  63
Pixel Matching
Affine constraint




                  64
Pixel Matching
          Results




4 correct matches out of 27   80 correct matches out of 84




1 correct matches out of 14   94 correct matches out of 100
                                                              65
Conclusion




•   Pixel matching can be made easier in urban scenes when we have some
    knowledge on the structure of the scene
•   Augmented-reality application




                                                                          66
Conclusion

•   Pixel matching can be made easier in urban scenes when we have some
    knowledge on the structure of the scene
•   Augmented-reality application




             Science Building                         Science Building




                                                              +StratAG data

                                                                              67
Acknowledgment




•   Thank you for you attention
•   Research presented in this presentation was funded by a Strategic
    Research Cluster grant (07/SRC/I1169) by Science Foundation Ireland
    under the National Development Plan. The authors gratefully
    acknowledge this support




                                                                          68

More Related Content

Viewers also liked

Depth preserving warping for stereo image retargeting
Depth preserving warping for stereo image retargetingDepth preserving warping for stereo image retargeting
Depth preserving warping for stereo image retargeting
LogicMindtech Nologies
 
present
presentpresent
Pixel matching for binocular stereovision by propagation of feature points ma...
Pixel matching for binocular stereovision by propagation of feature points ma...Pixel matching for binocular stereovision by propagation of feature points ma...
Pixel matching for binocular stereovision by propagation of feature points ma...
Guillaume Gales
 
A Vision-Based Mobile Platform for Seamless Indoor/Outdoor Positioning
A Vision-Based Mobile Platform for Seamless Indoor/Outdoor PositioningA Vision-Based Mobile Platform for Seamless Indoor/Outdoor Positioning
A Vision-Based Mobile Platform for Seamless Indoor/Outdoor Positioning
Guillaume Gales
 
A Comparison of Block-Matching Motion Estimation Algorithms
A Comparison of Block-Matching Motion Estimation AlgorithmsA Comparison of Block-Matching Motion Estimation Algorithms
A Comparison of Block-Matching Motion Estimation Algorithms
Multimedia and Vision Laboratory at Universidad del Valle
 
Technical seminar ppt
Technical seminar pptTechnical seminar ppt
Technical seminar ppt
Tejas_srinivas
 
Stegnography
StegnographyStegnography
Stegnography
keerthi505
 
Stegnography
StegnographyStegnography
Stegnography
Mayank Garg
 
A novel data embedding method using adaptive pixel
A novel data embedding method using adaptive pixelA novel data embedding method using adaptive pixel
A novel data embedding method using adaptive pixel
Renuka Verma
 
Image encryption
Image encryptionImage encryption
Image encryption
rakshit2105
 
Reversible Data Hiding in Encrypted Image: A Review
Reversible Data Hiding in Encrypted Image: A ReviewReversible Data Hiding in Encrypted Image: A Review
Reversible Data Hiding in Encrypted Image: A Review
Editor IJMTER
 
Image Encryption in java ppt.
Image Encryption in java ppt.Image Encryption in java ppt.
Image Encryption in java ppt.
Pradeep Vishwakarma
 
Image proceesing with matlab
Image proceesing with matlabImage proceesing with matlab
Image proceesing with matlab
Ashutosh Shahi
 
Basics of Image Processing using MATLAB
Basics of Image Processing using MATLABBasics of Image Processing using MATLAB
Basics of Image Processing using MATLAB
vkn13
 
Image encryption and decryption
Image encryption and decryptionImage encryption and decryption
Image encryption and decryption
Aashish R
 
Real time image processing ppt
Real time image processing pptReal time image processing ppt
Real time image processing ppt
ashwini.jagdhane
 
Introduction to digital image processing
Introduction to digital image processingIntroduction to digital image processing
Introduction to digital image processing
Hossain Md Shakhawat
 
Steganography
Steganography Steganography
Steganography
Uttam Jain
 
Cryptography.ppt
Cryptography.pptCryptography.ppt
Cryptography.ppt
kusum sharma
 
Encryption presentation final
Encryption presentation finalEncryption presentation final
Encryption presentation final
adrigee12
 

Viewers also liked (20)

Depth preserving warping for stereo image retargeting
Depth preserving warping for stereo image retargetingDepth preserving warping for stereo image retargeting
Depth preserving warping for stereo image retargeting
 
present
presentpresent
present
 
Pixel matching for binocular stereovision by propagation of feature points ma...
Pixel matching for binocular stereovision by propagation of feature points ma...Pixel matching for binocular stereovision by propagation of feature points ma...
Pixel matching for binocular stereovision by propagation of feature points ma...
 
A Vision-Based Mobile Platform for Seamless Indoor/Outdoor Positioning
A Vision-Based Mobile Platform for Seamless Indoor/Outdoor PositioningA Vision-Based Mobile Platform for Seamless Indoor/Outdoor Positioning
A Vision-Based Mobile Platform for Seamless Indoor/Outdoor Positioning
 
A Comparison of Block-Matching Motion Estimation Algorithms
A Comparison of Block-Matching Motion Estimation AlgorithmsA Comparison of Block-Matching Motion Estimation Algorithms
A Comparison of Block-Matching Motion Estimation Algorithms
 
Technical seminar ppt
Technical seminar pptTechnical seminar ppt
Technical seminar ppt
 
Stegnography
StegnographyStegnography
Stegnography
 
Stegnography
StegnographyStegnography
Stegnography
 
A novel data embedding method using adaptive pixel
A novel data embedding method using adaptive pixelA novel data embedding method using adaptive pixel
A novel data embedding method using adaptive pixel
 
Image encryption
Image encryptionImage encryption
Image encryption
 
Reversible Data Hiding in Encrypted Image: A Review
Reversible Data Hiding in Encrypted Image: A ReviewReversible Data Hiding in Encrypted Image: A Review
Reversible Data Hiding in Encrypted Image: A Review
 
Image Encryption in java ppt.
Image Encryption in java ppt.Image Encryption in java ppt.
Image Encryption in java ppt.
 
Image proceesing with matlab
Image proceesing with matlabImage proceesing with matlab
Image proceesing with matlab
 
Basics of Image Processing using MATLAB
Basics of Image Processing using MATLABBasics of Image Processing using MATLAB
Basics of Image Processing using MATLAB
 
Image encryption and decryption
Image encryption and decryptionImage encryption and decryption
Image encryption and decryption
 
Real time image processing ppt
Real time image processing pptReal time image processing ppt
Real time image processing ppt
 
Introduction to digital image processing
Introduction to digital image processingIntroduction to digital image processing
Introduction to digital image processing
 
Steganography
Steganography Steganography
Steganography
 
Cryptography.ppt
Cryptography.pptCryptography.ppt
Cryptography.ppt
 
Encryption presentation final
Encryption presentation finalEncryption presentation final
Encryption presentation final
 

Recently uploaded

Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Pixlogix Infotech
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
akankshawande
 
AWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptxAWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptx
HarisZaheer8
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
tolgahangng
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - HiikeSystem Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
Hiike
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
saastr
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
Tatiana Kojar
 
dbms calicut university B. sc Cs 4th sem.pdf
dbms  calicut university B. sc Cs 4th sem.pdfdbms  calicut university B. sc Cs 4th sem.pdf
dbms calicut university B. sc Cs 4th sem.pdf
Shinana2
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Zilliz
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
Intelisync
 
Azure API Management to expose backend services securely
Azure API Management to expose backend services securelyAzure API Management to expose backend services securely
Azure API Management to expose backend services securely
Dinusha Kumarasiri
 
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Tatiana Kojar
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
innovationoecd
 

Recently uploaded (20)

Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
 
AWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptxAWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptx
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - HiikeSystem Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
 
dbms calicut university B. sc Cs 4th sem.pdf
dbms  calicut university B. sc Cs 4th sem.pdfdbms  calicut university B. sc Cs 4th sem.pdf
dbms calicut university B. sc Cs 4th sem.pdf
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
 
Azure API Management to expose backend services securely
Azure API Management to expose backend services securelyAzure API Management to expose backend services securely
Azure API Management to expose backend services securely
 
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
 

Pixel Matching from Stereo Images (Callan seminar)

  • 1.
  • 2. Pixel Matching From Stereo Images Outline • Part One: Pixel Matching in 3D Reconstruction • Introduction • Basic local algorithm • Propagation-based algorithm • Reliability measure for propagation-based stereo matching • Conclusion • Part Two: Pixel Matching in Urban Environment • Introduction • Viewpoint normalization • Pixel matching in viewpoint normalized space • Conclusion 2
  • 3. Introduction Acquisition Calibration Pixel Matching Reconstruction 3
  • 4. Pixel Matching in 3D Reconstruction Acquisition Calibration Pixel Matching Reconstruction 4
  • 5. Pixel Matching in 3D Reconstruction Acquisition Calibration Pixel Matching Reconstruction 5
  • 6. 6
  • 7. Introduction Acquisition Calibration Pixel Matching Reconstruction Y Z X j Left image j Right image P i • i pl pr 7
  • 8. Introduction Acquisition Calibration Pixel Matching Reconstruction Y Z X j Left image j Right image P i • i pl pr 8
  • 9. Introduction Acquisition Calibration Pixel Matching Reconstruction 9
  • 10. Introduction Acquisition Calibration Pixel Matching Reconstruction 10
  • 11. Introduction Acquisition Calibration Pixel Matching Reconstruction Y Z X j Left image j Right image P i • i pl pr 11
  • 12. Introduction Acquisition Calibration Pixel Matching Reconstruction Y Z X j Right image Left image i P i • pr pl disparity vector pr = pl + dl  0 = pl + d 12
  • 13. Introduction Acquisition Calibration Pixel Matching Reconstruction Left image Right image Disparity map 13
  • 14. Introduction Acquisition Calibration Pixel Matching Reconstruction 14
  • 15. Pixel Matching From Stereo Images Outline • Part One: Pixel Matching in 3D Reconstruction • Introduction • Basic local algorithm • Propagation-based algorithm • Reliability measure for propagation-based stereo matching • Conclusion • Part Two: Pixel Matching in Urban Environment • Introduction • Viewpoint normalization • Pixel matching in viewpoint normalized space • Conclusion 15
  • 16. Basic Local Algorithm Example Difficulties Left image Right image 16
  • 17. Basic Local Algorithm Example Difficulties Left image Right image 17
  • 18. Basic Local Algorithm Example Difficulties Left image Right image 18
  • 19. Basic Local Algorithm Example Difficulties Left image Right image 19
  • 20. Basic Local Algorithm Example Difficulties Left image Right image 20
  • 21. Basic Local Algorithm Example Difficulties Search area Left image Right image 21
  • 22. Basic Local Algorithm Example Difficulties Left image Right image correlation Candidates 22
  • 23. Basic Local Algorithm Example Difficulties Left image Right image correlation Candidates 23
  • 24. Basic Local Algorithm Example Difficulties 24
  • 25. Basic Local Algorithm Example Difficulties 25
  • 26. Basic Local Algorithm Example Difficulties 26
  • 27. Basic Local Algorithm Example Difficulties 27
  • 28. Basic Local Algorithm Example Difficulties 28
  • 29. Basic Local Algorithm Example Difficulties 29
  • 30. Pixel Matching From Stereo Images Outline • Part One: Pixel Matching in 3D Reconstruction • Introduction • Basic local algorithm • Propagation-based algorithm • Reliability measure for propagation-based stereo matching • Conclusion • Part Two: Pixel Matching in Urban Environment • Introduction • Viewpoint normalization • Pixel matching in viewpoint normalized space • Conclusion 30
  • 31. Propagation-Based Algorithm Idea • Hypothesis: almost everywhere, two neighboring pixels are the projections of two neighboring scene points • almost everywhere, two neighboring pixels have almost the same disparity • reduction of the search area to the neighborhood of reliable matches (seeds) • reduction of ambiguities and The hypothesis is not valid computation time at depth discontinuities 31
  • 34. Propagation-Based Algorithm Example Search area Left image Right image 34
  • 38. Propagation-Based Algorithm Example • Sequential approach: best-first strategy 38
  • 39. Pixel Matching From Stereo Images Outline • Part One: Pixel Matching in 3D Reconstruction • Introduction • Basic local algorithm • Propagation-based algorithm • Reliability measure for propagation-based stereo matching • Conclusion • Part Two: Pixel Matching in Urban Environment • Introduction • Viewpoint normalization • Pixel matching in viewpoint normalized space • Conclusion 39
  • 40. Reliability Measure • Correlation can be ambiguous • Reliability measure • Unambiguity term • Continuity term • Color consistency term 40
  • 41. Reliability Measure Unambiguity term Continuity term Color consistency term • The lower other candidates are unlikely to be matches, the higher the confidence is correlation correlation Candidates Candidates Ambiguity No ambiguity 41
  • 42. Reliability Measure Unambiguity term Continuity term Color consistency term • The lower other candidates are unlikely to be matches, the higher the confidence is Unambiguity Unambiguity Candidates Candidates Ambiguity No ambiguity 42
  • 43. Reliability Measure Unambiguity term Continuity term Color consistency term • The better a candidate satisfies the hypothesis that two neighbors should have almost the same disparity, the higher the confidence is Search area Left image Right image Continuity Distance of the candidate 0 to the disparity given by the seed 43
  • 44. Reliability Measure Unambiguity term Continuity term Color consistency term • The better a candidate satisfies the hypothesis that two neighbors should have almost the same disparity, the higher the confidence is Search area Left image Right image Continuity Distance of the candidate 0 to the disparity given by the seed 44
  • 45. Reliability Measure Unambiguity term Continuity term Color consistency term • Two neighboring pixels are more likely to have almost the same color, the smallest the color difference between the left pixel of a seed and the current pixel is, the highest the confidence is Color consistency Color difference between the pixel to match and 0 the left pixel of the seed 45
  • 46. Reliability Measure Unambiguity term Continuity term Color consistency term • Two neighboring pixels are more likely to have almost the same color, the smallest the color difference between the left pixel of a seed and the current pixel is, the highest the confidence is Color consistency Color difference between the pixel to match and 0 the left pixel of the seed 46
  • 47. Reliability Measure Unambiguity term Continuity term Color consistency term • Two neighboring pixels are more likely to have almost the same color, the smallest the color difference between the left pixel of a seed and the current pixel is, the highest the confidence is Color consistency Color difference between the pixel to match and 0 the left pixel of the seed 47
  • 48. Reliability Measure Unambiguity term Continuity term Color consistency term • Two neighboring pixels are more likely to have almost the same color, the smallest the color difference between the left pixel of a seed and the current pixel is, the highest the confidence is Color consistency Color difference between the pixel to match and 0 the left pixel of the seed 48
  • 49. Reliability Measure Results Left image PRM Pcorr B.L.A.3⇥3 B.L.A.9⇥9 49
  • 50. Reliability Measure Evaluation 100 80 PRM C (%) C(%) Pcorr 60 B.L.A. 3x3 40 B.L.A. 9x9 50%  D < 75% 0 5 10 15 20 25 30 35 Images 1.Cloth1 2.Aloe 3.Cloth3 4.Rocks2 5.Cloth4 6.Rocks1 7.Cloth2 8.Cones 9.Sawtooth 10.Barn1 11.D 100 19.Tsukuba 20.Baby1 21.Venus 22.Bowling2 23-24.Baby2-3 25.Laundry 26.Reindeer 27-28.Midd1- Fig. 2.80 Comparison of different methods of matching. These graphs sho C(%) obtained with a density 50%  D < 75% (left) and the ones obtained w 60 40 75%  D M ETHOD 50%  D < 75% D 75% able s PRM ( s , c , t) 5 6 (1, 750, 4.7 ⇥ 10 20) 6 (1, 1000, 30 ⇥ 1035 ) 2.8 35 0 Pcorr (s, t) 10 15 (2, 0.8) 25 (2, 0.65) This Images B.L.A. (t3⇥3 ; t9⇥9 ) (0.825; 0.825) (0.725; 0.65) ation 50
  • 51. Conclusion • Pixel matching is difficult in a general context • Constraints help to simplify the problem • Geometrical constraint: epipolar geometry • Reliability measure for propagation-based matching: unambiguity, continuity and color consistency constraints 51
  • 52. Pixel Matching From Stereo Images Outline • Part One: Pixel Matching in 3D Reconstruction • Introduction • Basic local algorithm • Propagation-based algorithm • Reliability measure for propagation-based stereo matching • Conclusion • Part Two: Pixel Matching in Urban Environment • Introduction • Viewpoint normalization • Pixel matching in viewpoint normalized space • Conclusion 52
  • 53. Introduction • Vision-based geotechnologies • Knowing where things are • Knowing where things are in relation to other things • Interacting and making more informed decisions • The problem of vision can be constrained by the fact that in an urban environment, the scene is composed of facades that can be approximated by planes 53
  • 54. Pixel Matching From Stereo Images Outline • Part One: Pixel Matching in 3D Reconstruction • Introduction • Basic local algorithm • Propagation-based algorithm • Reliability measure for propagation-based stereo matching • Conclusion • Part Two: Pixel Matching in Urban Environment • Introduction • Viewpoint normalization • Pixel matching in viewpoint normalized space • Conclusion 54
  • 62. Pixel Matching From Stereo Images Outline • Part One: Pixel Matching in 3D Reconstruction • Introduction • Basic local algorithm • Propagation-based algorithm • Reliability measure for propagation-based stereo matching • Conclusion • Part Two: Pixel Matching in Urban Environment • Introduction • Viewpoint normalization • Pixel matching in viewpoint normalized space • Conclusion 62
  • 65. Pixel Matching Results 4 correct matches out of 27 80 correct matches out of 84 1 correct matches out of 14 94 correct matches out of 100 65
  • 66. Conclusion • Pixel matching can be made easier in urban scenes when we have some knowledge on the structure of the scene • Augmented-reality application 66
  • 67. Conclusion • Pixel matching can be made easier in urban scenes when we have some knowledge on the structure of the scene • Augmented-reality application Science Building Science Building +StratAG data 67
  • 68. Acknowledgment • Thank you for you attention • Research presented in this presentation was funded by a Strategic Research Cluster grant (07/SRC/I1169) by Science Foundation Ireland under the National Development Plan. The authors gratefully acknowledge this support 68

Editor's Notes

  1. Hello, thanks for coming. \nI&amp;#x2019;m going to talk about pixel matching \nthis presentation is divided into two parts...\n
  2. First, I&amp;#x2019;m going to talk about pixel matching in 3D reconstruction. \nThis is a part of some work I&amp;#x2019;ve made for my Ph.D. at the University of Toulouse with Alain Crouzil and Sylvie Chambon.\nThis is quite a general context. \n\nThen, in the second part, I will present some work made in the StratAG computer vision group where I&amp;#x2019;ve started my postdoc last October with John McDonald. \nThis part deals with computer vision in a geotechnology context, more especially about pixel matching in urban environment. \n\nThe two part are more or less unrelated. The thing is that computer vision requires pixel matching at different level, this is true for reconstruction (part 1) but pixel matching can also be needed for recognition or registration. \nThis presentation is going to show you how, according to a given context, we may use different constraints to simplify the problem by limiting the possible number of solutions. \n\nLet&amp;#x2019;s start with pixel matching in 3D reconstruction...\n
  3. First, I&amp;#x2019;m going to talk about pixel matching in 3D reconstruction. \nThis is a part of some work I&amp;#x2019;ve made for my Ph.D. at the University of Toulouse with Alain Crouzil and Sylvie Chambon.\nThis is quite a general context. \n\nThen, in the second part, I will present some work made in the StratAG computer vision group where I&amp;#x2019;ve started my postdoc last October with John McDonald. \nThis part deals with computer vision in a geotechnology context, more especially about pixel matching in urban environment. \n\nThe two part are more or less unrelated. The thing is that computer vision requires pixel matching at different level, this is true for reconstruction (part 1) but pixel matching can also be needed for recognition or registration. \nThis presentation is going to show you how, according to a given context, we may use different constraints to simplify the problem by limiting the possible number of solutions. \n\nLet&amp;#x2019;s start with pixel matching in 3D reconstruction...\n
  4. First, I&amp;#x2019;m going to talk about pixel matching in 3D reconstruction. \nThis is a part of some work I&amp;#x2019;ve made for my Ph.D. at the University of Toulouse with Alain Crouzil and Sylvie Chambon.\nThis is quite a general context. \n\nThen, in the second part, I will present some work made in the StratAG computer vision group where I&amp;#x2019;ve started my postdoc last October with John McDonald. \nThis part deals with computer vision in a geotechnology context, more especially about pixel matching in urban environment. \n\nThe two part are more or less unrelated. The thing is that computer vision requires pixel matching at different level, this is true for reconstruction (part 1) but pixel matching can also be needed for recognition or registration. \nThis presentation is going to show you how, according to a given context, we may use different constraints to simplify the problem by limiting the possible number of solutions. \n\nLet&amp;#x2019;s start with pixel matching in 3D reconstruction...\n
  5. First, I&amp;#x2019;m going to talk about pixel matching in 3D reconstruction. \nThis is a part of some work I&amp;#x2019;ve made for my Ph.D. at the University of Toulouse with Alain Crouzil and Sylvie Chambon.\nThis is quite a general context. \n\nThen, in the second part, I will present some work made in the StratAG computer vision group where I&amp;#x2019;ve started my postdoc last October with John McDonald. \nThis part deals with computer vision in a geotechnology context, more especially about pixel matching in urban environment. \n\nThe two part are more or less unrelated. The thing is that computer vision requires pixel matching at different level, this is true for reconstruction (part 1) but pixel matching can also be needed for recognition or registration. \nThis presentation is going to show you how, according to a given context, we may use different constraints to simplify the problem by limiting the possible number of solutions. \n\nLet&amp;#x2019;s start with pixel matching in 3D reconstruction...\n
  6. First, I&amp;#x2019;m going to talk about pixel matching in 3D reconstruction. \nThis is a part of some work I&amp;#x2019;ve made for my Ph.D. at the University of Toulouse with Alain Crouzil and Sylvie Chambon.\nThis is quite a general context. \n\nThen, in the second part, I will present some work made in the StratAG computer vision group where I&amp;#x2019;ve started my postdoc last October with John McDonald. \nThis part deals with computer vision in a geotechnology context, more especially about pixel matching in urban environment. \n\nThe two part are more or less unrelated. The thing is that computer vision requires pixel matching at different level, this is true for reconstruction (part 1) but pixel matching can also be needed for recognition or registration. \nThis presentation is going to show you how, according to a given context, we may use different constraints to simplify the problem by limiting the possible number of solutions. \n\nLet&amp;#x2019;s start with pixel matching in 3D reconstruction...\n
  7. First, I&amp;#x2019;m going to talk about pixel matching in 3D reconstruction. \nThis is a part of some work I&amp;#x2019;ve made for my Ph.D. at the University of Toulouse with Alain Crouzil and Sylvie Chambon.\nThis is quite a general context. \n\nThen, in the second part, I will present some work made in the StratAG computer vision group where I&amp;#x2019;ve started my postdoc last October with John McDonald. \nThis part deals with computer vision in a geotechnology context, more especially about pixel matching in urban environment. \n\nThe two part are more or less unrelated. The thing is that computer vision requires pixel matching at different level, this is true for reconstruction (part 1) but pixel matching can also be needed for recognition or registration. \nThis presentation is going to show you how, according to a given context, we may use different constraints to simplify the problem by limiting the possible number of solutions. \n\nLet&amp;#x2019;s start with pixel matching in 3D reconstruction...\n
  8. For reconstruction: 4 steps...\n
  9. For reconstruction: 4 steps...\n
  10. For reconstruction: 4 steps...\n
  11. For reconstruction: 4 steps...\n
  12. For reconstruction: 4 steps...\n
  13. \n
  14. \n
  15. \n
  16. \n
  17. \n
  18. \n
  19. \n
  20. \n
  21. \n
  22. \n
  23. \n
  24. \n
  25. \n
  26. \n
  27. \n
  28. \n
  29. \n
  30. \n
  31. \n
  32. \n
  33. \n
  34. \n
  35. \n
  36. \n
  37. \n
  38. \n
  39. \n
  40. \n
  41. \n
  42. \n
  43. \n
  44. To simplify the problem, we can use epipolar geometry.\nThere is a plane going through the scene point and the 2 pixels. This is the epipolar plane.\nThis plane intersects the image planes with two constrained lines, called epipolar lines. \nThis is interesting because these lines have a nice property: the correspondent of a pixel from one line is on the other constrained line.\n
  45. To simplify the problem, we can use epipolar geometry.\nThere is a plane going through the scene point and the 2 pixels. This is the epipolar plane.\nThis plane intersects the image planes with two constrained lines, called epipolar lines. \nThis is interesting because these lines have a nice property: the correspondent of a pixel from one line is on the other constrained line.\n
  46. \n
  47. What are we looking for ?\n
  48. \n
  49. \n
  50. \n
  51. \n
  52. My Ph.D. work focused on the pixel matching step and this is the part I&amp;#x2019;m going to detail a little bit more now. \n
  53. My Ph.D. work focused on the pixel matching step and this is the part I&amp;#x2019;m going to detail a little bit more now. \n
  54. They are 3 kinds of pixel matching methods:\n- local, the one I&amp;#x2019;m going to detail, they are called local because they are based on local similarities between pixels\n- global, they try to minimize a global cost function on the error of matching,\n- region-based, they are based on constraints given by regions of homogeneous color. These methods give the best results according to the Middlebury evaluation protocol which is the reference evaluation protocol, but they require an initialization step which is usually done by using a local algorithm.\n
  55. They are 3 kinds of pixel matching methods:\n- local, the one I&amp;#x2019;m going to detail, they are called local because they are based on local similarities between pixels\n- global, they try to minimize a global cost function on the error of matching,\n- region-based, they are based on constraints given by regions of homogeneous color. These methods give the best results according to the Middlebury evaluation protocol which is the reference evaluation protocol, but they require an initialization step which is usually done by using a local algorithm.\n
  56. They are 3 kinds of pixel matching methods:\n- local, the one I&amp;#x2019;m going to detail, they are called local because they are based on local similarities between pixels\n- global, they try to minimize a global cost function on the error of matching,\n- region-based, they are based on constraints given by regions of homogeneous color. These methods give the best results according to the Middlebury evaluation protocol which is the reference evaluation protocol, but they require an initialization step which is usually done by using a local algorithm.\n
  57. They are 3 kinds of pixel matching methods:\n- local, the one I&amp;#x2019;m going to detail, they are called local because they are based on local similarities between pixels\n- global, they try to minimize a global cost function on the error of matching,\n- region-based, they are based on constraints given by regions of homogeneous color. These methods give the best results according to the Middlebury evaluation protocol which is the reference evaluation protocol, but they require an initialization step which is usually done by using a local algorithm.\n
  58. They are 3 kinds of pixel matching methods:\n- local, the one I&amp;#x2019;m going to detail, they are called local because they are based on local similarities between pixels\n- global, they try to minimize a global cost function on the error of matching,\n- region-based, they are based on constraints given by regions of homogeneous color. These methods give the best results according to the Middlebury evaluation protocol which is the reference evaluation protocol, but they require an initialization step which is usually done by using a local algorithm.\n
  59. They are 3 kinds of pixel matching methods:\n- local, the one I&amp;#x2019;m going to detail, they are called local because they are based on local similarities between pixels\n- global, they try to minimize a global cost function on the error of matching,\n- region-based, they are based on constraints given by regions of homogeneous color. These methods give the best results according to the Middlebury evaluation protocol which is the reference evaluation protocol, but they require an initialization step which is usually done by using a local algorithm.\n
  60. They are 3 kinds of pixel matching methods:\n- local, the one I&amp;#x2019;m going to detail, they are called local because they are based on local similarities between pixels\n- global, they try to minimize a global cost function on the error of matching,\n- region-based, they are based on constraints given by regions of homogeneous color. These methods give the best results according to the Middlebury evaluation protocol which is the reference evaluation protocol, but they require an initialization step which is usually done by using a local algorithm.\n
  61. They are 3 kinds of pixel matching methods:\n- local, the one I&amp;#x2019;m going to detail, they are called local because they are based on local similarities between pixels\n- global, they try to minimize a global cost function on the error of matching,\n- region-based, they are based on constraints given by regions of homogeneous color. These methods give the best results according to the Middlebury evaluation protocol which is the reference evaluation protocol, but they require an initialization step which is usually done by using a local algorithm.\n
  62. They are 3 kinds of pixel matching methods:\n- local, the one I&amp;#x2019;m going to detail, they are called local because they are based on local similarities between pixels\n- global, they try to minimize a global cost function on the error of matching,\n- region-based, they are based on constraints given by regions of homogeneous color. These methods give the best results according to the Middlebury evaluation protocol which is the reference evaluation protocol, but they require an initialization step which is usually done by using a local algorithm.\n
  63. They are 3 kinds of pixel matching methods:\n- local, the one I&amp;#x2019;m going to detail, they are called local because they are based on local similarities between pixels\n- global, they try to minimize a global cost function on the error of matching,\n- region-based, they are based on constraints given by regions of homogeneous color. These methods give the best results according to the Middlebury evaluation protocol which is the reference evaluation protocol, but they require an initialization step which is usually done by using a local algorithm.\n
  64. They are 3 kinds of pixel matching methods:\n- local, the one I&amp;#x2019;m going to detail, they are called local because they are based on local similarities between pixels\n- global, they try to minimize a global cost function on the error of matching,\n- region-based, they are based on constraints given by regions of homogeneous color. These methods give the best results according to the Middlebury evaluation protocol which is the reference evaluation protocol, but they require an initialization step which is usually done by using a local algorithm.\n
  65. They are 3 kinds of pixel matching methods:\n- local, the one I&amp;#x2019;m going to detail, they are called local because they are based on local similarities between pixels\n- global, they try to minimize a global cost function on the error of matching,\n- region-based, they are based on constraints given by regions of homogeneous color. These methods give the best results according to the Middlebury evaluation protocol which is the reference evaluation protocol, but they require an initialization step which is usually done by using a local algorithm.\n
  66. \n
  67. \n
  68. \n
  69. \n
  70. \n
  71. \n
  72. \n
  73. \n
  74. \n
  75. \n
  76. \n
  77. \n
  78. \n
  79. Explain correlation = similarity measure. Different metrics have been proposed, if you are interested come back next month, Sylvie Chambon is going to talk about these in more details. \n
  80. Explain correlation = similarity measure. Different metrics have been proposed, if you are interested come back next month, Sylvie Chambon is going to talk about these in more details. \n
  81. \n
  82. \n
  83. - A lot of errors, because pixel matching is difficult...\n
  84. \n
  85. \n
  86. \n
  87. \n
  88. \n
  89. \n
  90. \n
  91. \n
  92. \n
  93. \n
  94. \n
  95. \n
  96. \n
  97. \n
  98. \n
  99. \n
  100. \n
  101. \n
  102. \n
  103. \n
  104. \n
  105. \n
  106. \n
  107. These difficulties, especially homogeneous areas and repetitive texture pattern, usually introduce ambiguities because, based on local similarities, we cannot say what pixel is the correspondent from a large list of candidates who are almost all alike.\nOne way to reduce these ambiguities, is to reduce the set of candidates. This is done by the epipolar rectification in some way, but we can go further using a propagation-based methods... \n
  108. These difficulties, especially homogeneous areas and repetitive texture pattern, usually introduce ambiguities because, based on local similarities, we cannot say what pixel is the correspondent from a large list of candidates who are almost all alike.\nOne way to reduce these ambiguities, is to reduce the set of candidates. This is done by the epipolar rectification in some way, but we can go further using a propagation-based methods... \n
  109. These difficulties, especially homogeneous areas and repetitive texture pattern, usually introduce ambiguities because, based on local similarities, we cannot say what pixel is the correspondent from a large list of candidates who are almost all alike.\nOne way to reduce these ambiguities, is to reduce the set of candidates. This is done by the epipolar rectification in some way, but we can go further using a propagation-based methods... \n
  110. These difficulties, especially homogeneous areas and repetitive texture pattern, usually introduce ambiguities because, based on local similarities, we cannot say what pixel is the correspondent from a large list of candidates who are almost all alike.\nOne way to reduce these ambiguities, is to reduce the set of candidates. This is done by the epipolar rectification in some way, but we can go further using a propagation-based methods... \n
  111. These difficulties, especially homogeneous areas and repetitive texture pattern, usually introduce ambiguities because, based on local similarities, we cannot say what pixel is the correspondent from a large list of candidates who are almost all alike.\nOne way to reduce these ambiguities, is to reduce the set of candidates. This is done by the epipolar rectification in some way, but we can go further using a propagation-based methods... \n
  112. These difficulties, especially homogeneous areas and repetitive texture pattern, usually introduce ambiguities because, based on local similarities, we cannot say what pixel is the correspondent from a large list of candidates who are almost all alike.\nOne way to reduce these ambiguities, is to reduce the set of candidates. This is done by the epipolar rectification in some way, but we can go further using a propagation-based methods... \n
  113. These difficulties, especially homogeneous areas and repetitive texture pattern, usually introduce ambiguities because, based on local similarities, we cannot say what pixel is the correspondent from a large list of candidates who are almost all alike.\nOne way to reduce these ambiguities, is to reduce the set of candidates. This is done by the epipolar rectification in some way, but we can go further using a propagation-based methods... \n
  114. These difficulties, especially homogeneous areas and repetitive texture pattern, usually introduce ambiguities because, based on local similarities, we cannot say what pixel is the correspondent from a large list of candidates who are almost all alike.\nOne way to reduce these ambiguities, is to reduce the set of candidates. This is done by the epipolar rectification in some way, but we can go further using a propagation-based methods... \n
  115. These difficulties, especially homogeneous areas and repetitive texture pattern, usually introduce ambiguities because, based on local similarities, we cannot say what pixel is the correspondent from a large list of candidates who are almost all alike.\nOne way to reduce these ambiguities, is to reduce the set of candidates. This is done by the epipolar rectification in some way, but we can go further using a propagation-based methods... \n
  116. These difficulties, especially homogeneous areas and repetitive texture pattern, usually introduce ambiguities because, based on local similarities, we cannot say what pixel is the correspondent from a large list of candidates who are almost all alike.\nOne way to reduce these ambiguities, is to reduce the set of candidates. This is done by the epipolar rectification in some way, but we can go further using a propagation-based methods... \n
  117. These difficulties, especially homogeneous areas and repetitive texture pattern, usually introduce ambiguities because, based on local similarities, we cannot say what pixel is the correspondent from a large list of candidates who are almost all alike.\nOne way to reduce these ambiguities, is to reduce the set of candidates. This is done by the epipolar rectification in some way, but we can go further using a propagation-based methods... \n
  118. These difficulties, especially homogeneous areas and repetitive texture pattern, usually introduce ambiguities because, based on local similarities, we cannot say what pixel is the correspondent from a large list of candidates who are almost all alike.\nOne way to reduce these ambiguities, is to reduce the set of candidates. This is done by the epipolar rectification in some way, but we can go further using a propagation-based methods... \n
  119. The propagation is based on the idea that almost everywhere, 2 neighboring pixels are the projections of 2 neighboring points from a same surface, and thus they should have almost the same disparity\nExample...\nKnowing that how can we reduce the search area? \n
  120. The propagation is based on the idea that almost everywhere, 2 neighboring pixels are the projections of 2 neighboring points from a same surface, and thus they should have almost the same disparity\nExample...\nKnowing that how can we reduce the search area? \n
  121. The propagation is based on the idea that almost everywhere, 2 neighboring pixels are the projections of 2 neighboring points from a same surface, and thus they should have almost the same disparity\nExample...\nKnowing that how can we reduce the search area? \n
  122. The propagation is based on the idea that almost everywhere, 2 neighboring pixels are the projections of 2 neighboring points from a same surface, and thus they should have almost the same disparity\nExample...\nKnowing that how can we reduce the search area? \n
  123. We start from a set of seeds = initial matches we assume reliable.\nTo do that, we use a feature point detection. \nThese points are special points in the image with special characteristics. \nWe are looking for pixels that stand out from the others such as corners or point with a lot of texture around \nbecause these points are easier to match than the others, \nso we can assume they can be matched with a high level of confidence.\n\n
  124. We select one seed, and now we are looking for ...\n
  125. We select one seed, and now we are looking for ...\n
  126. \n
  127. \n
  128. Then, the new match is added to the set of seeds and the process is reiterated until no new matches can be found.\n\n
  129. \n
  130. \n
  131. \n
  132. \n
  133. \n
  134. \n
  135. One thing I did not say is How to select the best seed at each iteration ? \nThe usual approach is to select the one with the highest correlation score. \nBut I realized that it&amp;#x2019;s not the best choice, and one of my contribution is to propose a reliability measure to use instead. \n\n
  136. One thing I did not say is How to select the best seed at each iteration ? \nThe usual approach is to select the one with the highest correlation score. \nBut I realized that it&amp;#x2019;s not the best choice, and one of my contribution is to propose a reliability measure to use instead. \n\n
  137. One thing I did not say is How to select the best seed at each iteration ? \nThe usual approach is to select the one with the highest correlation score. \nBut I realized that it&amp;#x2019;s not the best choice, and one of my contribution is to propose a reliability measure to use instead. \n\n
  138. One thing I did not say is How to select the best seed at each iteration ? \nThe usual approach is to select the one with the highest correlation score. \nBut I realized that it&amp;#x2019;s not the best choice, and one of my contribution is to propose a reliability measure to use instead. \n\n
  139. One thing I did not say is How to select the best seed at each iteration ? \nThe usual approach is to select the one with the highest correlation score. \nBut I realized that it&amp;#x2019;s not the best choice, and one of my contribution is to propose a reliability measure to use instead. \n\n
  140. One thing I did not say is How to select the best seed at each iteration ? \nThe usual approach is to select the one with the highest correlation score. \nBut I realized that it&amp;#x2019;s not the best choice, and one of my contribution is to propose a reliability measure to use instead. \n\n
  141. One thing I did not say is How to select the best seed at each iteration ? \nThe usual approach is to select the one with the highest correlation score. \nBut I realized that it&amp;#x2019;s not the best choice, and one of my contribution is to propose a reliability measure to use instead. \n\n
  142. One thing I did not say is How to select the best seed at each iteration ? \nThe usual approach is to select the one with the highest correlation score. \nBut I realized that it&amp;#x2019;s not the best choice, and one of my contribution is to propose a reliability measure to use instead. \n\n
  143. One thing I did not say is How to select the best seed at each iteration ? \nThe usual approach is to select the one with the highest correlation score. \nBut I realized that it&amp;#x2019;s not the best choice, and one of my contribution is to propose a reliability measure to use instead. \n\n
  144. One thing I did not say is How to select the best seed at each iteration ? \nThe usual approach is to select the one with the highest correlation score. \nBut I realized that it&amp;#x2019;s not the best choice, and one of my contribution is to propose a reliability measure to use instead. \n\n
  145. One thing I did not say is How to select the best seed at each iteration ? \nThe usual approach is to select the one with the highest correlation score. \nBut I realized that it&amp;#x2019;s not the best choice, and one of my contribution is to propose a reliability measure to use instead. \n\n
  146. One thing I did not say is How to select the best seed at each iteration ? \nThe usual approach is to select the one with the highest correlation score. \nBut I realized that it&amp;#x2019;s not the best choice, and one of my contribution is to propose a reliability measure to use instead. \n\n
  147. Why correlation is not the best choice ? \nBecause correlation is ambiguous. \nIt tells you if two neighborhoods are similar, but it doesn&amp;#x2019;t tell you anything about the other candidates that may be also similar (in homogeneous areas and repetitive texture patterns).\nSo I proposed to use a reliability measure instead of a correlation measure. This value is precomputed for the initial set of seed, then, it is computed during the matching step, instead of the correlation. \nThis RM uses an unambiguity term instead that takes into account this information.\nThen, I am also looking for two others conditions: continuity and color-consistency. \nBut let&amp;#x2019;s see these terms in more details...\n\n
  148. Why correlation is not the best choice ? \nBecause correlation is ambiguous. \nIt tells you if two neighborhoods are similar, but it doesn&amp;#x2019;t tell you anything about the other candidates that may be also similar (in homogeneous areas and repetitive texture patterns).\nSo I proposed to use a reliability measure instead of a correlation measure. This value is precomputed for the initial set of seed, then, it is computed during the matching step, instead of the correlation. \nThis RM uses an unambiguity term instead that takes into account this information.\nThen, I am also looking for two others conditions: continuity and color-consistency. \nBut let&amp;#x2019;s see these terms in more details...\n\n
  149. Why correlation is not the best choice ? \nBecause correlation is ambiguous. \nIt tells you if two neighborhoods are similar, but it doesn&amp;#x2019;t tell you anything about the other candidates that may be also similar (in homogeneous areas and repetitive texture patterns).\nSo I proposed to use a reliability measure instead of a correlation measure. This value is precomputed for the initial set of seed, then, it is computed during the matching step, instead of the correlation. \nThis RM uses an unambiguity term instead that takes into account this information.\nThen, I am also looking for two others conditions: continuity and color-consistency. \nBut let&amp;#x2019;s see these terms in more details...\n\n
  150. \n
  151. \n
  152. \n
  153. \n
  154. \n
  155. \n
  156. \n
  157. \n
  158. \n
  159. \n
  160. \n
  161. \n
  162. \n
  163. \n
  164. \n
  165. \n
  166. \n
  167. \n
  168. \n
  169. \n
  170. \n
  171. \n
  172. \n
  173. \n
  174. \n
  175. I&amp;#x2019;ve started my post-doc in october within the StratAG Computer Vision group. The context is different. \n
  176. I&amp;#x2019;ve started my post-doc in october within the StratAG Computer Vision group. The context is different. \n
  177. I&amp;#x2019;ve started my post-doc in october within the StratAG Computer Vision group. The context is different. \n
  178. I&amp;#x2019;ve started my post-doc in october within the StratAG Computer Vision group. The context is different. \n
  179. I&amp;#x2019;ve started my post-doc in october within the StratAG Computer Vision group. The context is different. \n
  180. I&amp;#x2019;ve started my post-doc in october within the StratAG Computer Vision group. The context is different. \n
  181. I&amp;#x2019;ve started my post-doc in october within the StratAG Computer Vision group. The context is different. \n
  182. I&amp;#x2019;ve started my post-doc in october within the StratAG Computer Vision group. The context is different. \n
  183. I&amp;#x2019;ve started my post-doc in october within the StratAG Computer Vision group. The context is different. \n
  184. I&amp;#x2019;ve started my post-doc in october within the StratAG Computer Vision group. The context is different. \n
  185. I&amp;#x2019;ve started my post-doc in october within the StratAG Computer Vision group. The context is different. \n
  186. I&amp;#x2019;ve started my post-doc in october within the StratAG Computer Vision group. The context is different. \n
  187. We are interested in vision-based geotechnologies. \nThis is about knowing where things are, \nhow they are related to each other, \nand interacting with these data.\n\nSo we are working with images taken in urban environment and we&amp;#x2019;ll see how the problem of vision can be constrained by the fact that we know the scenes show facades and that these facades can be approximated by planes.\n\nSo first I&amp;#x2019;m going to present Yanpeng&amp;#x2019;s work, the previous post-doc, on how to extract these planes from the images\n
  188. We are interested in vision-based geotechnologies. \nThis is about knowing where things are, \nhow they are related to each other, \nand interacting with these data.\n\nSo we are working with images taken in urban environment and we&amp;#x2019;ll see how the problem of vision can be constrained by the fact that we know the scenes show facades and that these facades can be approximated by planes.\n\nSo first I&amp;#x2019;m going to present Yanpeng&amp;#x2019;s work, the previous post-doc, on how to extract these planes from the images\n
  189. We are interested in vision-based geotechnologies. \nThis is about knowing where things are, \nhow they are related to each other, \nand interacting with these data.\n\nSo we are working with images taken in urban environment and we&amp;#x2019;ll see how the problem of vision can be constrained by the fact that we know the scenes show facades and that these facades can be approximated by planes.\n\nSo first I&amp;#x2019;m going to present Yanpeng&amp;#x2019;s work, the previous post-doc, on how to extract these planes from the images\n
  190. We are interested in vision-based geotechnologies. \nThis is about knowing where things are, \nhow they are related to each other, \nand interacting with these data.\n\nSo we are working with images taken in urban environment and we&amp;#x2019;ll see how the problem of vision can be constrained by the fact that we know the scenes show facades and that these facades can be approximated by planes.\n\nSo first I&amp;#x2019;m going to present Yanpeng&amp;#x2019;s work, the previous post-doc, on how to extract these planes from the images\n
  191. We are interested in vision-based geotechnologies. \nThis is about knowing where things are, \nhow they are related to each other, \nand interacting with these data.\n\nSo we are working with images taken in urban environment and we&amp;#x2019;ll see how the problem of vision can be constrained by the fact that we know the scenes show facades and that these facades can be approximated by planes.\n\nSo first I&amp;#x2019;m going to present Yanpeng&amp;#x2019;s work, the previous post-doc, on how to extract these planes from the images\n
  192. We are interested in vision-based geotechnologies. \nThis is about knowing where things are, \nhow they are related to each other, \nand interacting with these data.\n\nSo we are working with images taken in urban environment and we&amp;#x2019;ll see how the problem of vision can be constrained by the fact that we know the scenes show facades and that these facades can be approximated by planes.\n\nSo first I&amp;#x2019;m going to present Yanpeng&amp;#x2019;s work, the previous post-doc, on how to extract these planes from the images\n
  193. This is what the viewpoint normalization does.\n\n
  194. This is what the viewpoint normalization does.\n\n
  195. This is what the viewpoint normalization does.\n\n
  196. This is what the viewpoint normalization does.\n\n
  197. This is what the viewpoint normalization does.\n\n
  198. This is what the viewpoint normalization does.\n\n
  199. This is what the viewpoint normalization does.\n\n
  200. This is what the viewpoint normalization does.\n\n
  201. This is what the viewpoint normalization does.\n\n
  202. This is what the viewpoint normalization does.\n\n
  203. This is what the viewpoint normalization does.\n\n
  204. This is what the viewpoint normalization does.\n\n
  205. Tilt rectification: the tilt-rectified image shows the scene as if the camera was parallel to the ground\n
  206. Tilt rectification: the tilt-rectified image shows the scene as if the camera was parallel to the ground\n
  207. Tilt rectification: the tilt-rectified image shows the scene as if the camera was parallel to the ground\n
  208. Strips -&gt; for each strip find dominant direction\nOptimization to refine and cluster together the found directions\n
  209. \n
  210. \n
  211. \n
  212. \n
  213. \n
  214. \n
  215. Now, having this knowledge on the scene, we&amp;#x2019;ll see how we can match pixels between two views\n
  216. Now, having this knowledge on the scene, we&amp;#x2019;ll see how we can match pixels between two views\n
  217. Now, having this knowledge on the scene, we&amp;#x2019;ll see how we can match pixels between two views\n
  218. Now, having this knowledge on the scene, we&amp;#x2019;ll see how we can match pixels between two views\n
  219. Now, having this knowledge on the scene, we&amp;#x2019;ll see how we can match pixels between two views\n
  220. Now, having this knowledge on the scene, we&amp;#x2019;ll see how we can match pixels between two views\n
  221. Now, having this knowledge on the scene, we&amp;#x2019;ll see how we can match pixels between two views\n
  222. Now, having this knowledge on the scene, we&amp;#x2019;ll see how we can match pixels between two views\n
  223. Now, having this knowledge on the scene, we&amp;#x2019;ll see how we can match pixels between two views\n
  224. Now, having this knowledge on the scene, we&amp;#x2019;ll see how we can match pixels between two views\n
  225. Now, having this knowledge on the scene, we&amp;#x2019;ll see how we can match pixels between two views\n
  226. Now, having this knowledge on the scene, we&amp;#x2019;ll see how we can match pixels between two views\n
  227. Here, instead of matching pixels in the original view, we are going to match pixels in the viewpoint normalized view because it adds a geometrical constraint: the transformation between 2 extracted plane from 2 different images is a translation and a scaling (affine transformation)\n\n
  228. Here, instead of matching pixels in the original view, we are going to match pixels in the viewpoint normalized view because it adds a geometrical constraint: the transformation between 2 extracted plane from 2 different images is a translation and a scaling (affine transformation)\n\n
  229. Here, instead of matching pixels in the original view, we are going to match pixels in the viewpoint normalized view because it adds a geometrical constraint: the transformation between 2 extracted plane from 2 different images is a translation and a scaling (affine transformation)\n\n
  230. Here, instead of matching pixels in the original view, we are going to match pixels in the viewpoint normalized view because it adds a geometrical constraint: the transformation between 2 extracted plane from 2 different images is a translation and a scaling (affine transformation)\n\n
  231. Here, instead of matching pixels in the original view, we are going to match pixels in the viewpoint normalized view because it adds a geometrical constraint: the transformation between 2 extracted plane from 2 different images is a translation and a scaling (affine transformation)\n\n
  232. Here, instead of matching pixels in the original view, we are going to match pixels in the viewpoint normalized view because it adds a geometrical constraint: the transformation between 2 extracted plane from 2 different images is a translation and a scaling (affine transformation)\n\n
  233. Here, instead of matching pixels in the original view, we are going to match pixels in the viewpoint normalized view because it adds a geometrical constraint: the transformation between 2 extracted plane from 2 different images is a translation and a scaling (affine transformation)\n\n
  234. Here, instead of matching pixels in the original view, we are going to match pixels in the viewpoint normalized view because it adds a geometrical constraint: the transformation between 2 extracted plane from 2 different images is a translation and a scaling (affine transformation)\n\n
  235. Here, instead of matching pixels in the original view, we are going to match pixels in the viewpoint normalized view because it adds a geometrical constraint: the transformation between 2 extracted plane from 2 different images is a translation and a scaling (affine transformation)\n\n
  236. Here, instead of matching pixels in the original view, we are going to match pixels in the viewpoint normalized view because it adds a geometrical constraint: the transformation between 2 extracted plane from 2 different images is a translation and a scaling (affine transformation)\n\n
  237. \n
  238. -don&amp;#x2019;t perform too well in the original view -&gt; same difficulties than in part 1 !\n-the affine constraint limits the number of candidates, thus performs better\n
  239. -don&amp;#x2019;t perform too well in the original view -&gt; same difficulties than in part 1 !\n-the affine constraint limits the number of candidates, thus performs better\n
  240. -don&amp;#x2019;t perform too well in the original view -&gt; same difficulties than in part 1 !\n-the affine constraint limits the number of candidates, thus performs better\n
  241. -don&amp;#x2019;t perform too well in the original view -&gt; same difficulties than in part 1 !\n-the affine constraint limits the number of candidates, thus performs better\n
  242. -don&amp;#x2019;t perform too well in the original view -&gt; same difficulties than in part 1 !\n-the affine constraint limits the number of candidates, thus performs better\n
  243. -don&amp;#x2019;t perform too well in the original view -&gt; same difficulties than in part 1 !\n-the affine constraint limits the number of candidates, thus performs better\n
  244. -don&amp;#x2019;t perform too well in the original view -&gt; same difficulties than in part 1 !\n-the affine constraint limits the number of candidates, thus performs better\n
  245. -don&amp;#x2019;t perform too well in the original view -&gt; same difficulties than in part 1 !\n-the affine constraint limits the number of candidates, thus performs better\n
  246. -don&amp;#x2019;t perform too well in the original view -&gt; same difficulties than in part 1 !\n-the affine constraint limits the number of candidates, thus performs better\n
  247. -don&amp;#x2019;t perform too well in the original view -&gt; same difficulties than in part 1 !\n-the affine constraint limits the number of candidates, thus performs better\n
  248. PM is difficult but if we can have some knowledge on the scene we can impose constraints to reduce the number of possible solutions and thus to find good solutions. \nThis was the case in reconstruction using the epipolar geometry and some constraints on the continuity and the color consistency. \nThis was the case in the urban environment with the affine constraint on the extracted planes.\nNow, an application to that is...\n\n
  249. PM is difficult but if we can have some knowledge on the scene we can impose constraints to reduce the number of possible solutions and thus to find good solutions. \nThis was the case in reconstruction using the epipolar geometry and some constraints on the continuity and the color consistency. \nThis was the case in the urban environment with the affine constraint on the extracted planes.\nNow, an application to that is...\n\n
  250. PM is difficult but if we can have some knowledge on the scene we can impose constraints to reduce the number of possible solutions and thus to find good solutions. \nThis was the case in reconstruction using the epipolar geometry and some constraints on the continuity and the color consistency. \nThis was the case in the urban environment with the affine constraint on the extracted planes.\nNow, an application to that is...\n\n
  251. PM is difficult but if we can have some knowledge on the scene we can impose constraints to reduce the number of possible solutions and thus to find good solutions. \nThis was the case in reconstruction using the epipolar geometry and some constraints on the continuity and the color consistency. \nThis was the case in the urban environment with the affine constraint on the extracted planes.\nNow, an application to that is...\n\n
  252. \n
  253. \n
  254. \n
  255. \n
  256. \n