Multispectral Domain Invariant Image for Retrieval-based Place Recognition
•Download as PPTX, PDF•
0 likes•143 views
Github: https://github.com/sejong-rcv/MDII Paper: https://github.com/sejong-rcv/MDII/blob/master/MDII_paper.pdf [Abstract] Multispectral recognition has attracted increasing attention from the research community due to its potential competence for many applications from day to night. However, due to the domain shift between RGB and thermal image, it has still many challenges to apply and to use RGB domain-based tasks. To reduce the domain gap, we propose multispectral domain invariant framework, which leverages the unpaired image translation method to generate a semantic and strongly discriminative invariant image by enforcing novel constraints in the objective function. We demonstrate the efficacy of the proposed method on mainly multispectral place recognition task and achieve significant improvement compared to previous works. Furthermore, we test on multispectral semantic segmentation and unsupervised domain adaptations to prove the scalability and generality of the proposed method. We will open our source code and dataset.
Recommended
More Related Content
What's hot
What's hot (19)
Similar to Multispectral Domain Invariant Image for Retrieval-based Place Recognition
Similar to Multispectral Domain Invariant Image for Retrieval-based Place Recognition (20)
Recently uploaded
Recently uploaded (20)
Multispectral Domain Invariant Image for Retrieval-based Place Recognition
- 1. MDII: Multispectral Domain Invariant Image for Retrieval-based Place Recognition ICRA2020, Paris Daechan Han*, YuJin Hwang*, Namil Kim, Yukyung Choi Sejong University RCV Lab. / All-day Vision Team SEJONG-RCV *Equal contribution
- 2. SEJONG-RCV Problem definition How do we know our current place in any time of the day? “GPS-based localization” is a common method. 2 *From medium However, it is dangerous to rely only on GPS for self-driving cars.
- 3. SEJONG-RCV Problem solution How do we know our current place in any time of the day? 3 Sensitive Illumination Change Cost Low (Mapping & Localization) Ex) Mobileye REM Robust Illumination Change Cost High (Mapping & Localization) Ex) HD-Map RGB3D LiDAR DayNight RGB based Localization Lidar based Localization
- 4. SEJONG-RCV Problem solution How do we know our current place in any time of the day? 4 Thermal Multispectral Domain Invariant Image based Localization using Thermal Camera Robust Illumination Change Cost Low (Mapping & Localization) Ex) Mobileye REM Compatible with existing Database RGB3D LiDAR DayNight
- 5. SEJONG-RCV Problem solution How do we know our current place in any time of the day? 5 Thermal Multispectral Domain Invariant Image based Localization using Thermal Camera Multispectral Domain Invariant Image (MDII) RGB3D LiDAR DayNight Robust Illumination Change Cost Low (Mapping & Localization) Ex) Mobileye REM Compatible with existing Database
- 6. SEJONG-RCV6 Then, How can we the new domain be easily applied for Retrieval-based Place Recognition? Key Ideas RGB domain Existing domain Thermal domain new domain common domain
- 7. SEJONG-RCV7 Move the Multispectral domain invariant space Key Ideas RGB domain Existing domain Thermal domain new domain Then, How can we the new domain be easily applied for Retrieval-based Place Recognition? multispectral domain
- 9. SEJONG-RCV Baseline Method: CycleGAN 9 : RGB : Thermal 𝐗 𝐀 𝐗 𝐀𝐁 𝐗 𝐀𝐁𝐀 𝐗 𝐁𝑿 𝑩𝑨 𝐗 𝐁𝐀𝐁 𝐃 𝐁 𝐃 𝐁 𝐃 𝐀 𝐃 𝐀 𝐆 𝐁 𝐆 𝐀 G = Generator, D = Discriminator, X=image
- 10. SEJONG-RCV Proposed Method 10 : RGB : Invariant Image : Thermal 𝐗 𝐀 𝐗 𝐀𝐁 𝐗 𝐀𝐁𝐀 𝐗 𝐁𝑿 𝑩𝑨 𝐗 𝐁𝐀𝐁 𝐃 𝐁 𝐃 𝐁 𝐃 𝐀 𝐃 𝐀 𝐆 𝐁 𝐆 𝐀 𝐌 = Multispectral Domain Invariant Image (MDII) Encoder, G = Generator, D = Discriminator, Y = MDI Image 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫 𝐘 𝐀 𝐘 𝐁𝐀 𝐘 𝐀𝐁 𝐘 𝐁
- 11. SEJONG-RCV Proposed Method 11 𝐗 𝐀 𝐌 = Multispectral Domain Invariant Image (MDII) Encoder, G = Generator, D = Discriminator, Y = MDI Image : RGB : Invariant Image : Thermal
- 12. SEJONG-RCV Proposed Method 12 𝐗 𝐀 𝐌 = Multispectral Domain Invariant Image (MDII) Encoder, G = Generator, D = Discriminator, Y = MDI Image 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫 𝐘 𝐀 : RGB : Invariant Image : Thermal
- 13. SEJONG-RCV Proposed Method 13 𝐗 𝐀 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫 𝐌 = Multispectral Domain Invariant Image (MDII) Encoder, G = Generator, D = Discriminator, Y = MDI Image 𝐘 𝐀 𝐗 𝐀𝐁𝐆 𝐁 : RGB : Invariant Image : Thermal
- 14. SEJONG-RCV Proposed Method 14 𝐗 𝐀 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫 𝐌 = Multispectral Domain Invariant Image (MDII) Encoder, G = Generator, D = Discriminator, Y = MDI Image 𝐘 𝐀 𝐗 𝐀𝐁𝐆 𝐁𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫𝐘 𝐀𝐁 : RGB : Invariant Image : Thermal
- 15. SEJONG-RCV Proposed Method 15 𝐗 𝐀 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫 𝐌 = Multispectral Domain Invariant Image (MDII) Encoder, G = Generator, D = Discriminator, Y = MDI Image 𝐘 𝐀 𝐗 𝐀𝐁𝐆 𝐁𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫𝐆 𝐀 𝐘 𝐀𝐁𝐗 𝐀𝐁𝐀 : RGB : Invariant Image : Thermal
- 16. SEJONG-RCV Proposed Method 16 𝐗 𝐀 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫 𝐌 = Multispectral Domain Invariant Image (MDII) Encoder, G = Generator, D = Discriminator, Y = MDI Image 𝐘 𝐀 𝐗 𝐀𝐁𝐆 𝐁 𝐆 𝐀 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫𝐘 𝐀𝐁𝐗 𝐀𝐁𝐀 𝐃 𝐀 𝐃 𝐀 : RGB : Invariant Image : Thermal
- 17. SEJONG-RCV Proposed Method 17 𝐗 𝐁𝑿 𝑩𝑨 𝐗 𝐁𝐀𝐁 𝐃 𝐁 𝐃 𝐁 𝐆 𝐁 𝐆 𝐀 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫 𝐘 𝐁𝐀 𝐘 𝐁 𝐌 = Multispectral Domain Invariant Image (MDII) Encoder, G = Generator, D = Discriminator, Y = MDI Image : RGB : Invariant Image : Thermal start
- 18. SEJONG-RCV Proposed Method 18 𝐗 𝐀 𝐗 𝐀𝐁 𝐗 𝐀𝐁𝐀 𝐗 𝐁𝑿 𝑩𝑨 𝐗 𝐁𝐀𝐁 𝐃 𝐁 𝐃 𝐁 𝐃 𝐀 𝐃 𝐀 𝐆 𝐁 𝐆 𝐀 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫 𝐌𝐃𝐈𝐈 𝐞𝐧𝐜𝐨𝐝𝐞𝐫 𝐘 𝐀 𝐘 𝐁𝐀 𝐘 𝐀𝐁 𝐘 𝐁 𝐌 = Multispectral Domain Invariant Image (MDII) Encoder, G = Generator, D = Discriminator, Y = MDI Image : RGB : Invariant Image : Thermal
- 19. SEJONG-RCV Proposed Loss function : Consistency constraints (Cyclic) 19 𝐿 𝑐𝑦𝑐𝑙𝑒(𝐴) = 𝔾 𝐴 𝕄 𝔾 𝐵 𝕄 𝑋𝐴 − 𝑋𝐴 1 XA YA XABYABXABA XB YB XBAYBAXBAB Cycle consistency constraints XA XABA= MDII 's characteristics Exactly same
- 20. SEJONG-RCV Proposed Loss function : Consistency constraints (L1) 20 XA YA XABYABXABA XB YB XBAYBAXBAB Domain invariance constraints (L1) YA YAB= Exactly same MDII 's characteristics 𝐿 𝑑𝑜𝑚𝑎𝑖𝑛(𝐴) = 𝕄 𝔾 𝐵 𝕄 𝑋𝐴 − 𝕄 𝑋𝐴 1
- 21. SEJONG-RCV Proposed Loss function : Feature representation constraints (SSIM) 21 XA YA XABYABXABA XB YB XBAYBAXBAB Domain invariance constraints (SSIM) YA YAB≅ Same local structure MDII 's characteristics 𝐿 𝑑𝑖𝑠𝑐(𝐴) = 1 𝑁 (𝑖,𝑗) (1 − 𝑆𝑆𝐼𝑀(𝑌𝐴, 𝑌𝐴𝐵)) 2 MDII get discriminability
- 22. SEJONG-RCV Proposed Loss function : Feature representation constraints (Triplet) 22 XA FA XAB𝐹ABXABA XB FB XBAFBAXBAB Feature representation constraints (Triplet) FA 𝐹AB FB Near information Far information MDII 's characteristics 𝐿 𝑑𝑖𝑣𝑠(𝐴) = 𝑚𝑎𝑥 𝑑 𝐹𝐴, 𝐹𝐴𝐵 − 𝑑 𝐹𝐴, 𝐹𝐵 + 𝛼, 0 MDII get diversity
- 23. SEJONG-RCV Proposed Loss function : Full objective 23 𝐿 𝐴 = 𝐿 𝑎𝑑𝑣 𝐴 + 𝐿 𝑐𝑦𝑐𝑙𝑒 𝐴 +{𝜆0 𝐿 𝑑𝑜𝑚𝑎𝑖𝑛 𝐴 + 𝜆1 𝐿 𝑑𝑖𝑠𝑐 𝐴 + 𝜆2 𝐿 𝑑𝑖𝑣𝑠 𝐴 }
- 25. SEJONG-RCV Quantitative results: Place Recognition 25 DataBase Query RGB Thermal RGB Translated RGB MDII (RGB based) MDII (Thermal based)
- 26. SEJONG-RCV Qualitative results: Place Recognition 26 Ours Pix2PixHD VLAD BoWExcavate GT localization Predict localization Correct result Incorrect result Input Query
- 27. SEJONG-RCV Qualitative results: Place Recognition 27 Ours Pix2PixHD VLAD BoWExcavate GT localization Predict localization Correct result Incorrect result Input Query
- 28. SEJONG-RCV Qualitative results: Place Recognition 28 Ours Pix2PixHD VLAD BoWExcavate GT localization Predict localization Correct result Incorrect result Input Query
- 29. SEJONG-RCV Qualitative results: Place Recognition 29 Ours Pix2PixHD VLAD BoWExcavate GT localization Predict localization Correct result Incorrect result Input Query
- 30. SEJONG-RCV Qualitative results: Place Recognition 30 Ours Pix2PixHD VLAD BoWExcavate GT localization Predict localization Correct result Incorrect result Input Query
- 31. SEJONG-RCV Experimental results: Place Recognition (Ablation study) 31 Constrains Cyclic Consistency Domain Consistency Diversity (Triplelet) Discriminability (SSIM) ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ 2.8 16.7 23.6 24.7 AccuracyofTop-1in3.0m
- 32. SEJONG-RCV32 https://github.com/sejong-rcv/MDII You can download Dataset & Code Thank you Q & A