Visual Nouns for Indoor/Outdoor Navigation


Published on

Portable and Mobile Systems in Assistive Technology - Visual Nouns for Indoor/Outdoor Navigation - Zhu, Zhigang (f)

Published in: Business, Health & Medicine
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Visual cues – signs, visual markers, indoor/outdoorExample – store in Times square, recognizing logos around it…
  • Icons come from US DOTSigns can be general – CAFÉ can be in different style/font
  • Most markers are static => have I been here before? => estimate camera pose
  • Pros/consSigns are readily understandable by everyone.
  • AIGA = American Institute of Graphic Arts
  • Huge repository
  • Input: sequence of video framesFeature extraction
  • Mention signs…Natural occurring signs – fire, door
  • Stitched panorama
  • EE-MSER = edge enhancedDetection is from panorama
  • Set of 2D points and corresponding 3D points Perspective N-Point (PnP) problem for extrinsic camera parameter estimationCompare single vs. panorama (for localization purposes)Red = visual nounGreen = additional sign provided
  • Single view => bad pose, 4 featuresPanorama => 4 degrees off, 4 inches offPreliminary result/work
  • Generated a lot of false positiveFuture work – providing direction based on a query Main work is NOT to do sign detection, but for navigation and orientation.
  • Visual Nouns for Indoor/Outdoor Navigation

    1. 1. ICCHP 2012Visual Nouns for Indoor/OutdoorNavigationEdgardo Molina, Dr. Zhigang Zhu, Dr. Yingli TianDept. of Computer Science and Dept. of Electrical EngineeringGrove School of Engineering, The City College of New YorkICCHP 2012 – July 11, 2012 1
    2. 2. ICCHP 2012Motivation Assist the visually impaired in navigating new locations Provide guidance to facilities and destinations  Restrooms  Exits  Elevators  Water fountains How do the sighted recall a location? Visual cues? 2
    3. 3. ICCHP 2012What are visual nouns? Signs Text Icons 3
    4. 4. ICCHP 2012Using Visual Nouns Provide contextual information  What is it telling us? Provide landmark markers  Have I been here before?  Camera pose estimation => User Orientation 4
    5. 5. ICCHP 2012Technology Alternatives RFID  Embedded info  Costly QR Codes / Bar Codes / Etc  Embedded info  Only meaningful to machines GPS  Outdoors only Signs and Text are already in our environment 5
    6. 6. ICCHP 2012Standardized Signage US DOT and AIGA signs Various Organizations have standard signage 6
    7. 7. ICCHP 2012Visual Noun Project 7
    8. 8. ICCHP 2012Localizing a user1. Capture video of surroundings2. Register video image frames to the first frame (will serve as reference)3. Generate a wide field-of-view panorama4. Extract and match visual-noun features5. Localize the user in 3D space relative to the visual nouns. 8
    9. 9. ICCHP 2012Framework 9
    10. 10. ICCHP 2012Input Video 10
    11. 11. ICCHP 2012Panorama 11
    12. 12. ICCHP 2012Visual Noun Extraction Maximally Stable Extremal Regions (MSER)  Performed on a sharpened image  Use of EE-MSER could be better [Chen et. al. - ICIP’11] 12
    13. 13. ICCHP 2012Localization Localization done using PnP algorithm Compared number of features in a single view vs. a panorama 13
    14. 14. ICCHP 2012Localization Results 14
    15. 15. ICCHP 2012Summary We propose to use the signage that’s already in our environment to provide orientation and context information to visually impaired users. Low cost to print out more signage  Also beneficial to sighted users 15
    16. 16. ICCHP 2012Thank you!Questions?Work is supported by US NSF Emerging Frontiers in Research and InnovationProgram under Award No. EFRI-1137172, and City SEEDs: City College 2011President Grant for Interdisciplinary Scientific Research Collaborations. 16