Computer Engineering
Designing Assistive Technology
Roberto Manduchi
A personal experience of
trial and error
Research at JPL (circa 2000)
Research at JPL (circa 2000)
Sensors
Can we go from here …
Can we go from here …
… to here?
Services that “provide equipment or
systems, standardized or individualized,
whose aim is to improve or maintain the
funct...
Significant vision loss
(25 million)
Normal vision
Legally blind
(1.3 million)
Blind or light
perception
(0.25 million)
18...
Educational attainment:






With vision loss
U.S. average
< High school
High school Some college
Bachelor’s
13% 27%
23% ...
1. You cannot drive your car
2. You cannot read the paper
3. You may trip over an obstacle
4. You may miss a sign far away...
Success stories
Screen readers
Screen magnifiers
Braille interfaces
Enlargers/telescopes
Success stories
Accessible GPS
Money reader
Object recognition/crowdsourcing
OCR
Moving safely, gracefully, and comfortably
through the environment. Mobility
depends in large part on perceiving the
prope...
Mobility aids
110,000 users 7,000 users
Mobility aids
110,000 users 7,000 users
very few users
A laser-based virtual cane
• Active triangulation for
range estimation
Camera
Laser

pointer
• Range tracking for environm...
The many things a white cane can do…
• Bumper: contacts things that are in the direct path of travel
• Probe: an extension...
The many things a white cane can do…
• Bumper: contacts things that are in the direct path of travel
• Probe: an extension...
The process of navigating through an
environment and traveling to places by
relatively direct paths.
R.G. Long, E.W. Hill
...
Wayfinding without sight
The	 Love	 Stone	 at	 Kiyomizu	 Temple,	 Kyoto
• Prior information
–Maps
–Verbal directions
• Path integration
–Continuous update of egocentric coordinates of
starting l...
• Prior information
–Maps (tactile)
–Verbal directions
• Path integration
–Continuous update of egocentric coordinates of
...
GPS is only a partial solution
•Works only outdoors
•~10m resolution
Will take you to locations
and won’t get you lost, bu...
• Tactile paving
• Accessible pedestrian signals
• Light beaconing (Talking Signs)
• RFID, iBeacons
• iBeacons
Supporting ...
The problem with indoor navigation systems
Turn-by turn guidance only useful if you
can visually access the scene context
...
The promise of cameras
c1
c2
gorithm for the image set con-
r is shown. Di↵erent line sets
c1
c2
rithm for the image set c...
Scene descriptions: A proposal
• What if we could crowdsource collection of
local scene descriptions in a similar way to
O...
(Computer) Vision Without Sight
•How does a blind person operate a
camera system to recognize a
particular target?
•How can we design a system that
takes ...
Reading text
Mobile OCR works well…
…when you can take a 

good picture of the document!
Accessible Mobile OCR
•Accessible OCR apps often offer an
opportunistic discovery feature
–The system analyzes the video s...
Guided Mobile OCR
• Real-time text spotting and line detection
• Computes whether current frame is OCR-
readable (enough r...
Guided OCR - results
• Without system assistance or prior training, it
can be extremely difficult to acquire
readable imag...
Thank you
Upcoming SlideShare
Loading in …5
×

“Designing Assistive Technology: A Personal Experience of Trial and Error “

269 views

Published on

Roberto Manduchi presentation on Cognitive Systems Institute Group weekly speaker series call on November 5, 2015. For an audio replay, find ID here: http://cognitive-science.info/community/weekly-update/

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
269
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

“Designing Assistive Technology: A Personal Experience of Trial and Error “

  1. 1. Computer Engineering Designing Assistive Technology Roberto Manduchi A personal experience of trial and error
  2. 2. Research at JPL (circa 2000)
  3. 3. Research at JPL (circa 2000) Sensors
  4. 4. Can we go from here …
  5. 5. Can we go from here … … to here?
  6. 6. Services that “provide equipment or systems, standardized or individualized, whose aim is to improve or maintain the functional capabilities of individuals with disabilities”. M.J. Fuhrer, NIH Assistive Technology:
  7. 7. Significant vision loss (25 million) Normal vision Legally blind (1.3 million) Blind or light perception (0.25 million) 18-44 45-64 65-74 ≥75 Prevalence Age distribution
  8. 8. Educational attainment: 
 
 
 With vision loss U.S. average < High school High school Some college Bachelor’s 13% 27% 23% 31% 28% 18% Employment rate : – 19% of legally blind Americans 18 and older were employed in 1994-95 (NHIS, 1994-95) Pre-readers Non-readers Aud.readers Visual readersBraillereaders
  9. 9. 1. You cannot drive your car 2. You cannot read the paper 3. You may trip over an obstacle 4. You may miss a sign far away 5. You may not be able to cross a street safely 6. You may not find what you are looking for at the supermarket 7. You may get lost in a new place 8. You may not receive a proper education 9. You may have problems finding a job 10. You may not recognize friends from a distance 11. You may lose objects in your home 12. You may have problems surfing the Web 13. You may not know who is in the room 14. You may not be able to read this line If you cannot see well...
  10. 10. Success stories Screen readers Screen magnifiers Braille interfaces Enlargers/telescopes
  11. 11. Success stories Accessible GPS Money reader Object recognition/crowdsourcing OCR
  12. 12. Moving safely, gracefully, and comfortably through the environment. Mobility depends in large part on perceiving the properties of the immediate surroundings. R.G. Long, E.W. Hill Mobility:
  13. 13. Mobility aids 110,000 users 7,000 users
  14. 14. Mobility aids 110,000 users 7,000 users very few users
  15. 15. A laser-based virtual cane • Active triangulation for range estimation Camera Laser
 pointer • Range tracking for environmental feature detection D. Yuan, R. Manduchi (2005). Proc. CVPR.
  16. 16. The many things a white cane can do… • Bumper: contacts things that are in the direct path of travel • Probe: an extension of the sense of touch. • Finds, verifies, and discriminates landmarks • Helps establish the line of direction of travel –e.g. by trailing a straight edge • Detects drop offs • Measurement tool and…
  17. 17. The many things a white cane can do… • Bumper: contacts things that are in the direct path of travel • Probe: an extension of the sense of touch. • Finds, verifies, and discriminates landmarks • Helps establish the line of direction of travel –e.g. by trailing a straight edge • Detects drop offs • Measurement tool and… • Is cheap • Never runs out of power • Is light, durable, and foldable • Always works • Identifies the blind traveler • Puts drivers on alert when it moves into a street before the blind traveler
  18. 18. The process of navigating through an environment and traveling to places by relatively direct paths. R.G. Long, E.W. Hill Finding the way is not a gift or a innate ability… it is a precondition for life itself. Knowing where I am, my location, is the precondition for knowing where I have to go, wherever it may be. Otl Aicher Wayfinding:
  19. 19. Wayfinding without sight The Love Stone at Kiyomizu Temple, Kyoto
  20. 20. • Prior information –Maps –Verbal directions • Path integration –Continuous update of egocentric coordinates of starting location –Remembering the path traversed, turns, etc. • Piloting –Sensing one’s positional information to determine one’s location –Reading signs –Noticing landmarks (acoustic, tactile, smells, heat…) Wayfinding for sighted people
  21. 21. • Prior information –Maps (tactile) –Verbal directions • Path integration –Continuous update of egocentric coordinates of starting location –Counting steps, turns, etc. • Piloting –Sensing one’s positional information to determine one’s location –Reading signs (Braille) –Noticing landmarks (acoustic, tactile, smells, heat…) Wayfinding for blind people
  22. 22. GPS is only a partial solution •Works only outdoors •~10m resolution Will take you to locations and won’t get you lost, but… …where is the door?
  23. 23. • Tactile paving • Accessible pedestrian signals • Light beaconing (Talking Signs) • RFID, iBeacons • iBeacons Supporting Infrastructure
  24. 24. The problem with indoor navigation systems Turn-by turn guidance only useful if you can visually access the scene context Blind travelers need some form of scene knowledge – a representation of objects, spaces and surface at a scale normally not represented in maps
  25. 25. The promise of cameras c1 c2 gorithm for the image set con- r is shown. Di↵erent line sets c1 c2 rithm for the image set con- is shown. Di↵erent line sets ially those at a planar junc- extract and localize the main planar structures in the scene from two images (see Fig. 2) with substantially higher accuracy than with traditional point-based SFM. When structural constraints such as Manhattan world can be expected, the geometry of the environment can be reconstructed even from a single image [15] [50] [33]. This relatively new approach exploits Figure 3. 3-D planar reconstruction from single images, using an algorithm similar to [50]. The main planar structures, automatically detected by the algorithm, are shown bordered by thick dark lines, and colored according to their orientation. (First two images from [60].) A portable camera coupled with robust computer vision algorithms could provide spatial context… …but how are you going to communicatespatial information? 3-D ambient reconstruction crosswalk localization text spotting
  26. 26. Scene descriptions: A proposal • What if we could crowdsource collection of local scene descriptions in a similar way to OpenStreetMap? • Scene descriptions could then be converted to narrative maps or to tactile maps
  27. 27. (Computer) Vision Without Sight
  28. 28. •How does a blind person operate a camera system to recognize a particular target? •How can we design a system that takes into account the human factors associated with its use?
  29. 29. Reading text
  30. 30. Mobile OCR works well… …when you can take a 
 good picture of the document!
  31. 31. Accessible Mobile OCR •Accessible OCR apps often offer an opportunistic discovery feature –The system analyzes the video stream from the camera –When a “good” frame is detected, this is passed on to OCR Prizmo Text Detective
  32. 32. Guided Mobile OCR • Real-time text spotting and line detection • Computes whether current frame is OCR- readable (enough resolution, enough margin) • If not, produces guidance instructions (‘up’, ‘left’,…) • Captures a high-resolution image for 
 OCR processing
  33. 33. Guided OCR - results • Without system assistance or prior training, it can be extremely difficult to acquire readable images •Guidance is more efficient than opportunistic discovery • By using our guidance app, our participants learnt to take better pictures – even without assistance!
  34. 34. Thank you

×