SlideShare a Scribd company logo

HandSight: The Design and Evaluation of a Finger-Mounted Camera and Feedback System to Enable Blind Persons to Read Printed Text

This talk was presented at the 2015 HCIL Annual Symposium put on by the University of Maryland's Human Computer Interaction Lab. We introduce the preliminary design of a novel vision-augmented touch system called HandSight intended to support activities of daily living (ADLs) by sensing and feeding back non-tactile information about the physical world as it is touched. Though we are interested in supporting a range of ADL applications, here we focus specifically on reading printed text. The recent miniaturization of cameras has enabled finger- based reading approaches, which provide blind and visually impaired readers with access to printed materials. Compared to handheld text scanners such as mobile phone applications, mounting a tiny camera on the user’s own finger has the potential to mitigate camera framing issues, enable a blind reader to better understand the spatial layout of a document, and provide better control over reading pace. A finger-based approach, however, also introduces the need to guide the reader in physically navigating a document, such as tracing along lines of text. While previous work has proposed audio and haptic directional finger guidance for this purpose, user studies of finger-based reading have been limited to 3 or 4 participants. To further investigate the feasibility of a finger-based sensing and feedback system for reading printed text, we conducted two lab studies: (i) a comparison of audio and haptic directional finger guidance with 20 blind participants using an iPad-based testbed, and (ii) a smaller proof-of- concept study with 4 blind participants using a preliminary wearable prototype called HandSight. Findings from the first study show similar performance between haptic and audio directional guidance, although audio may offer an accuracy advantage for line tracing. Subjective feedback also highlights tradeoffs between the two types of guidance, such as the interference of audio guidance with speech output and the potential for desensitization to haptic guidance. While several participants appreciated the direct access to layout information provided by finger-based exploration, important concerns also arose about ease of use and the amount of concentration required. We close with a discussion on the advantages of finger-based reading for blind readers, its feasibility, and potential design improvements to the HandSight prototype.

1 of 48
HandSight: The Design and Evaluation of a
Finger-Mounted Camera and Feedback System
to Enable Blind Persons to Read Printed Text
Lee Stearns1, Ruofei Du1, Uran Oh1, Catherine Jou1, Yumeng Wang2,
Leah Findlater3, Rama Chellappa4, David A. Ross5, Jon E. Froehlich1
University of Maryland: Computer Science1, Architecture2, Information Studies3, Electrical Engineering4,
Atlanta VA R&D Center of Excellence in Vision and Neurocognitive Rehabilitation (CVNR)5
There are 285 million people with visual impairments
worldwide—including 39 million who are blind
There are 285 million people with visual impairments
worldwide—including 39 million who are blind
Visual impairments can negatively impact a
person’s ability to perform activities of daily living
(ADLs)
Previous research has explored using mobile cameras
with computer vision for at-a-distance tasks
but they fail to support touch-based interactions
Our Approach: HandSight
a vision augmented touch system

Recommended

What Makes Great Infographics
What Makes Great InfographicsWhat Makes Great Infographics
What Makes Great InfographicsSlideShare
 
Masters of SlideShare
Masters of SlideShareMasters of SlideShare
Masters of SlideShareKapost
 
10 Ways to Win at SlideShare SEO & Presentation Optimization
10 Ways to Win at SlideShare SEO & Presentation Optimization10 Ways to Win at SlideShare SEO & Presentation Optimization
10 Ways to Win at SlideShare SEO & Presentation OptimizationOneupweb
 
STOP! VIEW THIS! 10-Step Checklist When Uploading to Slideshare
STOP! VIEW THIS! 10-Step Checklist When Uploading to SlideshareSTOP! VIEW THIS! 10-Step Checklist When Uploading to Slideshare
STOP! VIEW THIS! 10-Step Checklist When Uploading to SlideshareEmpowered Presentations
 
How To Get More From SlideShare - Super-Simple Tips For Content Marketing
How To Get More From SlideShare - Super-Simple Tips For Content MarketingHow To Get More From SlideShare - Super-Simple Tips For Content Marketing
How To Get More From SlideShare - Super-Simple Tips For Content MarketingContent Marketing Institute
 
2015 Upload Campaigns Calendar - SlideShare
2015 Upload Campaigns Calendar - SlideShare2015 Upload Campaigns Calendar - SlideShare
2015 Upload Campaigns Calendar - SlideShareSlideShare
 
What to Upload to SlideShare
What to Upload to SlideShareWhat to Upload to SlideShare
What to Upload to SlideShareSlideShare
 

More Related Content

Recently uploaded

green chemistry, clean sustainable environment.ppt
green chemistry, clean sustainable environment.pptgreen chemistry, clean sustainable environment.ppt
green chemistry, clean sustainable environment.pptRashmiSanghi1
 
Presentacion Mariana Arango- biología molecular
Presentacion Mariana Arango- biología molecularPresentacion Mariana Arango- biología molecular
Presentacion Mariana Arango- biología molecularmarianaarangop
 
The ExoGRAVITY project - observations of exoplanets from the ground with opti...
The ExoGRAVITY project - observations of exoplanets from the ground with opti...The ExoGRAVITY project - observations of exoplanets from the ground with opti...
The ExoGRAVITY project - observations of exoplanets from the ground with opti...Advanced-Concepts-Team
 
LIGHT Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
LIGHT  Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...LIGHT  Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
LIGHT Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...Abhinav S
 
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...AmalDhivaharS
 
Weak-lensing detection of intracluster filaments in the Coma cluster
Weak-lensing detection of intracluster filaments in the Coma clusterWeak-lensing detection of intracluster filaments in the Coma cluster
Weak-lensing detection of intracluster filaments in the Coma clusterSérgio Sacani
 
Seminario biología molecular Lina Charris
Seminario biología molecular Lina CharrisSeminario biología molecular Lina Charris
Seminario biología molecular Lina CharrisLinaMarcelaCharrisRa
 
discussion on the endocrine system for science grade10.pptx
discussion on the endocrine system for science grade10.pptxdiscussion on the endocrine system for science grade10.pptx
discussion on the endocrine system for science grade10.pptxShePerezDelaCruz
 
electrophoresis: types, advantages, disadvantages and applications.
electrophoresis: types, advantages, disadvantages and applications.electrophoresis: types, advantages, disadvantages and applications.
electrophoresis: types, advantages, disadvantages and applications.Silpa Selvaraj
 
Differences between syrups and elixirs .pptx
Differences between  syrups and elixirs .pptxDifferences between  syrups and elixirs .pptx
Differences between syrups and elixirs .pptxushakiranmai4
 
Hydro-Thermal Liquefaction Of Lignocellulosic biomass to produce Bio-Crude oil
Hydro-Thermal Liquefaction Of Lignocellulosic biomass to produce Bio-Crude oilHydro-Thermal Liquefaction Of Lignocellulosic biomass to produce Bio-Crude oil
Hydro-Thermal Liquefaction Of Lignocellulosic biomass to produce Bio-Crude oilZeeshan Nazir
 
Introduction to Chromatography (Column chromatography)
Introduction to Chromatography (Column chromatography)Introduction to Chromatography (Column chromatography)
Introduction to Chromatography (Column chromatography)Ahmed Metwaly
 
1.0 - The Light Miscroscope.ppt microscopy
1.0 - The Light Miscroscope.ppt microscopy1.0 - The Light Miscroscope.ppt microscopy
1.0 - The Light Miscroscope.ppt microscopystephenopokuasante
 
Automatic Stainer & Screener technique.pptx
Automatic Stainer & Screener technique.pptxAutomatic Stainer & Screener technique.pptx
Automatic Stainer & Screener technique.pptxSagarBhakare1
 
dkNET Webinar: The Collaborative Microbial Metabolite Center – Democratizing ...
dkNET Webinar: The Collaborative Microbial Metabolite Center – Democratizing ...dkNET Webinar: The Collaborative Microbial Metabolite Center – Democratizing ...
dkNET Webinar: The Collaborative Microbial Metabolite Center – Democratizing ...dkNET
 
Elbow joint - Anatomy of the Elbow joint
Elbow joint - Anatomy of the Elbow jointElbow joint - Anatomy of the Elbow joint
Elbow joint - Anatomy of the Elbow jointTELISHA2
 
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsOpen Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsPeter Coles
 
Salesforce Starter Package Presentation.
Salesforce Starter Package Presentation.Salesforce Starter Package Presentation.
Salesforce Starter Package Presentation.Naresh Gupta
 
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsOpen Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsPeter Coles
 

Recently uploaded (20)

green chemistry, clean sustainable environment.ppt
green chemistry, clean sustainable environment.pptgreen chemistry, clean sustainable environment.ppt
green chemistry, clean sustainable environment.ppt
 
Presentacion Mariana Arango- biología molecular
Presentacion Mariana Arango- biología molecularPresentacion Mariana Arango- biología molecular
Presentacion Mariana Arango- biología molecular
 
The ExoGRAVITY project - observations of exoplanets from the ground with opti...
The ExoGRAVITY project - observations of exoplanets from the ground with opti...The ExoGRAVITY project - observations of exoplanets from the ground with opti...
The ExoGRAVITY project - observations of exoplanets from the ground with opti...
 
LIGHT Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
LIGHT  Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...LIGHT  Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
LIGHT Community Medicine LIGHT IS A SOURCE OF ENERGY THERE ARE TWO TYPE OF S...
 
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...
Anti-Obesity Activity of Anthocyanins and Corresponding Introduction in Dieta...
 
Weak-lensing detection of intracluster filaments in the Coma cluster
Weak-lensing detection of intracluster filaments in the Coma clusterWeak-lensing detection of intracluster filaments in the Coma cluster
Weak-lensing detection of intracluster filaments in the Coma cluster
 
Seminario biología molecular Lina Charris
Seminario biología molecular Lina CharrisSeminario biología molecular Lina Charris
Seminario biología molecular Lina Charris
 
discussion on the endocrine system for science grade10.pptx
discussion on the endocrine system for science grade10.pptxdiscussion on the endocrine system for science grade10.pptx
discussion on the endocrine system for science grade10.pptx
 
electrophoresis: types, advantages, disadvantages and applications.
electrophoresis: types, advantages, disadvantages and applications.electrophoresis: types, advantages, disadvantages and applications.
electrophoresis: types, advantages, disadvantages and applications.
 
Differences between syrups and elixirs .pptx
Differences between  syrups and elixirs .pptxDifferences between  syrups and elixirs .pptx
Differences between syrups and elixirs .pptx
 
Hydro-Thermal Liquefaction Of Lignocellulosic biomass to produce Bio-Crude oil
Hydro-Thermal Liquefaction Of Lignocellulosic biomass to produce Bio-Crude oilHydro-Thermal Liquefaction Of Lignocellulosic biomass to produce Bio-Crude oil
Hydro-Thermal Liquefaction Of Lignocellulosic biomass to produce Bio-Crude oil
 
Introduction to Chromatography (Column chromatography)
Introduction to Chromatography (Column chromatography)Introduction to Chromatography (Column chromatography)
Introduction to Chromatography (Column chromatography)
 
REGULATION OF METABOLISM IN PLANTS AND THE DIFFERENT MECHANISMS
REGULATION OF METABOLISM IN PLANTS  AND THE DIFFERENT MECHANISMSREGULATION OF METABOLISM IN PLANTS  AND THE DIFFERENT MECHANISMS
REGULATION OF METABOLISM IN PLANTS AND THE DIFFERENT MECHANISMS
 
1.0 - The Light Miscroscope.ppt microscopy
1.0 - The Light Miscroscope.ppt microscopy1.0 - The Light Miscroscope.ppt microscopy
1.0 - The Light Miscroscope.ppt microscopy
 
Automatic Stainer & Screener technique.pptx
Automatic Stainer & Screener technique.pptxAutomatic Stainer & Screener technique.pptx
Automatic Stainer & Screener technique.pptx
 
dkNET Webinar: The Collaborative Microbial Metabolite Center – Democratizing ...
dkNET Webinar: The Collaborative Microbial Metabolite Center – Democratizing ...dkNET Webinar: The Collaborative Microbial Metabolite Center – Democratizing ...
dkNET Webinar: The Collaborative Microbial Metabolite Center – Democratizing ...
 
Elbow joint - Anatomy of the Elbow joint
Elbow joint - Anatomy of the Elbow jointElbow joint - Anatomy of the Elbow joint
Elbow joint - Anatomy of the Elbow joint
 
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsOpen Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
 
Salesforce Starter Package Presentation.
Salesforce Starter Package Presentation.Salesforce Starter Package Presentation.
Salesforce Starter Package Presentation.
 
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of AstrophysicsOpen Access Publishing in Astrophysics and the Open Journal of Astrophysics
Open Access Publishing in Astrophysics and the Open Journal of Astrophysics
 

Featured

AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfmarketingartwork
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024Neil Kimberley
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)contently
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024Albert Qian
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsKurio // The Social Media Age(ncy)
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summarySpeakerHub
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentLily Ray
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best PracticesVit Horky
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project managementMindGenius
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Applitools
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at WorkGetSmarter
 

Featured (20)

AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work
 
ChatGPT webinar slides
ChatGPT webinar slidesChatGPT webinar slides
ChatGPT webinar slides
 
More than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike RoutesMore than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike Routes
 

HandSight: The Design and Evaluation of a Finger-Mounted Camera and Feedback System to Enable Blind Persons to Read Printed Text

  • 1. HandSight: The Design and Evaluation of a Finger-Mounted Camera and Feedback System to Enable Blind Persons to Read Printed Text Lee Stearns1, Ruofei Du1, Uran Oh1, Catherine Jou1, Yumeng Wang2, Leah Findlater3, Rama Chellappa4, David A. Ross5, Jon E. Froehlich1 University of Maryland: Computer Science1, Architecture2, Information Studies3, Electrical Engineering4, Atlanta VA R&D Center of Excellence in Vision and Neurocognitive Rehabilitation (CVNR)5
  • 2. There are 285 million people with visual impairments worldwide—including 39 million who are blind
  • 3. There are 285 million people with visual impairments worldwide—including 39 million who are blind Visual impairments can negatively impact a person’s ability to perform activities of daily living (ADLs)
  • 4. Previous research has explored using mobile cameras with computer vision for at-a-distance tasks
  • 5. but they fail to support touch-based interactions
  • 6. Our Approach: HandSight a vision augmented touch system
  • 7. Our Approach: HandSight a vision augmented touch system Tiny CMOS cameras, micro-haptic actuators mounted on one or more fingers
  • 8. Our Approach: HandSight a vision augmented touch system Tiny CMOS cameras, micro-haptic actuators mounted on one or more fingers SmartWatch for power, processing, speech output
  • 9. Our Approach: HandSight a vision augmented touch system Tiny CMOS cameras, micro-haptic actuators mounted on one or more fingers SmartWatch for power, processing, speech output Computer vision, machine learning algorithms to support fingertip-based sensing
  • 10. Our Vision for HandSight Augment the sense of touch with a device that is unobtrusive, always available, and that allows seamless switching between the physical world and HandSight enabled applications.
  • 11. Interacting with the Physical World: Example Applications
  • 12. 1. Overview of Current Prototype 2. Study on Finger Guidance for Reading
  • 13. Finger-mounted camera Awaiba NanEye 2C CMOS camera 1x1mm2, 250x250 pixel 44 fps Self-illuminated with 4-LED ring
  • 14. Wrist-mounted controller Current Design: Arduino Pro Micro with Bluetooth 4.0 Powers and controls haptic feedback
  • 16. Output Finger-mounted vibration motors Speech / audio via onboard speakers or wireless headset
  • 17. Objectives for HandSight • Extensible platform • Develop algorithms for touch-based interactions • Investigate physical designs and feedback modalities
  • 18. Objectives for HandSight • Extensible platform • Develop algorithms for touch-based interactions • Investigate physical designs and feedback modalities • Proof-of-concept applications • Reading • Dressing • Technology Access
  • 19. 1. Overview of Current Prototype 2. Study on Finger Guidance for Reading
  • 22. Mobile Apps (KNFB Reader) Wearables (OrCam) Popular Reading Methods
  • 23. Touch-based Reading • Potential advantages • Immediate access, no alignment issues
  • 24. Touch-based Reading • Potential advantages • Immediate access, no alignment issues • Spatial awareness and interactions—may improve comprehension and document understanding
  • 25. Touch-based Reading • Potential advantages • Immediate access, no alignment issues • Spatial awareness and interactions—may improve comprehension and document understanding FingerReader
  • 26. Touch-based Reading • Problem: How can we guide a blind person’s finger? • Following a line of text • Locating the start of the next line • Exploring a document’s layout
  • 27. Continuous audio cues indicate content beneath the user’s finger System Design: Exploration Mode
  • 28. Continuous audio cues indicate content beneath the user’s finger System Design: Exploration Mode
  • 29. Audio or haptic directional guidance helps users to stay on the line or locate the start of new line. Audio cues indicate start/end of a line or paragraph. System Design: Reading Mode
  • 30. Audio or haptic directional guidance helps users to stay on the line or locate the start of new line. Audio cues indicate start/end of a line or paragraph. System Design: Reading Mode
  • 31. System Design: Algorithms We use Tesseract OCR with custom preprocessing and motion tracking algorithms
  • 32. Study Overview Study I: feasibility and directional guidance evaluation (20 participants) Study II: proof-of-concept (4 participants)
  • 33. Study I Method Simulated reading experience using an iPad
  • 34. Study I Method Simulated reading experience using an iPad 20 participants Average Age 48.3 (SD=11.7, Range=26-67) Gender 12 Male, 8 Female Vision Level 11 Totally Blind, 9 Light Sensitive
  • 35. Study I Method Simulated reading experience using an iPad 20 participants Within-subjects, 2 conditions: audio and haptic
  • 36. Study I Method Simulated reading experience using an iPad 20 participants Within-subjects, 2 conditions: audio and haptic Participants read 2 documents for each condition plain magazine
  • 37. Study I Method Simulated reading experience using an iPad 20 participants Within-subject, 2 conditions: audio and haptic Participants read 2 documents for each condition Analysis: reading speed and accuracy, comprehension, subjective feedback audio haptic
  • 38. Study I Findings audio haptic Audio vs. Haptic: Similar performance. Audio was significantly more accurate for line tracing on the magazine documents (p = 0.018)
  • 39. Study I Findings Audio vs. Haptic: Similar performance. Audio was significantly more accurate for line tracing on the magazine documents (p = 0.018) Preferences split (12 haptic, 7 audio, 1 equal preference)
  • 40. Study I Findings Audio vs. Haptic: Similar performance. Audio was significantly more accurate for line tracing on the magazine documents (p = 0.018) Preferences split (12 haptic, 7 audio, 1 equal preference) Participant comments: audio interferes with speech, desensitization to haptics over time
  • 41. Study I Findings Pros: Learning curve < braille Access to spatial information Overall Reading Experience:
  • 42. Study I Findings Pros: Cons: Learning curve < braille Hard to use Access to spatial information Requires concentration Overall Reading Experience:
  • 43. Study II: Proof of Concept 4 participants
  • 44. Future Work Study utility of spatial layout information
  • 45. Future Work Study utility of spatial layout information Reduce cognitive and physical effort
  • 46. Future Work Study utility of spatial layout information Reduce cognitive and physical effort Explore possibilities for camera placement
  • 48. Questions? Lee Stearns1, Ruofei Du1, Uran Oh1, Catherine Jou1, Yumeng Wang2, Leah Findlater3, Rama Chellappa4, David A. Ross5, Jon E. Froehlich1 University of Maryland: Computer Science1, Architecture2, Information Studies3, Electrical Engineering4, Atlanta VA R&D Center of Excellence in Vision and Neurocognitive Rehabilitation (CVNR)5 Thank you to our participants and the Maryland State Library for the Blind and Physically Handicapped. This research was funded by the Department of Defense. Images from Flickr used under Creative Commons license. Contact: lstearns@umd.edu | jonf@cs.umd.edu

Editor's Notes

  1. Hello everyone. My name is Lee Stearns, and today I’m going to be presenting my group’s research on a system we call HandSight, which we’ve designed to help people with visual impairments perform everyday tasks like reading printed text.
  2. According to a 2010 study, there are 285 million people living with visual impairments worldwide, including 39 million who are blind. In the United States and other countries with populations that are aging, that number is only increasing.
  3. Visual impairments can reduce a person’s ability to perform activities of daily living, like reading or getting dressed.
  4. There has been a lot of previous research that’s looked at using mobile cameras and computer vision to help people with visual impairments with tasks that are performed at a distance, like navigation or face recognition.
  5. But very few have tried to support the direct touch-based interactions that make up so many of the activities we perform every day. Without sight, touch is one of the primary means of acquiring information about the physical world. Many people with visual impairments have a highly attuned sense of touch.
  6. Our approach augments the sense of touch, sensing and feeding back non-tactile information about the physical world as it is touched.
  7. We envision a system called HandSight that’s made up of tiny cameras and haptics integrated directly with the user’s fingers.
  8. It will be built into a smart-watch platform for power, processing, and speech output,
  9. and use computer vision and machine learning algorithms to support fingertip-based sensing.
  10. By putting the camera and feedback directly on the user’s fingers and augmenting the sense of touch, our hope is that we can create new and more intuitive assistive applications. Our goal is to offer a device that is unobtrusive, always available, and that allows seamless switching between the physical world and the applications that HandSight supports.
  11. There are a wide range of applications for this technology. One of the first things that jumps to mind is to use it to read printed text in newspapers, bills, menus… not everything available digitally yet. You could also use it to explore maps and drawings, to get a better idea of their spatial layouts. Imagine moving your fingers over the page and being about to feel the texture of the lines, exploring the shapes and relative positions. Another application is getting dressed in the morning, quickly identifying colors and visual textures that you might not be able to distinguish by the feel alone. Even something as simple as shopping for groceries – imagine picking up a bell pepper in the store and being able to instantly identify by touch whether it’s red or green. All images licensed for use under Creative Commons USA image: Wikimedia Commons Clothing image: Flickr user perspicacious Bell pepper image: Flickr user Thomas Hawk
  12. I’ll give a brief overview of the hardware we’re using and the design of our current prototype, and then the rest of the talk will be devoted to describing a recent study we completed on finger guidance for reading.
  13. The cameras we’re using were originally designed to be used in minimally invasive surgeries, but their tiny size makes them perfect for our use as well. The images on the left show how small the sensors are, with and without a ring of tiny LEDs. We use the LED version in our prototypes to provide light for the camera. We’re still exploring different form factors and camera positions, whether the camera should be on top of the finger or on the side, whether it should be up close to the touch surface or further back. We’re also looking at whether it might be useful to have these cameras attached to more than one finger.
  14. Our goal is to power and control everything from a smart watch platform on the user’s wrist. Our early experiments with mobile phones have shown that they’re already powerful enough for the kinds of image processing tasks we’d need to do. Our current design uses a computer for most of the processing, but it also includes a wrist-mounted microcontroller that communicates with the computer over Bluetooth and powers and controls the haptic feedback on the user’s finger.
  15. We’re still experimenting with several different haptic actuators and other types of tactile feedback, which could be mounted on the fingers or the wrist. The motors we’re using now are larger than we’d like, but they’re cheap and reliable. We use them to augment the sense of touch by providing notifications or directional guidance.
  16. We also provide speech feedback and other audio cues through built-in speakers or over a lightweight wireless headset.
  17. Our objective is to create a platform that we can extend to support many different activities of daily life. We’re developing algorithms to support touch-based interactions using a finger-mounted camera. We’re also investigating different physical designs for the system and different types of haptic and audio feedback, with the goal of making the system intuitive and comfortable enough for everyday use.
  18. We plan to demonstrate our system’s potential through three proof-of-concept applications. Reading and exploring printed documents. Identifying and conveying non-tactile information to help users coordinate clothing when they’re getting dressed. And enabling more intuitive access to cell phones and other technology through interactions on the user’s own body.
  19. Next I’m going to talk about a recent study that we completed, where we were looking at the touch-based reading experience in general, and more specifically at two different types of directional finger guidance.
  20. Since we’re trying to create a touch-based reading experience, we took a close look at braille. It has a high learning curve, but the immediate access to information and the spatial interactions through touch are very similar to the experience we want to provide.
  21. We also looked at existing technology for reading printed documents. Desktop scanners and screen reader software work, but they’re not very portable.
  22. Cell phone apps and wearable cameras can scan a document and read it back to you, but if you’re totally blind they can be difficult to line up with the document and know whether you’ve captured all of the text. None of them use the spatial layout information or allow for easy exploration.
  23. Devices that help people with visual impairments read text are fairly common, but touch-based reading offers some potential advantages over the more traditional approaches. Reading by touch enables immediate access—you place your finger on the page, and you can immediately start getting feedback and exploring the document. You don’t need to worry about lining up the document with the camera or waiting for it to be scanned and processed.
  24. It may also provide a more intuitive way for users to explore a document’s spatial layout, which we hope could improve comprehension and document understanding, especially for documents with complex layouts like newspapers or menus.
  25. There’s a project from MIT called FingerReader, which also uses a finger mounted camera and haptic feedback to help users read printed text. Our goals are the same, but we provide some different features and evaluated the directional guidance much more extensively.
  26. Directional finger guidance is also a basic problem for any touch-based application that we might support. With reading, guidance especially important, because unless you keep the camera centered over a line of text the system won’t be able to see what to read. We’ve explored a few different types of haptic and audio cues to help users stay centered over the line, and to locate the start of the next line. We’ve also developed cues to help users to explore a document’s layout, identifying the content that’s underneath their finger.
  27. Our system has 2 modes: exploration mode and reading mode. In the exploration mode, the user can move around the page however they’d like, and the system will provide continuous audio feedback to identify the content that’s underneath their finger. We chose distinctive high and low pitched sounds to represent text or pictures, and silence to represent white space.
  28. [play video]
  29. In the reading mode, the user moves sequentially along each line and the system reads each word out loud as their finger passes over it. Users can use their right index finger to read and their left to serve as an anchor to help them find the start of the next line. We provide directional guidance to help the user move vertically to stay on a line or to locate the start of a new line.
  30. [play video] Continuous audio cues indicate when the user’s finger drifts off of the line, and the frequency changes depending the direction and distance to the line. The haptic directional guidance we provide is very similar, but it uses the intensity and location of the vibrations in place of the audio frequency. We also play distinctive audio cues to identify the start and end of each line or paragraph.
  31. We use the Tesseract OCR library to recognize text in the video frames from the camera, along with some custom preprocessing algorithms to clean up the images and track motion between frames.
  32. We recently completed 2 studies on touch-based reading. The first was focused on the feasibility of our approach and looked closely at the user experience. We also performed a detailed comparison of two types of directional finger guidance. Our second study was a proof-of-concept evaluation of our prototype, with a much smaller set of participants.
  33. For our first study we were focused just on the user experience and directional guidance, so we used an iPad to simulate the reading experience. That also allowed us to collected precise finger trace data to use in our analysis.
  34. We recruited 20 participants, all of whom were totally blind or with only minimal light perception.
  35. Our study used a within-subjects design with 2 directional finger guidance conditions: audio and haptic.
  36. We asked each participant to explore and read 2 documents for each condition, a plain text document and a magazine-style document with a more complicated layout.
  37. Throughout the study sessions we collected the participants’ reading speed and accuracy, their comprehension, and subjective feedback.
  38. The speed and accuracy of the two types of guidance were very similar, but audio may offer some advantages to accuracy since it was significantly more accurate than haptics while reading the magazine documents.
  39. The participants’ preferences were also split, although a majority of them preferred haptic guidance.
  40. From our participants’ comments, both conditions may have some limitations; audio may interfere more with the speech feedback, and the haptics may lead to desensitization over time.
  41. Many participants said that they appreciated the lower learning curve compared to braille, and some liked the direct access to spatial information.
  42. But there were also some concerns about the ease of use and the amount of concentration that was required to complete the reading tasks.
  43. We conducted a follow-up study with 4 participants randomly selected to return and test out a prototype version of our system. They also tried out the KNFB Reader iPhone app, which is the current state of the art. Unfortunately, I don’t have time to go into any detail about the study, but our results showed that both approaches have some strengths and weaknesses. We’ll need to do further research to determine how to best support reading using our system.
  44. These studies left a lot of room for future work, with several new questions to be answered. We’ll need to do a more in depth study to see how useful the spatial layout information might be in everyday use.
  45. We’ll also need to iterate on our design to reduce the cognitive and physical effort required. It might be that finger-based reading isn’t practical when reading longer passages, but maybe we can optimize it for quick access to smaller amounts of text.
  46. We’ve also considered the idea that a hybrid system may be a good compromise. It could use a body mounted camera to determine the context and read back longer passages, but then use the finger-mounted camera for immediate access and exploration.
  47. Finally, reading is only one of the applications we hope to support. We’ll continue to work towards a wearable device that can augment the sense of touch and help support people with visually impairments throughout a variety of activities of daily living.
  48. Before I end my presentation, I’d like to take a moment to acknowledge everyone who has contributed to this project. We have a large and diverse research team, and it’s been a great experience working with all of them. Thank you.