Your SlideShare is downloading. ×
Eyes-Free Barcode Scanning and Nutrition Information Extraction on Smartphones
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Eyes-Free Barcode Scanning and Nutrition Information Extraction on Smartphones

877
views

Published on

Published in: Technology, Health & Medicine

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
877
On Slideshare
0
From Embeds
0
Number of Embeds
18
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Eyes-Free Barcode Scanning and Nutrition Information Extraction on Smartphones Vladimir Kulyukin Department of Computer Science Utah State University Logan, UT, USA University of Ljubljana, Ljubljana, Slovenia
  • 2. CS Department, Utah State Universityhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 3. Big Picture Independent Shopping for Visually Impaired (VI) and Blind Individualshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 4. Background Statistics ● According to the WHO, there are 285 000 000 visually impaired people worldwide, of whom 39 000 000 are completely blind ● Independent shopping is a major quality of life issue for many individuals with visual impairments and a major functional access barrierhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 5. Why Is Blind Shopping Difficult? A modern supermarket has a median area of 4 300 m2, stocks an average of 38 718 products, and services approximately 600 customers per hour (6 000 to 7 000 per day)http://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 6. What Some VI Shoppers Want* ● Store Browsing & Navigation ● Product Search & Retrieval ● Shopping List Preparation *Source: Blind Shopping Experiments (2004 – present) and focus groups at USU Center for Persons with Disabilitieshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 7. Motivational Quote “Shopping independently would be the next best thing to driving.” A blind individual at our focus group meeting at the USU Center for Persons with Disabilitieshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 8. Research Principles ● The main objective of assistive technology is the design, development, and evaluation of accommodation systems for individuals with specific disabilities for specific environments ● Advances in computer vision, robotics, wearable and mobile computing are desirable by- products, but are not, in and of themselves, the end objectivehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 9. In Other Words ● Think Small ● Beware of Grand Visions ● Remember that Perfection Requires Infinite Budgetshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 10. Presentation Outline ● Past R&D – Robot-Assisted & Wearable Blind Shopping ● Present R&D – Eyes-free Mobile Barcode Scanning – Nutrition Information Extraction ● Future R&D – Eyesight Sharing through Remote Caregiving – Leveraging Cloud & Mobile Computinghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 11. Past R&D Robot-Assisted & Wearable Blind Shoppinghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 12. RoboCarthttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 13. RoboCart in Lees Market Place, Logan, UThttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 14. ShopTalkhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 15. ShopTalk in Lees Market Place, Logan, UThttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 16. Current R&D Eyes-Free Barcode Scanning and Nutrition Information Extraction on Smartphones in ShopMobilehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 17. Claim Computer Vision + Camera Alignment = Independent Eyes-Free Store Browsing on Smartphoneshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 18. Hardware Evolution 2003-2005 2006-2008 2008-2010 2010 -Now RoboCart ShopTalk ShopMobilehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 19. Two Parallel Threads of Computation Barcode Scanning Camera Alignmenthttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 20. ShopMobile at Fresh Market, Logan, UThttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 21. Barcode Detection Detect the presence of a barcode in a framehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 22. Image Regions on Product Surfaceshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 23. X and Y Gradients X Gradient Y Gradienthttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 24. Barcode Detection with SVMshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 25. Barcode Localization Determine the exact location of a barcode in a frame and segment ithttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 26. Alternating Frequency & Vertical Similarity ● Alternating frequency is the number of 0-1 and 1-0 transitions in a one-pixel wide bit string ● Vertical continuity is the consecutive bit string similarityhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 27. Barcode Scanning Lab Experiment Mean Times: 83.6 secs (Participant 1); 93.4 secs (Participant 2)http://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 28. Nutrition Information Extraction Image Table Lines Table Line Segmentor Segmentor Open CV Text TTS OCR Engine Enginehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 29. Horizontal and Vertical Line Filtering Vertical Line Filter Horizontal Line Filterhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 30. Horizontal and Vertical Projections Vertical Projections Horizontal Projectionshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 31. Horizontal and Vertical Bounds Vertical Bounds Horizontal Boundshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 32. Future R&D Eyesight Sharing & Remote Caregiving through Cloud Computinghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 33. Remote Caregiving ● Fully automated approaches tend to be error-prone ● Cloud computing infrastructures and real-time video streaming protocols make it possible for sighted individuals to share their sight with their VI friends remotely ● Hearing can also be shared remotely ● Amount of caregiving can be dynamically adjusted at run timehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 34. TeleShop Eyes-Free Shopping through Teleassistancehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 35. TeleShop Caregiver Blind Shopper Caregiver Wi-Fi/3G/4Ghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 36. TeleShop Architecture Blind Shopper Caregiverhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 37. Leveraging Cloud & Mobile Computinghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 38. A Cloud-Based Caregiving Architecture Clienthttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 39. A Cloud-Based Caregiving Architecture Client Caregiverhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 40. A Cloud-Based Caregiving Architecture Amazon EC2 Client Elastic Computing Caregiver Servicehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 41. A Cloud-Based Caregiving Architecture Amazon EC2 Client Elastic Computing Caregiver Service Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 42. A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 43. A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Notification Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 44. A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Notification Notification Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 45. A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Notification Help Notification Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 46. A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Help Notification Help Notification Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 47. A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Help Client Notification Caregiver Client Help Notification Caregiver Client Caregiver Android C2DM Client Cloud 2 Device Caregiver Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 48. What about Crowdsourcing? ● Crowdsourcing may not be suitable for time sensitive tasks and tasks that require mutual trust ● Time sensitivity can be addressed through increased helper volume, which is unlikely to materialize for smaller disabled populations because the payoff is not great ● Trust may be a much harder issue for VI crowdsourcinghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com
  • 49. Feedback Errors, questions, comments to vladimir.kulyukin@usu.eduhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com