Eyes-Free Barcode Scanning and Nutrition Information Extraction on Smartphones

Eyes-Free Barcode Scanning and Nutrition Information Extraction on Smartphones Vladimir Kulyukin Department of Computer Science Utah State University Logan, UT, USA University of Ljubljana, Ljubljana, Slovenia

CS Department, Utah State Universityhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Big Picture Independent Shopping for Visually Impaired (VI) and Blind Individualshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Background Statistics ● According to the WHO, there are 285 000 000 visually impaired people worldwide, of whom 39 000 000 are completely blind ● Independent shopping is a major quality of life issue for many individuals with visual impairments and a major functional access barrierhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Why Is Blind Shopping Difficult? A modern supermarket has a median area of 4 300 m2, stocks an average of 38 718 products, and services approximately 600 customers per hour (6 000 to 7 000 per day)http://www.youtube.com/csatlusu www.vkedco.blogspot.com

What Some VI Shoppers Want* ● Store Browsing & Navigation ● Product Search & Retrieval ● Shopping List Preparation *Source: Blind Shopping Experiments (2004 – present) and focus groups at USU Center for Persons with Disabilitieshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Motivational Quote “Shopping independently would be the next best thing to driving.” A blind individual at our focus group meeting at the USU Center for Persons with Disabilitieshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Research Principles ● The main objective of assistive technology is the design, development, and evaluation of accommodation systems for individuals with specific disabilities for specific environments ● Advances in computer vision, robotics, wearable and mobile computing are desirable by- products, but are not, in and of themselves, the end objectivehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

In Other Words ● Think Small ● Beware of Grand Visions ● Remember that Perfection Requires Infinite Budgetshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Presentation Outline ● Past R&D – Robot-Assisted & Wearable Blind Shopping ● Present R&D – Eyes-free Mobile Barcode Scanning – Nutrition Information Extraction ● Future R&D – Eyesight Sharing through Remote Caregiving – Leveraging Cloud & Mobile Computinghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Past R&D Robot-Assisted & Wearable Blind Shoppinghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

RoboCarthttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

RoboCart in Lees Market Place, Logan, UThttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

ShopTalkhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

ShopTalk in Lees Market Place, Logan, UThttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Current R&D Eyes-Free Barcode Scanning and Nutrition Information Extraction on Smartphones in ShopMobilehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Claim Computer Vision + Camera Alignment = Independent Eyes-Free Store Browsing on Smartphoneshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Hardware Evolution 2003-2005 2006-2008 2008-2010 2010 -Now RoboCart ShopTalk ShopMobilehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Two Parallel Threads of Computation Barcode Scanning Camera Alignmenthttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

ShopMobile at Fresh Market, Logan, UThttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Barcode Detection Detect the presence of a barcode in a framehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Image Regions on Product Surfaceshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

X and Y Gradients X Gradient Y Gradienthttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Barcode Detection with SVMshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Barcode Localization Determine the exact location of a barcode in a frame and segment ithttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Alternating Frequency & Vertical Similarity ● Alternating frequency is the number of 0-1 and 1-0 transitions in a one-pixel wide bit string ● Vertical continuity is the consecutive bit string similarityhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Barcode Scanning Lab Experiment Mean Times: 83.6 secs (Participant 1); 93.4 secs (Participant 2)http://www.youtube.com/csatlusu www.vkedco.blogspot.com

Nutrition Information Extraction Image Table Lines Table Line Segmentor Segmentor Open CV Text TTS OCR Engine Enginehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Horizontal and Vertical Line Filtering Vertical Line Filter Horizontal Line Filterhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Horizontal and Vertical Projections Vertical Projections Horizontal Projectionshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Horizontal and Vertical Bounds Vertical Bounds Horizontal Boundshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Future R&D Eyesight Sharing & Remote Caregiving through Cloud Computinghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Remote Caregiving ● Fully automated approaches tend to be error-prone ● Cloud computing infrastructures and real-time video streaming protocols make it possible for sighted individuals to share their sight with their VI friends remotely ● Hearing can also be shared remotely ● Amount of caregiving can be dynamically adjusted at run timehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

TeleShop Eyes-Free Shopping through Teleassistancehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

TeleShop Caregiver Blind Shopper Caregiver Wi-Fi/3G/4Ghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

TeleShop Architecture Blind Shopper Caregiverhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Leveraging Cloud & Mobile Computinghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Clienthttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Client Caregiverhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Amazon EC2 Client Elastic Computing Caregiver Servicehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Amazon EC2 Client Elastic Computing Caregiver Service Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Notification Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Notification Notification Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Notification Help Notification Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Help Notification Help Notification Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Help Client Notification Caregiver Client Help Notification Caregiver Client Caregiver Android C2DM Client Cloud 2 Device Caregiver Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

What about Crowdsourcing? ● Crowdsourcing may not be suitable for time sensitive tasks and tasks that require mutual trust ● Time sensitivity can be addressed through increased helper volume, which is unlikely to materialize for smaller disabled populations because the payoff is not great ● Trust may be a much harder issue for VI crowdsourcinghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Feedback Errors, questions, comments to vladimir.kulyukin@usu.eduhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

http://vkedco.blogspot.com	136
http://www.vkedco.blogspot.com	54
http://www.vkedco.blogspot.in	2
http://vkedco.blogspot.in	1
http://vkedco.blogspot.mx	1

Eyes-Free Barcode Scanning and Nutrition Information Extraction on Smartphones

by Vladimir Kulyukin on Jan 25, 2012

Accessibility

Categories

Tags

Upload Details

Usage Rights

5 Embeds 194

Statistics

Eyes-Free Barcode Scanning and Nutrition Information Extraction on Smartphones — Presentation Transcript