Eyes-Free Barcode Scanning and Nutrtition Information Extraction on Smartphones

Eyes-Free Barcode Scanning and Nutrition Information Extraction on Smartphones Vladimir Kulyukin Department of Computer Science Utah State University Logan, UT, USA Masaryk University, Brno, Moravia, Czech Republic

CS Department, Utah State Universityhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Big Picture Independent Shopping for Visually Impaired (VI) and Blind Individualshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Background Statistics ● According to the WHO, there are 285 000 000 visually impaired people worldwide, of whom 39 000 000 are completely blind ● According to the Teiresias Centre, there are 60 000 visually impaired individuals in Czech Republic as of 2004, of whom 3 000 to 4 500 are completely blind ● Independent shopping is a major quality of life issue for many individuals with visual impairments and a major functional access barrierhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Why Is Blind Shopping Difficult? A modern supermarket has a median area of 4 300 m2, stocks an average of 38 718 products, and services approximately 600 customers per hour (6 000 to 7 000 per day)http://www.youtube.com/csatlusu www.vkedco.blogspot.com

What Some VI Shoppers Want* ● Store Browsing & Navigation ● Product Search & Retrieval ● Shopping List Preparation *Source: Blind Shopping Experiments (2004 – present) and focus groups at USU Center for Persons with Disabilitieshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Motivational Quote “Shopping independently would be the next best thing to driving.” A blind individual at our focus group meeting at the USU Center for Persons with Disabilitieshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Research Principles ● The main objective of assistive technology is the design, development, and evaluation of accommodation systems for individuals with specific disabilities for specific environments ● Advances in computer vision, robotics, wearable and mobile computing are desirable by- products, but are not, in and of themselves, the end objectivehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

In Other Words ● Think Small ● Beware of Grand Visions ● Remember that Perfection Requires Infinite Budgetshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Presentation Outline ● Past R&D – Robot-Assisted & Wearable Blind Shopping ● Present R&D – Eyes-free Mobile Barcode Scanning – Nutrition Information Extraction ● Future R&D – Eyesight Sharing through Remote Caregiving – Leveraging Cloud & Mobile Computinghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Past R&D Robot-Assisted & Wearable Blind Shoppinghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

RoboCarthttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

RoboCart in Lees Market Place, Logan, UThttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

ShopTalkhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

ShopTalk in Lees Market Place, Logan, UThttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Current R&D Eyes-Free Barcode Scanning and Nutrition Information Extraction on Smartphones in ShopMobilehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Claim Computer Vision + Camera Alignment = Independent Eyes-Free Store Browsing on Smartphoneshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Hardware Evolution 2003-2005 2006-2008 2008-2010 2010 -Now RoboCart ShopTalk ShopMobilehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Two Parallel Threads of Computation Barcode Scanning Camera Alignmenthttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

ShopMobile at Fresh Market, Logan, UThttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Barcode Detection Detect the presence of a barcode in a framehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Image Regions on Product Surfaceshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

X and Y Gradients X Gradient Y Gradienthttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Barcode Detection with SVMshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Barcode Localization Determine the exact location of a barcode in a frame and segment ithttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Alternating Frequency & Vertical Similarity ● Alternating frequency is the number of 0-1 and 1-0 transitions in a one-pixel wide bit string ● Vertical continuity is the consecutive bit string similarityhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Barcode Scanning Lab Experiment Mean Times: 83.6 secs (Participant 1); 93.4 secs (Participant 2)http://www.youtube.com/csatlusu www.vkedco.blogspot.com

Nutrition Information Extraction Image Table Lines Table Line Segmentor Segmentor Open CV Text TTS OCR Engine Enginehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Horizontal and Vertical Line Filtering Vertical Line Filter Horizontal Line Filterhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Horizontal and Vertical Projections Vertical Projections Horizontal Projectionshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Horizontal and Vertical Bounds Vertical Bounds Horizontal Boundshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Future R&D Eyesight Sharing & Remote Caregiving through Cloud Computinghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Remote Caregiving ● Fully automated approaches tend to be error-prone ● Cloud computing infrastructures and real-time video streaming protocols make it possible for sighted individuals to share their sight with their VI friends remotely ● Hearing can also be shared remotely ● Amount of caregiving can be dynamically adjusted at run timehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

TeleShop Eyes-Free Shopping through Teleassistancehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

TeleShop Caregiver Blind Shopper Caregiver Wi-Fi/3G/4Ghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

TeleShop Architecture Blind Shopper Caregiverhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Leveraging Cloud & Mobile Computinghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Clienthttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Client Caregiverhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Amazon EC2 Client Elastic Computing Caregiver Servicehttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Amazon EC2 Client Elastic Computing Caregiver Service Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Notification Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Notification Notification Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Notification Help Notification Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Help Notification Help Notification Android C2DM Cloud 2 Device Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

A Cloud-Based Caregiving Architecture Request Amazon EC2 Client Elastic Computing Caregiver Service Help Client Notification Caregiver Client Help Notification Caregiver Client Caregiver Android C2DM Client Cloud 2 Device Caregiver Messaginghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

What about Crowdsourcing? ● Crowdsourcing may not be suitable for time sensitive tasks and tasks that require mutual trust ● Time sensitivity can be addressed through increased helper volume, which is unlikely to materialize for smaller disabled populations because the payoff is not great ● Trust may be a much harder issue for VI crowdsourcinghttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Research Venues ● Consumption of cloud services on mobile devices ● Software design patterns for cloud computing ● Identification of problems that can be solved or executed via caregiving vs. crowdsourcing ● Identification of business processes that can benefit from caregiving vs. crowdsourcing ● Engagement of social networks in caregiving serviceshttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Promising Design Patterns for Cloud Computing ● Structural Patterns – Bridge – decoupling abstractions from implementations – Decorator – adding dynamic responsibilities to objects – Flyweight – supporting multiple objects via sharing ● Behavioral Patterns – Chain of Responsibility – decoupling request senders from request receivers – Mediator – encapsulating interactions with object sets – Observer – defining object dependencies for object state notificationhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

Feedback Errors, questions, comments to vladimir.kulyukin@usu.eduhttp://www.youtube.com/csatlusu www.vkedco.blogspot.com

http://vkedco.blogspot.com	142
http://www.vkedco.blogspot.com	76
http://vkedco.blogspot.in	5
http://vkedco.blogspot.mx	1

Eyes-Free Barcode Scanning and Nutrtition Information Extraction on Smartphones

by Vladimir Kulyukin on Jan 12, 2012

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 224

Statistics

Eyes-Free Barcode Scanning and Nutrtition Information Extraction on Smartphones — Presentation Transcript