Toward Real Time Eyes-Free Barcode Scanning on Smartphones in Video Mode


Published on

Our experiments indicate that smartphones
can be used for real time eyes-free barcode
scanning in video mode. Barcode localization
was observed to improve the decoding rates by
as much as 20%. The TLB-AF can be eliminated
without a significant reduction in barcode
decoding rates. Two VI participants were able
to successfully scan most UPC barcodes on
various grocery products using our algorithm.

Published in: Science, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Toward Real Time Eyes-Free Barcode Scanning on Smartphones in Video Mode

  1. 1. TOWARD REAL TIME EYES-FREE BARCODE SCANNING ON SMARTPHONES IN VIDEO MODE Aliasgar Kutiyanawala1 Vladimir Kulyukin1 John Nicholson2 1 Department of Computer Science, Utah State University. 2Department of Computer Science and Information Technology, Austin Peay State University INTRODUCTION taken continuously and processed by the barcode localization and decoding modules. If a In our previous research [1], we have barcode is decoded, the user is notified throughshown that visually impaired (VI) individuals text-to-speech.can shop independently by scanning MSI Interactive Camera Alignment Loopbarcodes on shelves and UPC barcodes onproducts. This system was called ShopTalk and We have previously conducted a lab studyused a hand-held barcode scanner, a shoulder- with one VI participant and four blindfoldedmounted keypad and headphones connected to sighted participants, where participants had toan ultra-portable OQO computer. ShopMobile-II find, retrieve and verify products from a[2, 3] is the next version of this system that simulated shopping aisle [2]. The participantsreduces the systems hardware complexity by scanned MSI barcodes on shelves to findallowing VI users to shop independently using product locations and UPC barcodes ononly a smartphone. products to identify them. MSI barcodes are a type of linear 1-D barcodes that are used ShopMobile-II has three software modules – mostly for inventory control, and markingan eyes-free barcode scanner, an OCR engine storage containers and shelves in warehouseand remote guidance. This paper describes our environments [7]. To scan a barcode,barcode scanning algorithm that operates in participants would first align the camera withvideo mode on the Google Nexus One Android the product or shelf and then slowly move the2.2 smartphone. Two pilot experiments are camera away. The VI participant wouldpresented: the first one, conducted in a grocery frequently misalign the camera in the pitch andstore evaluated the contributions of the yaw planes as he moved it away fromalgorithm’s modules; the second one, barcodes. This misalignment caused skewperformed in a laboratory with two VI users, distortions in barcode images, which resulted inevaluated the effectiveness of the algorithm in several decoding failures.finding UPC barcodes on various groceryproducts. The interactive camera alignment module is designed to minimize skew distortions by EYES-FREE BARCODE SCANNER keeping the camera aligned with the barcode in the pitch and yaw frames. The shopper starts RedLaser [4] and ZXing [5] are two the loop by pressing the touch screen. Theapplications for scanning barcodes on system captures the phones orientationsmartphones. However, they require users to sensors readings of the pitch and yaw planescarefully align the camera with the barcodes for subsequent reference. The system takesprior to decoding and cannot decode MSI these readings as the shopper moves thebarcodes. Our algorithm is designed to find camera away from the barcode and comparesboth UPC and MSI barcodes in real video mode them with the reference readings. When thewithout prior camera alignment [6]. readings deviate from the reference readings, the phone begins to vibrate until the camera is The eyes-free barcode scanning algorithm is realigned.comprised of three modules – interactivecamera alignment loop, barcode localization,and barcode decoding. The barcode scanningalgorithm operates in video mode. Images are
  2. 2. Barcode Localization – luminance based method (LM), two-level binary method (TLB) and a two-level binary The area of the barcode region in the image method with alternating frequency filter (TLB-varies with the distance between the barcode AF).and the camera. If the camera is held close tothe barcode, the barcode region occupies a LM populates the scanline with the centrallarge area in the image and the barcode (n/2th) row of the strip, and each pixel in thedecoding stage can easily decode the barcode. scanline pi ∈ [0, 255]. TLB binarizes the stripHowever, if the camera is held at a distance to using a modified Niblack filter [8] beforethe barcode, the barcode region is small and populating the scanline with the central row ofthere is a possibility of other components, such the strip. Each pixel in the scanline pi ∈ {0,text and graphics, being alongside the barcode, 255}. TLB-AF takes this process a step furtherwhich may cause a detection failure. Localizing by filtering out areas with low alternatingand segmenting possible barcode regions prior frequencies in the binarized strip beforeto decoding increases the probability of populating the scanline.accurate decoding. The processed scanline is now converted to A barcode is a small homogeneous region a line-widths representation W = {w1, w2 …consisting of alternate black and white lines. wk}). To obtain W, strings of consecutive zerosWe define two properties – alternating and ones in the scanline are replaced with theirfrequency and vertical continuity that run lengths. A separate data structure C = {c1,characterize a region as a potential barcode. If c2, …, ck} records the color associated witha line is drawn horizontally across the barcode, each line-width wi. For example, if the scanlinealternating frequency is the number of black to S = {255, 0, 0, 0, 255, 255}, W = {1, 3, 2}white and white to black transitions along that and C= {1, 0, 1}.line. Vertical continuity is the continuity of black Table 1: UPC barcode symbology in line-and white lines along vertical lines. Vertical widths representation.continuity is estimated by the similaritybetween two parallel horizontal lines, separated Pattern / Digit Template of line widths Tmvertically, placed over the region. The longest Start Pattern 111common subsequence is used as the similarity Middle Pattern 11111measure. End Pattern 111 The first step in the barcode localization is Digit 0 3211to scale the image down to the 320 x 240resolution for efficiency. This scaled image is Digit 1 2221passed through a line filter presented in [2], Digit 2 2122which allows vertical lines to pass through and Digit 3 1411filters out everything else. The filtered image is Digit 4 1132scanned in a rasterized pattern to look forareas with high alternating frequencies and Digit 5 1231vertical continuities. These are translated into Digit 6 1114the corresponding areas of the original image Digit 7 1312and segmented from it. Digit 8 1213Barcode Decoding Digit 9 3112 Our barcode decoding algorithm usesscanlines and consists of three main procedures A UPC barcode consists of two sets of six– strip generation, strip processing, and digits D enclosed by start (S)/end (E) patternsscanline decoding. The strip generation and separated by a middle pattern (M) asprocedure selects horizontal and diagonal n- SDDDDDDMDDDDDDE [9]. The start, middlepixel wide strips in the image. The strips are and end patterns as well as the digits areconverted to one-pixel wide scanlines. This encoded by a series of alternating black andconversion is achieved through three methods white lines of varying widths. Table 1 shows the
  3. 3. UPC barcode symbology in the line-widths shown in Fig. 2, we found that the LM decodedrepresentation. To decode the barcode in the the most barcodes (53), followed by TLB (26)scanline W, we find the start pattern index (s) and finally TLB-AF (15). Only one barcode outand the end pattern index (e) within W using of the 15 decoded by TLB-AF is not decoded bythe following equations: the other methods. Hence, this method can be eliminated without a significant impact on the s = arg max( wi −1 − std ( wi , wi +1 , wi + 2 )) overall decoding rate. e = arg max( wi + 3 − std ( wi , wi +1 , wi + 2 ))where, ci = 0. Each digit d is encoded by four lines and theindex i of the jth digit djj within W can be foundas: i = s + ( j − 1) × 4, if j ≤ 6 i = s + ( j − 1) × 4 + 5, if j > 6 If Tm represents the template of the digit m Figure 1. Contributions of the barcodein Table 1., the value of the jth digit dj can be localization stage.found as follows: 3 d j = arg min( ∑ (w k =0 i+k − Tkm ) 2 )where, 0 ≤ j ≤ 12 and 0 ≤ m ≤ 9. The twelfth digit is a checksum digit toverify accurate decoding. EXPERIMENTS The objective of the first experiment was tofind the contribution of barcode localization aswell as the contributions of the three stripprocessing methods. Our claim is that barcode Figure 2. Contributions of the different striplocalization prior to decoding increases the processing methods.decoding rate. To test this claim, a database of68 product images, which included boxes, cans, To test the accuracy of barcode recognitionand bottles, was obtained in a grocery store in video mode, we conducted two single subjectwith the Google Nexus One smartphone with experiments at the Smith Kettlewell EyeAndroid 2.2. We then logged the number of Research Institute in San Francisco, CA. Thebarcodes decoded in the scans with and without software was implemented on the Googlethe localization stage. As Fig. 1 shows, Nexus One smartphone with Android 2.2. Bothlocalization prior to decoding resulted in an subjects were completely blind staff electricalincrease of 14 (20.59%) decoded barcodes. engineers. One had a cellphone; the other never owned or used one. The camera As mentioned earlier, the three methods are alignment module was not tested. The cameraused to convert n-pixel wide strip to one-pixel operated in video mode.wide scanlines. We wanted to test thecontributions of each method to decoding. As
  4. 4. The experiment consisted of a tutorial and In informal feedback collected from bothan actual test. In the tutorial, subjects after the experiments, they indicated aeach subject was given two sample products (a preference for run time image alignmentplastic bottle and a juice box) and was shown signals when the camera is misaligned with thehow to move the camera along the surface of object. They also said that the camera shouldthe product. The tutorial continued until each work over a wider range of positions and thesubject could detect the barcodes on both software must operate much faster.products without the experimenters help. Thefirst subjects tutorial took approximately ten CONCLUSIONminutes. The second subjects tutorial tookapproximately eight minutes. During the test, Our experiments indicate that smartphoneseach subject was given ten products: a cereal can be used for real time eyes-free barcodebox, a small tea box, two small juice bottles, a scanning in video mode. Barcode localizationsmall milk carton, a Pringles tube, a toothpaste was observed to improve the decoding rates bybox, a larger juice bottle, a small water bottle, as much as 20%. The TLB-AF can be eliminatedand a yogurt cup. The detection time for each without a significant reduction in barcodeproduct was recorded on the phone. When the decoding rates. Two VI participants were ablesubject could not detect the barcode for over to successfully scan most UPC barcodes onfive minutes, the detection was considered a various grocery products using our algorithm.failure and the subject was given the nextproduct. The order of the products was REFERENCESrandomized for each subject. [1] J. Nicholson, V. Kulyukin, and D. Coster, “ShopTalk: independent blind shopping through verbal route directions Table 2: Results of barcode scanning and barcode scans”, The Open Rehabilitation Journal, ISSN:experiments with VI participants. 1874-9437 Volume 2, 2009, pp. 11-23, DOI Participant 1 Participant 2 10.2174/1874943700902010011.Product [2] V. Kulyukin, and A. Kutiyanawala, “Eyes-free barcode Time False Time False Taken Positives Taken Positives localization and decoding for visually impaired mobile phone users”, Proceedings of the 2010 InternationalCereal box 33 0 40 0 Conference on Image Processing, Computer Vision, andTea box 140 0 100 1 Pattern Recognition, 2010. [3] V. Kulyukin, and A. Kutiyanawala, “From ShopTalk toSmall juice 15 0 Fail 0 ShopMobile: vision-based barcode scanning with mobilebottle 1 phones for independent blind grocery shopping”,Small juice Proceedings of the 33-rd Annual Conference of the 142 0 151 0bottle 2 Rehabilitation Engineering and Assistive Technology SocietyMilk carton 56 0 51 1 of North America, June 2010, Las Vegas, Nevada.Pringles 37 0 111 0 [4] Occipital, LLC. RedLaser [5] The Zebra Crossing Barcode Decoding Library.Toothpaste 184 0 Fail 0 juice [6] Eyes-Free Barcode Scanning in Video Mode on Google 125 0 140 0bottle Nexus One Smartphone.Water bottle 87 0 121 0 [8] Wikipedia, MSI barcode.Yogurt cup 17 0 41 0 [8] W. Niblack, “An introduction to image processing”, The average barcode recognition times were Prentice-Hall, Englewood Cliff, NJ, pp. 115-1 16, 1986.83.6 seconds and 93.4 seconds for subject 1 [9] Wikipedia, UPC barcode.and 2, respectively. Subject 2 had two failures Product Code.where the software failed to detect the barcodein five minutes. There were also two falsepositives when the software detected a barcodeover printed text.