The ability to validate credit cards using a mobile device is fruitful for many e-commerce platforms including Uber. Not only does this provide a seamless experience to users, but it also enables the company to verify that a user has physical possession of the credit card. In this talk we will discuss our new application that uses object detection neural networks to scan credit cards. Traditionally, machine learning models are hosted server side, but with challenges including high bandwidth inputs, low network speeds, and a greater focus on user privacy, hosting these models server side is not always feasible. Recent advancements bring up the possibility of deploying these models directly on the mobile devices. We will discuss the challenges we faced in designing the vision model to run on a mobile device. These include reducing the model’s size footprint and optimizing the model to run on various different types of mobile hardware. Speakers Richard Ash, Mobile Software Engineer, Uber Lenny Evans, Data Scientists, Uber

1. Lenny Evans, Data Scientist
Richard Ash, Software Engineer
Karthik Ramasamy, Senior Software Engineer
uScan

2. Uber in Numbers
4 Billion Trips in 2017
15 Million Uber trips per day
75 Million monthly active riders
600+ cities across 78 countries

3. Payment Fraud Account
Takeovers
GPS Spoofing

4. ● Give users better experience when adding credit cards
○ Potential for easier signup experience
● Prevent payment fraud
○ Fraudsters rarely have physical card
○ Ask users to produce a card with correct number
○ Soft challenge is a better FP experience
Scanning Credit Cards

6. Demo

7. Why is this hard?
● Card numbers in different
places
● Many numbers (e.g. phone
numbers) on card
● Cards use different (even non-
monospace!) fonts
● Want robustness to rotations
and tilt for better UX

8. Design Choices: Mobile vs Server Side Inferencing
Mobile inferencing
+ Better UX
+ Better privacy
+ Less use of mobile bandwidth
- Less compute resources
- Less control over model deployment/
deployment frameworks still in infancy
- Increase in app size

9. Why deep learning?
● Hand-engineered features not robust enough for our needs
● Recent advancements allow concurrent detection and classification
● Easy to include detection of other features (like logos!)

10. 5424 1811 1111 1458
Object
detection
model
Post
Processing

11. How does uScan SSD model detect and classify text?
(Actual input boxes are more
dense)
Model adjusts input boxes to
detect numbers and classify
Conv.
Layer
s

12. Mobilenet
1x1 Convolution
BB offsets, BB class confidences
Input Image
Non-Maximum Suppression
Feature
Extractor
BB Predictor
Dedup
e
Simplified Single Shot Detection (SSD) Model
Architecture is similar to typical
convnets, except model
outputs many outputs
Only need one feature
extractor as text is all similar in
size

13. Mobilenet
1x1 Convolution
BB offsets, BB class confidences
Input Image
Non-Maximum Suppression
Why SSD?
SSD is fully convolutional
Mobilenet allows for efficient
inference

14. Training Data
● Collecting cards, taking pictures, and labeling is an expensive,
time-consuming process
● Synthetically generated images
○ Nearly infinite dataset
○ Challenge is making synthetic images look like real cards and
match real-world conditions

15. Training Data
Generate ~1M synthetic images

16. Training Process
● AWS P3 instances with V100 GPUs
● Training (2 epochs) takes ~1 day
with 1 GPU
● Always initialize weights with
previous best model

17. Validation
Manually label single images
of credit cards
Compare precision/recall of
models on real cards

18. Post-processing
● How do we get card number from a list of bounding boxes?
○ Clustering! (DBScan)
○ Perform checks (e.g. checksum) on each cluster to
determine if credit card number
● Aggregate multiple frames for even better coverage
9 4

19. Deploying on Mobile
Which Framework?
● CoreML
○ + Performance
○ - iOS 11 only
● TensorFlow
○ + Flexibility
○ - Binary Size/Performance

20. Optimizing for Mobile
Three metrics:
● TensorFlow library size (MB)
● uScan model size (MB)
● Performance (inferences/second)

21. Optimizing for Mobile
Start:
● TensorFlow Library Size — 25 MB
● uScan model Size — 25 MB
● Performance — 0.5-1.0 inferences/second

22. Optimizing for Mobile
Current:
● TensorFlow Library Size — 1.14 MB
● uScan model Size — 1 MB
● Performance — 5 inferences/second
uScan reads >97% of cards!

23. Optimizing for Mobile
How?
● TensorFlow Library Size — 25 -> 1.14 MB
○ Selective Registration
● uScan model Size — 25 -> 1 MB
○ Quantization, model optimizations
● Performance — 1 -> 5 inferences/second
○ Threading, memory & model optimizations

24. Cool way to get 20% more inferences per second on
iPhone!

26. Optimizing for Mobile
Next:
● TensorFlow Library Size — <300 KB
○ TF Lite, CoreML
● uScan model Size — 0.5-2 MB
○ MobileNetV2, detect more objects
● Performance — 10-15 inferences/second
○ MobileNetV2, TF Lite

27. The Future
● Object detection is not limited to text, we can look for card features
● Performance optimizations
○ GPUs for inference (TF-Lite + CoreML)
○ Use MobileNet v2
● Android
○ Model optimizations, deployment framework
● Other applications of on-device OCR at Uber

28. Thanks!