Manu Mukerji walks you through Acme Corporation’s machine learning example for universal catalogs, explaining how the training and test sets are generated and annotated; how they were created when there is no public training data available; how the model is pushed to production, automatically evaluated, and used; how Acme Corporation built a Hadoop/Spark pipeline using different types of models predicting various values; production issues that arise when applying ML at scale in production; and lessons learned along the way.

1. Machine Learning in Production
Manu Mukerji

2. What is this talk about?
 Agenda:
- Introduction to the business problem
- Normal ML Flow
- Training Data
- Test Data
- Model Creation
- Testing
- How it ties together: The production flow
- Team Setup
- ML Production Examples
- Questions?
- Live Demo!

3. Typical ML Example
 This is the ”hello world” equivalent of what you find online:

4. Generalized ML Flow
1. Gather Data
2. Train Model
3. Test for accuracy (most examples end here)
4. Save model for external consumption
5. Use saved model for prediction

5. Our use case
 Categorization of products
 Categorization of products into ~4800 categories
 Categorization of 6B products into ~4800 categories across 30 countries
 Categorization of 6B products into ~4800 categories across 30 countries every day!

6. Training Data
Data is the asset!

7. Training data …lack of training data

8. Gathering Training Data
Annotation UI
HDFSElasticsearch
• Annotate at category level (Less than 10K/country)
• Try and map customer category to Google Category
• Expand training data and infer GCAT from mapping (this brings
it into millions of products)
• What else did we try:
• Mechanical Turk
• External Companies that provide data

9. More About Training Data
 Bad data, bad predictions
 Overfitting: if you have a hammer, everything looks like a nail
 When to retrain?
 When to add more data?

10. Test Set
 Test set selection
- Normal method: split 10-20% from training data
- Production method: custom test set based on business value
 Scoring of test set
- Normal method correct/total
- Problem that can occur with test set scoring
- Advanced version: negative points for negative customer value

11. Manual Overrides… Why?
 Prediction will never be 100% accurate
 When its wrong it impacts business

12. Testing your model
 Training takes time, this is why your test set is really important
 Automate the build pipeline to run evaluation and deploy your model only if its better than
existing one
 Canary test the whole pipeline
 Advanced Resource:
- Chase Roberts: https://medium.com/@keeper6928/how-to-unit-test-machine-learning-code-57cf6fd81765

13. Scaling Out
 Now we need to do this in 30 countries
 Easier with Latin alphabet languages
 Deep learning to the rescue: this isn't just a cool thing that folks are talking about
4.7 英寸 Retina HD 示器。一款采用显 64 位 面 架 的桌 级 构 A8 芯片。 焦点像素带
的 8MP iSight 相机。触摸 ID ...
4.7 inch Retina HD display. A 64-bit desktop architecture with the A8 chip.
8MP iSight with focus pixel. Touch ID

14. Accuracy over time
 70% accuracy ….1 month
 80% accuracy ….3-6 months
 ~90% accuracy …. 1 year
 The last mile……. Rest of your life!

15. Team setup for ML
 Team setup
- UI/API Team
- ML/AI Engineers
- ML/AI Research

16. Knowing when to stop?
 This is hard!
 Get something out!
 Don’t work in a vacuum
 Get the circular data flow working
 Remember business value, don’t over engineer it

17. How it all ties in together

18. Examples of interesting AI in production
 Self driving cars
- Some Components:
- Lots of sensors, cameras
- Object detection, and distinguishing what can move vs not
- Lane detection
- Red light vs green

19. Self Driving..continued
 If I trained a self driving car model with just 50hrs of data would you trust it?
 Probably not…
 According to the DMV in order to get a license: “Have completed 50 hours of practice with
an adult 25 of age years or older.”1
 50 Hours at 60 MPH is 3000 miles…
 Would you trust 1,300,000,000 miles?
1
https://www.dmv.ca.gov/portal/dmv/detail/teenweb/permit_btn1/permit
2
https://www.bloomberg.com/news/articles/2016-12-20/the-tesla-advantage-1-3-billion-miles-of-data

21. The future

22. Questions?

23. Live Demo: Help me with training data
 You remember how I said Training data is hard to get!

24. Thank You
 TensorFlow
 Pandas, Scikit-learn
 Andrew Ng!