This document discusses the journey of a startup called Experience to productionize data science. In 2016, Experience had 13 engineers and 1 data scientist. The goal for 2017 was to make an impact on customers through predictive modeling and deploying models into production in real-time using minimal engineering resources. Experience explored using H2O for scalable machine learning due to its Java implementation and ability to export models. This allowed Experience to create a production pipeline using H2O, Python for preprocessing, and services like Docker and ECS for deployment with no additional engineering effort. While successful, there were limitations using only H2O algorithms and not leveraging Python more. Overall, the document outlines Experience's process to operationalize data science within a startup

1. Productionizing Data Science at
A Startup's (ongoing) Journey building Machine Learning Products

2. Agenda
• Intro
• Setting the Scene
• Proposed Pipeline
• A Look Back

3. Who am I?
• Matt Mills
• Born and raised in Atlanta
• BS in Industrial and Systems Engineering 2014, MS in Analytics 2015
• @statmills or www.statmills.com

4. Experience's mobile commerce, ticketing, and data
solutions empower sports and entertainment
leaders to generate new revenue streams, sell more
tickets, and make smarter decisions.
www.expapp.com/solutions
What is Experience?

5. What is Experience?

6. What is Experience?

7. Agenda
• Intro
• Setting the Scene
• Proposed Pipeline
• A Look Back

8. Data Science at the end of 2016
• ~13 Engineers and 1 (me!) Data Scientist

9. Data Science at the end of 2016
• ~13 Engineers and 1 (me!) Data Scientist
• What happens to my work?

10. Data Science at the end of 2016
• ~13 Engineers and 1 (me!) Data Scientist
• What happens to my work?
Manager /
Management
Other
Departments
Partners

11. Data Science at the end of 2016
• ~13 Engineers and 1 (me!) Data Scientist
• What happens to my work?
Manager /
Management
Other
Departments
Partners

12. Goal for 2017
• Make an Impact on our Customers (Fans)
Influence
Fan
Behavior
Manager /
Management
Other
Departments
Partners

13. Goal for 2017
• Make an Impact on our Customers (Fans)
Predictive
Model
Influence
Fan
Behavior
Manager /
Management
Other
Departments
Partners

14. Goal for 2017: Continued
• Create a process to deploy models into production and use
predictions in real time

15. Goal for 2017: Continued
• Create a process to deploy models into production and use
predictions in real time
• Some considerations
• Minimal use of limited Engineering Resources
• Scalable (speed and processing power)
• Cheap, like, super cheap (read: Free)
• Had to handle data cleansing

16. Some Potential Solutions

17. Some Potential Solutions
• Build own R/Python Server

18. Some Potential Solutions
• Build own R/Python Server
• Learn Scala/Spark

19. Some Potential Solutions
• Build own R/Python Server
• Learn Scala/Spark
• Pay for ML Service

20. Some Potential Solutions
• Build own R/Python Server
• Learn Scala/Spark
• Pay for ML Service

21. Scaling Experience Data Science with h2o
• ML Algorithms written in pure Java
• APIs written for R, Python, Scala, Spark
• Built for scale
• parallel and distributed out of the box
• Open Source

22. Scaling Experience Data Science with h2o
• ML Algorithms written in pure Java
• APIs written for R, Python, Scala, Spark
• Built for scale
• parallel and distributed out of the box
• Open Source
• Models exportable as Java Objects
to embed in other apps
• Can embed python pre-processing
scripts within the POJO

23. Agenda
• Intro
• Setting the Scene
• Proposed Pipeline
• A Look Back

24. h2o Architecture
https://github.com/h2oai/h2o-meetups/blob/master/2017_09_12_Dublin/
2017_09_12_H2O_Intro_and_AutoML.pdf

25. h2o Algorithm List

26. h2o vs scikit-learn Syntax and Process
https://github.com/h2oai/h2o-meetups/blob/master/2015_05_14_H2O_Overview/H2O_Overview.pdf

27. Experience Production Pipeline

28. Experience Production Pipeline

29. Experience Predictive Modeling Pipeline
1. App Sends Data
2. Data Cleaning in Python 3. Predictions done in h2o
4. App Gets Prediction
input()
sys.stdout.flush()
{JSON}

30. Benefits of Using Open Source Software

31. Experience Predictive Modeling Setup

32. Model Deployment
Code
Pulled via Github
Terraform
to create infrastructure
and manage state
Served
via ECS
Dockerize
via Dockerfile and
stored in ECR
Discovery
via Consul

33. Agenda
• Intro
• Setting the Scene
• Proposed Pipeline
• A Look Back

34. Pros and Cons of Current Set-Up
Pros
• Automated process to deploy
models into production
• Can iterate models with no/limited
effort from engineering
Cons
• Can only use algorithms available
to h2o (e.g. no multilevel models,
GAMs, Bayesian)
• h2o drives Python, why not the
other way around?

35. Conclusion and Questions
1. Lack of skills and/or support doesn’t have to stop you from putting models
into production
2. What’s best for your Data Scientists might not be best for your Engineers
and vice-versa

36. Conclusion and Questions
1. Lack of skills and/or support doesn’t have to stop you from putting models
into production
2. What’s best for your Data Scientists might not be best for your Engineers
and vice-versa
www.statmills.com
? http://docs.h2o.ai/
https://www.expapp.com/about/#careers