How can you ensure that your work and use of ML gets the most impact in the domain you apply it to? From collaborating with all stake-holders to simulating how predictions will really be used, evaluating them domain-side and deploying models at scale in production, I’ll share some of the lessons I’ve learnt when it comes to integrate ML in real-world applications. Also, I’ll review some research problems and new open source software aimed at making it easier to create, experiment with, and operationalise predictive models.

1. Pragmatic Machine Learning
@louisdorard
#MLSpain - 18 Jan 2016

2. I’M LAZY

5. “Programming is for lazy people
who want to automate things
— AI is for lazier people who want
to automate programming”

6. • Consider manual classification task
• Automate with ML model?
• Build PoC
• Deploy in production
• Maintain
• Monitor performance
• Update with new data
6
The Lazy MLer
Phrase problem as ML task
Engineer features
Prepare data (csv)
Learn model
Make predictions
Deploy model & integrate
Evaluate model
Measure impact

7. • Florian Douetteau at PAPIs Connect in May 2015)
• Top companies invested more than 5M$ in their ML production
platform (Facebook, Amazon, LinkedIn, Spotify…)
7
Cost of ML projects

8. • Real-world ML is/was complicated and costly (especially at
web scale)
• Do I really need ML?
• How about Human API? (e.g. Amazon Mechanical Turk)
• → Back to Square 1 (but someone else’s problem!)
• → Baseline! (performance, time, cost)
8
The Lazy MLer

9. Performance evaluation

10. How do you evaluate the
performance of an ML system?

11. Accuracy
Latency
Throughput
11
Performance measures

13. • Go beyond accuracy… example: recommendations
• Get clicks!
• → Simulate how many you’d get with your model
• → Need to learn accurately what people like — not what they dislike
• Better decisions with ML
• Revenue increase (A/B test)
• Decisions can have a cost (e.g. give special offer/pricing to customer)… ROI?
13
Domain-specific evaluation

14. Decisions from predictions

15. 1. Descriptive
2. Predictive
3. Prescriptive
15
Types of analytics

17. 1. Show churn rate against time
2. Predict which customers will churn next
3. Suggest what to do about each customer
(e.g. propose to switch plan, send promotional offer, etc.)
17
Churn analysis

18. • Who: SaaS company selling monthly subscription
• Question asked:“Is this customer going to leave within 1
month?”
• Input: customer
• Output: no-churn or churn
• Data collection: history up until 1 month ago
18
Churn prediction

19. • #TP (we predict customer churns and he does)
• #FP (we predict customer churns but he doesn’t)
• #FN (we predict customer doesn’t churn but he does)
19
Churn prediction accuracy

20. Assume we know who’s going to churn. What do we do?
• Contact them (in which order?)
• Switch to different plan
• Give special offer
• No action?
20
Churn prevention

21. “3. Suggest what to do about each customer”
→ prioritised list of actions, based on…
• Customer representation + context (e.g. competition)
• Churn prediction (& action prediction?)
• Uncertainty in predictions
• Revenue brought by customer & cost of action
• Constraints on frequency of solicitations
21
Churn prevention

22. • Taking action for each TP (and FP) has a cost
• For each TP we“gain”: (success rate of action) * (revenue /cust. /month)
• Imagine…
• perfect predictions
• revenue /cust. /month = 10€
• success rate of action = 20%
• cost of action = 2€
• Which ROI?
22
Churn prevention ROI

23. • We predicted customer would churn but they didn’t…
• That’s actually good! Prevention worked!
• Need to store which actions were taken
• Is ML really helping?
• Compare to baseline,
e.g. if no usage for more than 15 days then predict churn
• Is fancy model really improving bottom line?
23
Churn prevention evaluation

24. 1. Show past demand against calendar
2. Predict demand for [product] at [store] in next 2 days
3. Suggest how much to ship
• Trade-off: cost of storage vs risk of lost sales
• Constraints on order size, truck volume, capacity of people
putting stuff into shelves
24
Replenishment

25. • Context
• Predictions
• Uncertainty in predictions
• Constraints
• Costs / benefits
• Competing objectives ( trade-offs to make)
• Business rules
25
Decisions are based on…

26. APIs are key

27. Software components for automated decisions:
• Create training dataset from historical data (merge sources, aggregate…)
• Provide predictive model from given training set (i.e. learn)
• Provide prediction against model for given context
• Provide optimal decision from given contextual data, predictions,
uncertainties, constraints, objectives, costs
• Apply given decision
27
Separation of concerns

28. Software components for automated decisions:
• Create training dataset from historical data (merge sources, aggregate…)
• Provide predictive model from given training set (i.e. learn)
• Provide prediction against model for given context
• Provide optimal decision from given contextual data, predictions,
uncertainties, constraints, objectives, costs
• Apply given decision
28
Operations Research component

29. Software components for automated decisions:
• Create training dataset from historical data (merge sources, aggregate…)
• Provide predictive model from given training set (i.e. learn)
• Provide prediction against model for given context
• Provide optimal decision from given contextual data, predictions,
uncertainties, constraints, objectives, costs
• Apply given decision
29
Machine Learning components

30. Software components for automated decisions:
• Create training dataset from historical data (merge sources, aggregate…)
• Provide predictive model from given training set (i.e. learn)
• Provide prediction against model for given context
• Provide optimal decision from given contextual data, predictions,
uncertainties, constraints, objectives, costs
• Apply given decision
30
Predictive APIs

31. The two methods of predictive APIs:
• model = create_model(‘training.csv’)
• predicted_output = create_prediction(model,
new_input)
31
Predictive APIs

32. Amazon ML
BigML
Google Prediction
PredicSis
… or your own company!
32
Providers of REST http Predictive APIs

36. ?

38. Experiment on “ScienceCluster”
• Distributed jobs
• Collaborative workspace
• Serialize chosen model
Deploy model as API on “ScienceOps”
• Load balancing
• Auto scaling
• Monitoring (API calls, accuracy)

40. • “Open source prediction server”in Scala
• Based on Spark, MLlib, Spray
• DASE framework: Data preparation, Algorithm, Serving,
Evaluation
• Amazon CloudFormation template → cluster
• Manual up/down scaling
40

41. → PAPI+

42. → PAPI+

44. Interesting research problems

45. 45
Concurrency for high-throughput ML APIs
Brian Gawalt (Senior Data Scientist at Upwork)
Talk at PAPIs ’15

46. upwork.com use case:
• predict freelancer availability
• huge web platform (millions of users)
→ need very high throughput and low latency
• things change quickly → need freshest data & predictions
46
Concurrency for high-throughput ML APIs

47. • event: invitation sent to freelancer
• steps to prediction:
• gather raw data from all sources
• featurize event
• make prediction
Concurrency for high-throughput ML APIs

48. • An actor…
• gets & sends messages
• makes computations
• Actors we need:
• “Historians”: one per data source
• “Featurizer”
• “Scorer”
48
Concurrency with Actor framework

49. 49
Concurrency for high-throughput ML APIs
before

50. 50
Concurrency for high-throughput ML APIs
after

51. • Python defacto standard: scikit-learn
• “Sparkit-learn aims to provide scikit-
learn functionality and API on PySpark. The main goal of the library is to cr
eate an API that stays close to sklearn’s.”
• REST standard: PSI (Protocols & Structures for Inference)
• Pretty similar to BigML API!
• Implementation for scikit available
• Easier benchmarking! Ensembles!
51
API standards?

52. • “AzureML: Anatomy of a machine learning service”
• “Deploying high throughput predictive models with the actor
framework”
• “Protocols and Structures for Inference: A RESTful API for Machine
Learning”
• Coming soon… JMLR W&CP Volume 50
• Get updates: @papisdotio or papis.io/updates
52
PAPIs ’15 Proceedings

53. 53
Simple MLaaS comparison
Amazon Google PredicSis BigML
Accuracy 0,862 0,743 0,858 0,790
Training 135s 76s 17s 5s
Test time 188s 369s 5s 1s
louisdorard.com/blog/machine-learning-apis-comparison

54. • With SKLL (SciKit Learn Laboratory)
• Wrap each service in a scikit estimator
• Specify evaluations to perform in a config file (datasets,
metrics, eval procedure)
• Need to also measure time…
• See papiseval on Github
54
Automated Benchmark?

55. • Return of the Lazy MLer!
• Model selection
• Find optimal values for n (hyper-)parameters
→ optimisation problem (function in n dimensions)
• Search space of parameters, efficiently → explore vs exploit
• Bayesian optimization?
55
AutoML

56. 56
Bayesian Optimization in 1 dimension
From CODE517E

57. 57
Bayesian Optimization in 1 dimension
From CODE517E

58. • Building ensembles
• Decide to continue training existing model, or to train new one
• Explore vs exploit again!
• Reward is accuracy. Let’s estimate reward for all options.
• Choose option with highest expected reward + uncertainty? (i.e.
upper confidence bound)
• Limited computational budget…
58
AutoML

59. • Zoubin Gharahmani & James Lloyd @ Uni Cambridge
• Gaussian Processes: find (mixture of) kernel(s) that maximises
data likelihood
• Also Bayesian!
59
Automatic Statistician

60. • Spearmint:“Bayesian optimization”for tuning parameters →
Whetlab → Twitter
• Auto-sklearn:“automated machine learning toolkit and drop-
in replacement for a scikit-learn estimator”
• See automl.org and challenge
60
Open Source AutoML libraries

61. 61
Scikit
from sklearn import svm
model = svm.SVC(gamma=0.001, C=100.)
from sklearn import datasets
digits = datasets.load_digits()
model.fit(digits.data[:-1], digits.target[:-1])
model.predict(digits.data[-1])

62. 62
Scikit
from sklearn import svm
model = svm.SVC(gamma=0.001, C=100.)
from sklearn import datasets
digits = datasets.load_digits()
model.fit(digits.data[:-1], digits.target[:-1])
model.predict(digits.data[-1])

63. 63
AutoML Scikit
import autosklearn
model = autosklearn.AutoSklearnClassifier()
from sklearn import datasets
digits = datasets.load_digits()
model.fit(digits.data[:-1], digits.target[:-1])
model.predict(digits.data[-1])

64. • Before learning:
• Automatic feature extraction from text?
• After learning:
• Monitor new predictions and automatically retrain models
when necessary?
• See panel discussion at PAPIs‘15
64
More automation ideas…

65. • Same as Azure ML?
• Scaling up? down?
65
Open Source Auto Scaling?

67. Tech talks:
• Intro to Spark
• Using ML to build an autonomous drone
• Demystifying Deep Learning (speaker needed!)
• Distributed Deep Learning with Spark on AWS
67
PAPIs Connect (14-15 March, Valencia)

68. Topics:
• Managing technology
• FinTech
• Enterprise, Retail, Operations
• AI for Society (Nuria Oliver, Scientific Director at Telefonica R&D)
• Future of AI (Ramon Lopez de Mantaras, Director AI Research at
Spanish Research Council)
68
PAPIs Connect (14-15 March, Valencia)

69. • Dev? Bring your manager!
• Manager? Bring your devs!
• Discount code: MLSVLC20
• papis.io/connect
69
PAPIs Connect (14-15 March, Valencia)