While modeling techniques in machine learning have matured drastically, the deployment of models at scale has been overlooked. These are some learnings that I've had over the years, that I presented at Cognizant in Amsterdam.

1. Notes on Deploying Machine-Learning Models at
Scale
Or, What they might not teach you in Data Science school!
Deep Kayal
11.10.2019 @

2. “The sexiest job of the 21st century” [1]
[1] https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century

3. So we’re sorted..or are we?
Machine learning offers a fantastically powerful toolkit for building useful complex prediction systems quickly. This paper
argues it is dangerous to think of these quick wins as coming for free. Using the software engineering framework of technical
debt, we find it is common to incur massive ongoing maintenance costs in real-world ML systems. [2]
--- Hidden Technical Debt in Machine Learning Systems, Google, NeurIPS 2015.
[2] https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf

4. How did we get here?
[3] https://www.bastagroup.nl/wp-content/uploads/2019/01/the-state-of-machine-learning-adoption-in-the-enterprise.pdf

5. The need for engineering in Data Science [4]
[4] https://www.oreilly.com/radar/what-are-machine-learning-engineers/
[5] https://www.oreilly.com/radar/we-need-to-build-machine-learning-tools-to-augment-machine-learning-engineers/
[6] https://www.youtube.com/watch?v=mJHvE2JLN3Q

6. Making useful models useful
- Let’s say you’re tasked with making a text classifier. You design a classifier..
- You fit it to your labeled dataset
- You check your cross-validation performance; it is very high, so you celebrate!
- What’s next?
Text TFIDF SVM LabelStem / Lemmatize

7. Why not do a small scale sanity check?
- Not an A/B test, but a sanity check
- Offline cross-validation performance is a great start
- But the first real acid test is to take completely unseen real data..
- ..pass it through the trained model..
- ..and get it fact-checked by an expert
- Example: a classifier to check if a paragraph is talking about a company’s annual report or not
- Expert maybe an risk analyst or an underwriter
- You can do this using something as simple as Excel:
- Expert select data points to validate
- You pass them through model
- You write data and labels to excel
- Expert checks if they are correct or wrong
- But this is uncontrolled, hand-held and chaotic

8. A better alternative
Pipeline
API
HTTP Server
VM
DB
Enter data
Predicted Label
Verify
Y N
Simple form
Expert
[7] https://towardsdatascience.com/publishing-machine-learning-api-with-python-flask-98be46fb2440
[8] https://pythonspot.com/flask-web-forms/

9. And with that, you’re on our way!
- Now you have re-usable template of a minimalist API you can build wrapping around your
intelligent model
- The web form and the objects exchanged over REST calls might change
- But the core can stay the same
- With our sanity check done, let’s move forward and deploy our model for incoming data

10. Deploying models at scale: A blueprint
.
.
.
Services
Load
Balancer
Registry
External
gateway
API
NGINX
Consul
Discover
Query/Result
Query/Result
Query/Result
Query/Result
Register
Service
Query/Result
Infrastructure as code
State
information

11. Types of deployment
- You’ve deployed a scalable API for streaming data
- This handles incoming requests
- And can be scaled horizontally
- But what about historical data, which needs to be processed?
- You’ve deployed a scalable API for streaming data
- A stream processor
- And a batch processor
- We can use the stream API as a batch processor, or…

12. Something for batch processing
Pipeline
Serialize
# model -> sklearn model
# sc -> spark context
spark_model = sc.broadcast(model)
def udf_predict(feature):
return spark_model.value.predict_proba(feature)
This way, you can leverage
Spark’s internal scalability
and fault tolerance aspects!
[9] https://towardsdatascience.com/deploy-a-python-model-more-efficiently-over-spark-497fc03e0a8d

13. Recap
- You have developed a machine learning model
- You have wrapped it up as an API
- You’ve tested it with real users
- Now, using best practice blueprints from software engineering, you can now deploy the model
seamlessly on forward-flow content
- Additionally, you’ve also managed to create a smart batch processor, by wrapping your model with a
Spark UDF
- With these two services, you can make a MLaaS for almost any project and task!
- Plus, all of this code (and effort) is reusable for new tasks. Just replace the model with a new one!

14. Final words
- Treating ML-software as any other software is our best bet moving forward
- But it’s hard!
- ML services are hard to unit test, for one
- It’s non-deterministic
- It produces a lot of binary artifacts to store and maintain
- …
- Consider metamorphic testing [10]
- Use data science version control systems such as DVC or MLFlow [11, 12]
- Version data, then you can recreate a model at any point in history
- Or version the model, to rollback easily
- Or better yet, version both!
[10] https://en.wikipedia.org/wiki/Metamorphic_testing
[11] https://dvc.org/
[12] https://mlflow.org/

15. Deep Kayal
Reach out, if there’s anything I can help
with: 
deep.kayal@pm.me
https://www.linkedin.com/in/subhradeepk/