This document provides guidance for machine learning product managers and technical leaders on building effective ML products. It discusses introducing ML in enterprises, defining product specifications, planning under uncertainty, and building balanced ML teams. It also covers the ML product lifecycle, including tracking experiments, centralized data storage, automated testing, continuous integration, and serverless architectures. Serverless computing can help simplify deployments, improve scalability, and reduce costs.

1. Helixa
Tech Leader’s Guide to
Effective Building of
Machine Learning
Products
Gianmario Spacagna
Chief Scientist @ Helixa
ML for Enterprises Conference
Rome, 28th October 2019

2. About Me
7+ years experience in building Machine Learning products
Currently leading a team of ML Scientists and ML Engineers
Background in Software Engineering of Distributed Systems
MBA Candidate
Co-author of Python Deep Learning
Contributor of the Professional Data Science Manifesto
Blogger of Data Science Vademecum
Founder of the DataScienceMilan.org community
Stockholm, London, Milan
Gianmario Spacagna
Chief Scientist, Helixa
gspacagna@helixa.ai

3. Agenda
Manager’s guide (40 minutes)
1. Introducing ML in the Enterprises
2. Defining the ML Product Specifications
3. Planning Under Uncertainty
4. Building a balanced ML Team
Tech Leaders’ guide (20 minutes)
5. ML Product Lifecycle
6. Serverless architectures

4. 5. ML Product Lifecycle

5. Cloud Providers Disclaimer
The following examples will focus on AWS stack but consider
that other cloud providers offers similar services.
It is not part of this talk to compare different cloud solutions.

6. * from “The Start-Up Trap” by Robert C. Martin (Uncle Bob)
“The only way to go fast is to go well” cit. Uncle Bob

7. Overview of a real-world ML production system
Source: https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems
Only a small fraction of real-world ML systems is
composed of the ML Code. The required surrounding
infrastructure is vast and complex.

8. Overview of a ML component lifecycle
Picture source: https://medium.com/microsoftazure/how-to-accelerate-devops-with-machine-learning-lifecycle-management-2ca4c86387a0

9. Manage the lifecycle with dedicated platforms
Picture source: www.mlflow.org
Predictions
Model
Serving
Training

10. Other Machine Learning lifecycle platforms and tools
Picture source: www.mlflow.org
TensorFlow Extended (TFX)
Data Version Control
HopsWorks

11. Native Cloud Object (Data) Storage
Benefits:
● Cheaper
● Elastic
● Highly available
● Performant
“The benefit of HDFS is minimal and not
worth the operational complexity”
Source: DataBricks

12. Keep Your Datasets Registered in a Catalog
Production solution: AWS Glue Data CatalogManual solution

13. Manage training labels using Snorkel
www.snorkel.org

14. Dev tool stack and workflow
Pull
Notebooks and data
stored in S3 in shared
folders
S3 buckets mounted locally
via Alluxio cache for fast and
cheap access to data
Commit
and
push
Dev Unix
Machine
Notebook name matching
branch ID
Header cell:
1. Pull the latest version of the
code and install locally
2. Print the git status and
dependency versions
Develop code in the laptop
using professional IDEs
Feature branches
matching Jira key
Branching models:
● GitFlow
● Trunk-based

15. Processing large datasets with Elastic MapReduce (EMR)
Picture source: https://dimensionless.in/different-ways-to-manage-apache-spark-applications-on-amazon-emr/
Ephemeral clusters on spot instances can dramatically
reduce the cost of operations compared to long running
ones.

16. Processing large datasets from notebooks using
and EMR within the same workflow
Picture source:

17. Port analysis findings into a production-quality modules with a
task-oriented design and entry points declared in makefiles
Picture source: https://medium.com/@davidstevens_16424/make-my-day-ta-science-easier-e16bc50e719c
Task:
1. Read
2. Transform
3. Write

18. Deliver jobs inside containers whenever is possible
Advantages:
● Isolated environment
● Different library requirements
● Different resources (memory, CPUs, GPUs)
● Simplified load balancing
● Scalable model serving

19. Processing chunks of data in parallel batch jobs
Source: https://spotinst.com/blog/cost-efficient-batch-computing-on-spot-instances-aws-batch-integration/
Containerized job logic

20. Orchestrate pools of containers using Kubernetes (K8s) for
inference services

21. Automated code testing pyramid
Unit tests
● Single methods of data
processing utils and major
components.
● Replace “assertEqual” with
uncertainty ranges on
predictions
70%
Integration tests
● Test the training, model
selection and tuning.
● Subset of component
integrations (e.g.
transformers followed by
model predictions)
20%
End-to-end tests
● Static and small dataset.
● Dry runs of the execution
plan.
● Check APIs work seamlessly
through every stage of the
pipeline.
10%

22. Bonus: Metamorphic testing allows to test ML algorithms by
generating complex, deep tests without the use of an oracle

23. Infrastructure-as-Code (IaC) is fundamental in order to have
fully-portable and consistent replicas of environments
Benefits:
● Reduced labor cost
● Speed of provisioning
● Minimizes errors and security violations

24. Automate tasks using Continuous Integration
On commit
Deployment tasks:
● Re-training of models
● Model selection
● Hyper-parameters tuning
● Update pipeline components
● Update microservices
● Publish builds and Docker containers
On release
Picture source: https://deploybot.com/blog/the-expert-guide-to-continuous-integration

25. Release without pain
Source: Spotify Engineering Culture — Part 1

26. Validate hypothesis and releases with A/B testing
Source: https://www.optimizely.com/optimization-glossary/ab-testing/

27. Centralized logging with the ELK stack
Generate Logs Aggregation &
Transformation
Storage & Indexing Visualization & Analysis

28. Infrastructure Monitoring and Alerting
Basic Monitoring:
AWS resources and
custom metrics
generated by your
applications and
services
Focus on IT Monitoring:
Cloud-scale monitoring of
logs, metrics and traces
from distributed, dynamic
and hybrid infrastructure.
Focus on App Monitoring:
All-in-one performance
management tool from the
end user experience,
through servers, down to
the line of application
code.

29. Governance and Auditability
Audit changes in the
configuration of resources.
Track account activity by
recording AWS console actions
and API calls.

30. Respect the Responsible AI principles
Source: https://ethical.institute/principles.html

31. Adopt the eXplainableAI Framework
Source: https://ethical.institute/xai.html

32. The 43 Rules of ML Engineering
Martin Zinkevich
Google Research Scientist
https://developers.google.com/machine-learning/guid
es/rules-of-ml/

33. 6. Serverless

34. Serverless, or how to
build and run
applications without
thinking about
servers

35. In serverless, the cloud provider is responsible for executing a
piece of code by dynamically allocating the resources
Traditional Serverful Way:
Serverless Way:
Source: https://serverless-stack.com/chapters/what-is-serverless.html

36. Philosophy behind Serverless
"If a tree falls in a forest and no one is
around to hear it, does it make a sound?"
“If a server runs in the cloud and no one
is around to use it, does it need to incur
any costs?”
WinterClouds

37. Reasons to migrate to Serverless
Secure Scalable Cheap
Always available Worry free Low maintenance

38. An overview of Serverless services available in AWS
Docker container
execution.
Script execution in
response of events.
Full list available at https://aws.amazon.com/serverless/
Orchestration of
components and
microservices
Queuing +
publisher/subscriber
message services.
NoSQL Key-Value
database.
REST API
management
service.
Query service to
analyze data at scale
using standard SQL
(like PrestoDB).
ETL service to crawl and
process large datasets on
a fully managed Spark
environment.

39. Lambda function: listing files in a specified S3 directory
Event object Result objectPython script
Lambda cost: $1.04 / million requests
S3 LIST request cost: $5 / million requests

40. Serverless.com application framework
Hybrid solution for:

41. Orchestrating functions using state machines via Step Functions

42. Serverless scientific computing and Map/Reduce with PyWren
Pictures source: https://www.slideshare.net/AmazonWebServices/massively-parallel-data-processing-with-pywren-and-aws-lambda-srv424-reinvent-2017

43. Final Remarks

44. * from “The Start-Up Trap” by Robert C. Martin (Uncle Bob)
“The only way to go fast is to go well” cit. Uncle Bob

45. Overview of a real-world ML production system
Source: https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems

46. Embrace the serverless movement

47. Read the
Manager’s Guide
(first part)

48. Gianmario Spacagna
Chief Scientist at Helixa.ai
gspacagna@helixa.ai
@gm_spacagna

49. Appendix A:
Summary Steps

50. Steps to Managing the ML Product Lifecycle
1. Familiarize with the whole lifecycle and most popular tools and libraries.
2. Adopt a platform such as MLflow to track and version models and experiments.
3. Notebooks are good for explorations but the implementation should be in a codebase.
4. Make analysis, code and infrastructure, reproducible and avoid manual operations.
5. Communicate analysis results effectively summarizing only what is relevant.
6. Invest on automated tests at different integration levels.
7. Exploit Continuous Integration (CI) for automating builds and releases.
8. Deliver models and components inside Docker containers, when possible.
9. Centralize the logs collection for debugging and troubleshooting.
10. Monitor the infrastructure health using specific tools.
11. Consider a strategy for implementing Governance and Auditability.

51. Steps to migrate to Serverless architectures
1. Reverse Conway’s law: “Organizations produce software that resemble their
organizational communication structures”.
2. Divide your architecture in separate and simple services.
3. Adopt the serverless.com framework to make easier to develop lambda functions.
4. Pick the most suitable serverless MapReduce architecture for your needs.
5. Enjoy your team having fun with simplified and scalable deployments.
6. Make a report to your boss showing the consistent amount of saved costs.

52. Appendix B:
Serverless MapReduce

53. How can I process
large datasets using
serverless?

54. Serverless MapReduce with PyWren serializes and run local
Python code and return results back to the driver
Pictures source: https://www.slideshare.net/AmazonWebServices/massively-parallel-data-processing-with-pywren-and-aws-lambda-srv424-reinvent-2017

55. Serverless MapReduce with events sourced from S3
Picture source: https://aws.amazon.com/it/blogs/compute/ad-hoc-big-data-processing-made-simple-with-serverless-mapreduce/

56. Serverless MapReduce with Parallel tasks invoking
synchronously up to 10 concurrent lambdas
* A single Lambda function only supports up to 10 concurrent executions when invoked synchronously

57. Serverless MapReduce with queue polling invoking
asynchronously many concurrent lambdas within AWS limits*
...
Mapper2
Mapper1
Mapper n
SQS queue
Poll the queue
Driver
* StepFunctions has a limit of 1000 transitions/second and a max execution history size of 25k events.

58. Serverless MapReduce with activity callbacks invoking unlimited
parallel executions without limits
Source: https://semantive.com/part-2-asynchronous-actions-within-aws-step-functions-without-servers/
...
...
mapper1 mapper n
Get activity token and wait for
mapper activity to complete
Start mapper activity asynchronously
with the corresponding token
Send activity task success
s
driver