How Convoy mitigates training-serving skew, the biggest challenge in deploying machine learning to production.

1. @MichaelFrasco
Deploying Machine Learning in
R with Amazon SageMaker

2. Where I work
● Founded in 2015
● A two-sided marketplace
for shippers and carriers
● Shippers
○ Companies that need
to move freight across
the country
● Carriers
○ Truck drivers that can
move the freight

3. Machine Learning is Critical to Our Success

4. Machine Learning is Critical to Our Success

5. Machine Learning Landscape
Data Scientists
Own the machine
learning algorithms

6. Machine Learning Landscape
Data Scientists
Own the machine
learning algorithms
Model
Training
Environment

7. Machine Learning Landscape
Software Engineers
Own the uptime of our
business systems

8. Machine Learning Landscape
Software Engineers
Own the uptime of our
business systems
Convoy
Production
Environment

9. Machine Learning Landscape
Software Engineers
Own the uptime of our
business systems
Convoy
Production
Environment
Data Scientists
Own the machine
learning algorithms
Model
Training
Environment

10. Training-Serving Skew

11. Training-serving skew is a difference
between a model’s behavior during
training and during serving.

12. Examples of training-serving skew
● Distance to the nearest metro
● Different definitions of the
standard deviation
○ MLE vs unbiased

13. Why does training-serving skew happen
● Iterating quickly and
constantly
● Communication issues
between data scientist and
software engineer
● Different programming
languages

14. How do we prevent training-serving skew

15. Upload trained
model
Model Training
Environment
Model Serving
Environment

16. Upload trained
model and
code
Model Training
Environment
Model Serving
Environment
Respond with result
Request prediction
Convoy
Production
Environment

17. ● REST APIs
○ Established technology
for software engineers
● plumber
○ Widely used R
package
○ Transforms your
functions into an API
Communication Between Languages

18. plumber requires a one line change
Tells the internet
how to find this function

19. But it is actually much more complicated
● Security!
● Reliability!
● Monitoring!
● Scaling!
● Unknowns!

20. Should we build it ourselves?
Should we buy it from someone else?

21. Decided to buy from AWS SageMaker

23. Upload to
Upload to

24. Upload to
Upload to

25. We have seen terrific benefits
● Increased ownership
○ Less reliance on
software engineers

26. We have seen terrific benefits
● Increased ownership
○ Less reliance on
software engineers
● Faster iterations between
model improvements

27. We have seen terrific benefits
● Increased ownership
○ Less reliance on
software engineers
● Faster iterations between
model improvements
● Reliable and trustworthy