This document summarizes best practices for engineering production-ready software with Apache Spark. It discusses how data scientists focus on iterative experimentation and model performance while engineers focus on stability, maintainability, and performance. Simply taking prototypes to production can cause issues. The document demonstrates building well-engineered software with Spark that enables both research and experimentation, getting the best of data science and engineering. It emphasizes using practices like modular and testable code, and designing interfaces that allow experimentation on engineered artifacts.

2. Best Practices for Engineering
Production-ready Software with
Apache Spark
Maximo Gurmendez - Chief Engineer, Montevideo Labs
Javier Buquet - Staff Software Engineer, Montevideo Labs

3. Agenda
Our work at Montevideo Labs
Production-ready software
Why notebooks are not enough?
Demo!
Takeaways

4. Goals of Data Scientist and Engineers
▪ Iterative experimentation
▪ Successful prototypes
▪ Performance (model
accuracy)
▪ Stability
▪ Maintainability &
testability
▪ Performance (low latency,
high throughput, etc)
Software & Data Engineers focus on:Data Scientists focus on:

5. What if I take my prototype to production as is?
▪ Future data might not have the same assumptions.
▪ Breaks CI/CD chain. There's a reason we have integration tests.
▪ Model behavior in a UI might not be intuitive (usability).
▪ Might not scale well / have deadlocks / memory leaks / etc.
▪ Easy way to assign blame.

6. This time it's going to be different...

7. How decoupled are prototypes from software products?
▪ Software products benefit from best engineering practices, but as
important in data products is enabling research and experimentation.
▪ Spark is a great common language that helps bringing these two closer
together.
▪ We will show how we build well engineered software with Apache
Spark that gets us the best of both worlds.
Research Engineering

8. DEMO!

9. Demo example from:

10. Recommend attractions using pics at locations

12. https://github.com/montevideolabs/attractions-recommender
Demo Code:

13. Takeaways
Building Smart Data Products takes Data Science and Engineering to
work in teams and using the learnings from both disciplines
To go from prototypes to products use best practices such as:
▪ cohesive modules
▪ reusable and composable artifacts
▪ testable units
Allow best of both worlds by:
▪ building proper artifacts from prototypes
▪ making these artifacts available for data science research
▪ designing good interfaces for experimentation

14. Feedback
Your feedback is important to us.
Don’t forget to rate and
review the sessions.