Big data, data science, and machine learning rose to prominence around a decade ago, and have become cemented in the tech landscape as the size and complexity of data continues to increase. However, many companies still are confused about how to best make use of their data, and try to hire "all-in-one" superstars who can do everything from research to creating efficient ETLs to maintaining machine learning projects in production. In this talk, we'll explain how most data science or machine learning work should be a collaboration between dedicated data scientists and data engineers. We'll talk about the core responsibilities of each role, where they can each add the most value to companies, and also where their roles overlap. We'll also discuss where specializations such as machine learning scientists, machine learning engineers, DBAs and ML ops fit in, and whether the future seems to be heading more towards generalization or specialization. We'll end with recommendations on how you can build the best combination of these roles for your company's needs.

1. Big Data LDN, September 2022
Dr. Jodie Burchell, Data Science Developer Advocate
Pasha Finkelshteyn, Big Data Developer Advocate
Avoid hiring data ninja
rockstars
How to build effective data teams

2. Who works with data?

3. Who works with data?
Of course, data scientists!

4. Who works with data?
Of course, data scientists! (not only)

5. No one can or even should do everything
The reality is …
Responsibilities should be distributed

6. Set up: Large fashion e-commerce Trasos
Task: recommender system
Example project

7. What the business says
Increase retention
Reduce cost of acquisition

8. What does the business mean?
Customers be like

9. Easier to convert non-customers to customers
Customers should return sooner
Business outcomes

10. Lots of simple events (clicks, scrolls,
interactions)
Backend is in Scala, but data science team
works with Python
Technical constraints

11. Approach
1. Define business question
2. Define required data
3. Prepare data
4. Research models
5. Productionise model
6. Monitor

12. Data skills
Domain
Analysts

13. Data skills
Domain
Analysts
Science
Data scientists
BI analysts

14. Data skills
Domain
Science Engineering
Analytics engineer
ML engineer
Data engineers

15. The three roles

16. Data Scientists
The research folk

17. Data scientist skills
● Data + science!
● Automate manual processes
● Build data products
● Mathematics
○ Statistics
○ Machine and deep learning
● Programming
● Data visualisation

18. Data scientist responsibilities
1. Translate business question
2. Research available data
3. Request data sample from data engineer

19. Data scientist responsibilities
4. Model building
● Feature engineering
● Iteratively find acceptable model
5. Handover model for productionisation
6. Monitor model metrics

20. Business intelligence
analysts
The data communication folk

21. BI analyst skills
● Deep understanding of business
● Data communication and visualisation
● Data wrangling
● Data exploration and analysis

22. BI analyst responsibilities
1. Understand feasibility of business goals
2. Design metrics to measure business success
3. Request data sample from data engineer

23. BI analyst responsibilities
5. Request tables needed for reporting
6. Monitor business metrics
● Creating dashboards
● Ad hoc reporting

24. Data Engineers
The big data folk

25. Data engineer skills
● Data lifecycle
● Strong engineering skills
● Monitoring data pipelines
● Manage data needs for multiple teams
● (Sometimes) responsible for ML models in production

26. Data scientist responsibilities
1. Check whether data will support project
2. Track down data
3. Extracting samples

27. Data scientist responsibilities
4. Input as to whether model is technically feasible
5. Production:
● Build data pipelines
● (Potentially) implement model
6. Monitor data metrics

28. What is the definition of done of the
model?
Overlaps

29. Who is responsible when things go
wrong in production?
Overlaps

30. Summary
1. There are many responsibilities around data
2. These responsibilities are broad
3. Work can be split between three dimensions
4. Hire team according to needs

31. Thank you!
@t_redactyl
t-redactyl.io
@asm0di0
Jodie Pasha