Managing ML Teams
ā¢
0 likesā¢14,634 views
Lecture 13: Machine Learning Teams (Full Stack Deep Learning - Spring 2021)
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Managing ML Teams
Similar to Managing ML Teams (20)
More from Sergey Karayev
More from Sergey Karayev (9)
Recently uploaded
Recently uploaded (20)
Managing ML Teams
- 1. Full Stack Deep Learning - UC Berkeley Spring 2021 Week 8 Machine Learning Teams
- 2. Full Stack Deep Learning - UC Berkeley Spring 2021 Running ML teams is hard 2 ML Teams - overview Running any technical team is hardā¦ ā¢ ML talent is expensive and scarce ā¢ ML teams have a diverse set of roles ā¢ Projects have unclear timelines and high uncertainty ā¢ The field is moving fast and ML is the āhigh- interest credit card of technical debtā ā¢ Leadership often doesnāt understand AI ā¢ Hiring great people ā¢ Managing and developing those people ā¢ Managing your teamās output and making sure your vectors are aligned ā¢ Making good long-term technical choices & managing technical debt ā¢ Managing expectations from leadership ā¦ And ML adds complexity
- 3. Full Stack Deep Learning - UC Berkeley Spring 2021 Goal of this module ā¢ Give you some insight into how to think about building and managing ML teams ā¢ Help you get a job in ML 3 ML Teams - overview
- 4. Full Stack Deep Learning - UC Berkeley Spring 2021 Managing Module overview 4 Hiring ā¢ How to manage a ML team ā¢ How to hire ML engineers. How to get hired. ML Teams - overview Orgs ā¢ How ML teams are organized and how they fit into the broader organization Roles ā¢ ML-related roles and the skills they require
- 5. Full Stack Deep Learning - UC Berkeley Spring 2021 Managing Module overview 5 Hiring ā¢ How to manage a ML team ā¢ How to hire ML engineers. How to get hired. Orgs ā¢ How ML teams are organized and how they fit into the broader organization Roles ā¢ ML-related roles and the skills they require ML Teams - roles
- 6. Full Stack Deep Learning - UC Berkeley Spring 2021 Most common ML roles ā¢ ML product manager ā¢ DevOps ā¢ Data engineer ā¢ ML engineer ā¢ ML researcher / ML scientist ā¢ Data scientist 6 ML Teams - roles
- 7. Full Stack Deep Learning - UC Berkeley Spring 2021 Most common ML roles ā¢ ML product manager ā¢ DevOps ā¢ Data engineer ā¢ ML engineer ā¢ ML researcher / ML scientist ā¢ Data scientist 7 Whatās the diļ¬erence? ML Teams - roles
- 8. Full Stack Deep Learning - UC Berkeley Spring 2021 Breakdown of job function by role 8 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc ML Teams - roles
- 9. Full Stack Deep Learning - UC Berkeley Spring 2021 Breakdown of job function by role 9 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc DevOps engineer Deploy & monitor production systems Deployed product AWS, etc. ML Teams - roles
- 10. Full Stack Deep Learning - UC Berkeley Spring 2021 Breakdown of job function by role 10 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc DevOps engineer Deploy & monitor production systems Deployed product AWS, etc. Data engineer Build data pipelines, aggregation, storage, monitoring Distributed system Hadoop, Kafka, Airflow ML Teams - roles
- 11. Full Stack Deep Learning - UC Berkeley Spring 2021 Breakdown of job function by role 11 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc DevOps engineer Deploy & monitor production systems Deployed product AWS, etc. Data engineer Build data pipelines, aggregation, storage, monitoring Distributed system Hadoop, Kafka, Airflow ML engineer Train & deploy prediction models Prediction system running on real data (often in production) Tensorflow, Docker ML Teams - roles
- 12. Full Stack Deep Learning - UC Berkeley Spring 2021 Breakdown of job function by role 12 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc DevOps engineer Deploy & monitor production systems Deployed product AWS, etc. Data engineer Build data pipelines, aggregation, storage, monitoring Distributed system Hadoop, Kafka, Airflow ML engineer Train & deploy prediction models Prediction system running on real data (often in production) Tensorflow, Docker ML researcher Train prediction models (often forward looking or not production-critical) Prediction model & report describing it Tensorflow, pytorch, Jupyter ML Teams - roles
- 13. Full Stack Deep Learning - UC Berkeley Spring 2021 Breakdown of job function by role 13 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc DevOps engineer Deploy & monitor production systems Deployed product AWS, etc. Data engineer Build data pipelines, aggregation, storage, monitoring Distributed system Hadoop, Kafka, Airflow ML engineer Train & deploy prediction models Prediction system running on real data (often in production) Tensorflow, Docker ML researcher Train prediction models (often forward looking or not production-critical) Prediction model & report describing it Tensorflow, pytorch, Jupyter Data scientist Blanket term used to describe all of the above. In some orgs, means answering business questions using analytics Prediction model or report SQL, Excel, Jupyter, Pandas, SKLearn, Tensorflow ML Teams - roles
- 14. Full Stack Deep Learning - UC Berkeley Spring 2021 What skills are needed for the roles? 14 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM Software engineering
- 15. Full Stack Deep Learning - UC Berkeley Spring 2021 What skills are needed for the roles? 15 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM Primarily a software engineering role. Often from standard SWE pipeline. Software engineering
- 16. Full Stack Deep Learning - UC Berkeley Spring 2021 What skills are needed for the roles? 16 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM SWE with ML team as an active customer Software engineering
- 17. Full Stack Deep Learning - UC Berkeley Spring 2021 What skills are needed for the roles? 17 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM Rare mix of ML skills and SWE skills. Often SWEs with significant self- teaching or science / engineering PhDs who worked as traditional SWEs after grad school Software engineering
- 18. Full Stack Deep Learning - UC Berkeley Spring 2021 What skills are needed for the roles? 18 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM ML experts. Usually have MS/PhD in CS or Stats or did an industrial fellowship program Software engineering
- 19. Full Stack Deep Learning - UC Berkeley Spring 2021 What skills are needed for the roles? 19 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM Wide range of backgrounds from undergrad-only to science PhD Software engineering
- 20. Full Stack Deep Learning - UC Berkeley Spring 2021 What skills are needed for the roles? 20 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM Traditional PMs, but with a deep understanding of the ML development process & mindset Software engineering
- 21. Full Stack Deep Learning - UC Berkeley Spring 2021 Questions? 21 ML Teams - roles
- 22. Full Stack Deep Learning - UC Berkeley Spring 2021 Managing Module overview 22 Hiring ā¢ How to manage a ML team ā¢ How to hire ML engineers. How to get hired. Orgs ā¢ How ML teams are organized and how they fit into the broader organization Roles ā¢ ML-related roles and the skills they require ML Teams - orgs
- 23. Full Stack Deep Learning - UC Berkeley Spring 2021 ML org structures - lessons learned ā¢ No consensus yet on the right way to structure a ML team ā¢ This lecture: taxonomy of best practices for diļ¬erent organizational maturity levels 23 ML Teams - orgs
- 24. Full Stack Deep Learning - UC Berkeley Spring 2021 ML organization archetypes 24 The ML Organization Mountain ML Teams - orgs
- 25. Full Stack Deep Learning - UC Berkeley Spring 2021 ML organization archetypes 25 The ML Organization Mountain Nascent / Ad-Hoc ML What it looks like ā¢ No one is doing ML, or ML is done on an ad-hoc basis ā¢ Little ML expertise in-house Example organizations ā¢ Most small-medium businesses ā¢ Less technology-forward large companies (education, logistics, etc) Advantages ā¢ Often low-hanging fruit for ML Dis- advantages ā¢ Little support for ML projects, diļ¬cult to hire and retain good talent ML Teams - orgs
- 26. Full Stack Deep Learning - UC Berkeley Spring 2021 ML R&D ML organization archetypes 26 What it looks like ā¢ ML eļ¬orts are centered in the R&D arm of the organization ā¢ Often hire researchers / PhDs & write papers Example organizations ā¢ Larger Oil & gas, manufacturing, telecom companies Advantages ā¢ Often can hire experienced researchers ā¢ Can work on long-term business priorities & big wins Dis- advantages ā¢ Diļ¬cult to get data ā¢ Rarely translates into actual business value, so usually the amount of investment remains small The ML Organization Mountain ML Teams - orgs
- 27. Full Stack Deep Learning - UC Berkeley Spring 2021 ML embedded into business / product teams ML organization archetypes 27 What it looks like ā¢ Certain product teams or business units have ML expertise along-side their software or analytics talent ā¢ ML reports up to the teamās engineering lead or tech lead Example organizations ā¢ Software / technology companies ā¢ Financial services companies Advantages ā¢ ML improvements are likely to lead to business value ā¢ Tight feedback cycle between idea and product improvement Dis- advantages ā¢ Hard to hire and develop top talent ā¢ Access to resources (data / compute) can lag ā¢ ML project cycles conflict with engineering mgmt ā¢ Long-term projects can be hard to justify The ML Organization Mountain ML Teams - orgs
- 28. Full Stack Deep Learning - UC Berkeley Spring 2021 Independent ML Function ML organization archetypes 28 What it looks like ā¢ ML division reporting to senior leadership (often CEO) ā¢ ML PMs work with MLRs, MLEs, and customers to build ML into products ā¢ Teams sometimes publish long-term research Example organizations ā¢ Large financial services companies Advantages ā¢ Talent density allows to hire & train top practitioners ā¢ Senior leaders can marshal data / compute resources ā¢ Can invest in tooling, practices, and culture around ML development Dis- advantages ā¢ Model handoļ¬s to lines of business can be challenging - users need to buy-in and be educated on model use ā¢ Feedback cycles can be slow The ML Organization Mountain ML Teams - orgs
- 29. Full Stack Deep Learning - UC Berkeley Spring 2021 ML-First Organizations ML organization archetypes 29 What it looks like ā¢ CEO buy-in ā¢ ML division working on challenging, long-term projects ā¢ ML expertise in every line of business focusing on quick wins and working with central ML division Example organizations ā¢ Large tech companies ā¢ ML-focused startups Advantages ā¢ Best data access: data thinking permeates the org ā¢ Recruiting: ML team works on hardest problems ā¢ Easiest deployment: product teams understand ML Dis- advantages ā¢ Hard to implement ā¢ Challenging & expensive to recruit enough talent ā¢ Culturally diļ¬cult to embed ML thinking everywhere ML Teams - orgs
- 30. Full Stack Deep Learning - UC Berkeley Spring 2021 ML team structures - design choices 30 Software engineering vs research Data ownership Model ownership Key questions ā¢ To what extent is the ML team responsible for building or integrating with software? ā¢ How important are SWE skills on the team? ā¢ How much control does the ML team have over data collection, warehousing, labeling, and pipelining? ā¢ Is the ML team responsible for deploying models into production? ā¢ Who maintains deployed models? ML Teams - orgs
- 31. Full Stack Deep Learning - UC Berkeley Spring 2021 ML team structures - design choices 31 Software engineering vs research Data ownership Model ownership ā¢ Research prioritized over SWE skills ā¢ Researcher-SWE collaboration lacking ML R&D ā¢ ML team has no control over data ā¢ ML team typically will not have data engineering component ā¢ Models are rarely deployed into production ML Teams - orgs
- 32. Full Stack Deep Learning - UC Berkeley Spring 2021 ML team structures - design choices 32 Software engineering vs research Data ownership Model ownership Embedded ML ā¢ SWE skills prioritized over research skills ā¢ Often, all researchers need strong SWE as everyone expected to deploy ā¢ ML team generally does not own data production / mgmt ā¢ Work with data engineers to build pipelines ā¢ ML engineers own the models that they deploy into production ML Teams - orgs ā¢ Research prioritized over SWE skills ā¢ Researcher-SWE collaboration lacking ML R&D ā¢ ML team has no control over data ā¢ ML team typically will not have data engineering component ā¢ Models are rarely deployed into production
- 33. Full Stack Deep Learning - UC Berkeley Spring 2021 ML team structures - design choices 33 Software engineering vs research Data ownership Model ownership ML Function ā¢ Each team has a strong mix of SWE and research skills ā¢ SWE and researchers work closely together within team ā¢ ML team has a voice in data governance discussions ā¢ ML team has strong internal data engineering function ā¢ ML team hands oļ¬ models to user, but is responsible for maintaining them ML Teams - orgs Embedded ML ā¢ Research prioritized over SWE skills ā¢ Researcher-SWE collaboration lacking ML R&D ā¢ ML team has no control over data ā¢ ML team typically will not have data engineering component ā¢ Models are rarely deployed into production ā¢ SWE skills prioritized over research skills ā¢ Often, all researchers need strong SWE as everyone expected to deploy ā¢ ML team generally does not own data production / mgmt ā¢ Work with data engineers to build pipelines ā¢ ML engineers own the models that they deploy into production
- 34. Full Stack Deep Learning - UC Berkeley Spring 2021 ML team structures - design choices 34 Software engineering vs research Data ownership Model ownership Embedded ML ā¢ Research prioritized over SWE skills ā¢ Researcher-SWE collaboration lacking ML R&D ML Function ML First ā¢ ML team has no control over data ā¢ ML team typically will not have data engineering component ā¢ Models are rarely deployed into production ā¢ SWE skills prioritized over research skills ā¢ Often, all researchers need strong SWE as everyone expected to deploy ā¢ Each team has a strong mix of SWE and research skills ā¢ SWE and researchers work closely together within team ā¢ Diļ¬erent teams are more or less research oriented ā¢ Research teams collaborate closely with SWE teams ā¢ ML team generally does not own data production / mgmt ā¢ Work with data engineers to build pipelines ā¢ ML engineers own the models that they deploy into production ā¢ ML team has a voice in data governance discussions ā¢ ML team has strong internal data engineering function ā¢ ML team hands oļ¬ models to user, but is responsible for maintaining them ā¢ ML team often owns company-wide data infrastructure ā¢ ML team hands oļ¬ models to user, who operates and maintains them ML Teams - orgs
- 35. Full Stack Deep Learning - UC Berkeley Spring 2021 ML team structures - design choices 35 Software engineering vs research Data ownership Model ownership Embedded ML ā¢ Research prioritized over SWE skills ā¢ Researcher-SWE collaboration lacking ML R&D ML Function ML First ā¢ ML team has no control over data ā¢ ML team typically will not have data engineering component ā¢ Models are rarely deployed into production ā¢ SWE skills prioritized over research skills ā¢ Often, all researchers need strong SWE as everyone expected to deploy ā¢ Each team has a strong mix of SWE and research skills ā¢ SWE and researchers work closely together within team ā¢ Diļ¬erent teams are more or less research oriented ā¢ Research teams collaborate closely with SWE teams ā¢ ML team generally does not own data production / mgmt ā¢ Work with data engineers to build pipelines ā¢ ML engineers own the models that they deploy into production ā¢ ML team has a voice in data governance discussions ā¢ ML team has strong internal data engineering function ā¢ ML team hands oļ¬ models to user, but is responsible for maintaining them ā¢ ML team often owns company-wide data infrastructure ā¢ ML team hands oļ¬ models to user, who operates and maintains them ML Teams - orgs
- 36. Full Stack Deep Learning - UC Berkeley Spring 2021 Questions? 36 ML Teams - orgs
- 37. Full Stack Deep Learning - UC Berkeley Spring 2021 Managing Module overview 37 Hiring ā¢ How to manage a ML team ā¢ How to hire ML engineers. How to get hired. Orgs ā¢ How ML teams are organized and how they fit into the broader organization Roles ā¢ ML-related roles and the skills they require ML Teams - managing
- 38. Full Stack Deep Learning - UC Berkeley Spring 2021 Managing ML teams is challenging ā¢ Itās hard to tell in advance how hard or easy something is 38 ML Teams - managing
- 39. Full Stack Deep Learning - UC Berkeley Spring 2021 Managing ML teams is challenging 39 https://medium.com/@l2k/why-are-machine-learning-projects-so-hard-to-manage-8e9b9cf49641 Itās hard to tell in advance how easy or hard something is ML Teams - managing
- 40. Full Stack Deep Learning - UC Berkeley Spring 2021 Managing ML teams is challenging ā¢ Itās hard to tell in advance how hard or easy something is ā¢ ML progress is nonlinear ā¢ Very common for projects to stall for weeks or longer ā¢ In early stages, diļ¬cult to plan project because unclear what will work ā¢ As a result, estimating project timelines is extremely diļ¬cult ā¢ I.e., production ML is still somewhere between āresearchā and āengineeringā 40 ML Teams - managing
- 41. Full Stack Deep Learning - UC Berkeley Spring 2021 Managing ML teams is challenging ā¢ Itās hard to tell in advance how hard or easy something is ā¢ ML progress is nonlinear ā¢ There are cultural gaps between research and engineering ā¢ Diļ¬erent values, backgrounds, goals, norms ā¢ In toxic cultures, the two sides often donāt value one another 41 ML Teams - managing
- 42. Full Stack Deep Learning - UC Berkeley Spring 2021 Managing ML teams is challenging ā¢ Itās hard to tell in advance how hard or easy something is ā¢ ML progress is nonlinear ā¢ There are cultural gaps between research and engineering ā¢ Leaders often donāt understand it 42 ML Teams - managing
- 43. Full Stack Deep Learning - UC Berkeley Spring 2021 How to manage ML teams better ā¢ Do ML Project planning probabilistically 43 ML Teams - managing
- 44. Full Stack Deep Learning - UC Berkeley Spring 2021 How to manage ML teams better ā¢ Do ML project planning probabilistically ā¢ From:āØ āØ āØ āØ āØ ā¢ To:āØ āØ 44 ML Teams - managing Task A Task C Task D Task E Task F Task G Week 1 Week 2 Week 3 Week 4
- 45. Full Stack Deep Learning - UC Berkeley Spring 2021 How to manage ML teams better ā¢ Do ML project planning probabilistically ā¢ From:āØ āØ āØ āØ āØ ā¢ To:āØ āØ 45 ML Teams - managing Task A Task C Task D Task E Task F Task G Week 1 Week 2 Week 3 Week 4 Task A (50%) Task B (25%) Task C (50%) Week 1 Week 2 Week 3 Week 4 Task D (75%)
- 46. Full Stack Deep Learning - UC Berkeley Spring 2021 How to manage ML teams better ā¢ Do ML project planning probabilistically ā¢ From:āØ āØ āØ āØ āØ ā¢ To:āØ āØ 46 ML Teams - managing Task A Task C Task D Task E Task F Task G Week 1 Week 2 Week 3 Week 4 Task A (50%) Task B (25%) Task C (50%) Week 1 Week 2 Week 3 Week 4 Task D (75%)
- 47. Full Stack Deep Learning - UC Berkeley Spring 2021 How to manage ML teams better ā¢ Do ML project planning probabilistically ā¢ From:āØ āØ āØ āØ āØ ā¢ To:āØ āØ 47 ML Teams - managing Task A Task C Task D Task E Task F Task G Week 1 Week 2 Week 3 Week 4 Task A Task B Task C Week 1 Week 2 Week 3 Week 4 Task D
- 48. Full Stack Deep Learning - UC Berkeley Spring 2021 How to manage ML teams better ā¢ Do ML project planning probabilistically ā¢ From:āØ āØ āØ āØ āØ ā¢ To:āØ āØ 48 ML Teams - managing Task A Task C Task D Task E Task F Task G Week 1 Week 2 Week 3 Week 4 Task A Task B Task C Week 1 Week 2 Week 3 Week 4 Task E (50%) Task F (50%) Task G (10%) Task D (75%)
- 49. Full Stack Deep Learning - UC Berkeley Spring 2021 How to manage ML teams better ā¢ Do ML Project planning probabilistically ā¢ Attempt a portfolio of approaches ā¢ Measure progress based on inputs, not results ā¢ Have researchers and engineers work together ā¢ Get end-to-end pipelines together quickly to demonstrate quick wins ā¢ Educate leadership on ML timeline uncertainty 49 ML Teams - managing
- 50. Full Stack Deep Learning - UC Berkeley Spring 2021 Resources for educating execs ā¢ https://a16z.com/2016/06/10/ai-deep-learning-machines/ ā¢ Pieterās upcoming AI Strategy class:āØ https://emeritus-executive.berkeley.edu/artificial-intelligence/ 50 ML Teams - managing
- 51. Full Stack Deep Learning - UC Berkeley Spring 2021 Questions? 51 ML Teams - managing
- 52. Full Stack Deep Learning - UC Berkeley Spring 2021 Managing Module overview 52 Hiring ā¢ How to manage a ML team ā¢ How to hire ML engineers. How to get hired. Orgs ā¢ How ML teams are organized and how they fit into the broader organization Roles ā¢ ML-related roles and the skills they require ML Teams - hiring
- 53. Full Stack Deep Learning - UC Berkeley Spring 2021 Hiring for ML - outline ā¢ The AI Talent Gap ā¢ Sourcing ā¢ Interviewing ā¢ Finding a job 53 ML Teams - hiring
- 54. Full Stack Deep Learning - UC Berkeley Spring 2021 Hiring for ML - outline ā¢ The AI Talent Gap ā¢ Sourcing ā¢ Interviewing ā¢ Finding a job 54 ML Teams - hiring
- 55. Full Stack Deep Learning - UC Berkeley Spring 2021 The AI Talent Gap 55 How many people know how to build AI systems? 5,000 (actively publishing research [Element AI]) 10,000 (estimated num people with the right skillset [Element AI]) 22,000 (PhD-educated AI researchers [Bloomberg]) 90,000 (upper bound on number of people [Element AI]) 200,000 - 300,000 (Number of AI researcher / practitioners [Tencent]) 3.6M (Number of software developers in the US) 18.2M (Number of software developers in the world) Sources: The AI Talent Shortage (Nikolai Yakovenko) https://medium.com/@Moscow25/the-ai-talent-shortage-704d8cf0c4cc Just How Shallow is the Artificial Intelligence Talent Pool (Jeremy Kahn) āØ https://www.bloomberg.com/news/articles/2018-02-07/just-how-shallow-is-the-artificial-intelligence-talent-pool ML Teams - hiring
- 56. Full Stack Deep Learning - UC Berkeley Spring 2021 56 The AI talent gap Fierce competition for AI talent āEveryone agrees that the competition to hire people whoĀ know how to build artificial intelligence systems is intense.Ā ItāsĀ turned once- staid academic conferences intoĀ frenziedĀ meet markets for corporate recruiters and driven theĀ salariesĀ of the top researchers to seven- figures.ā āØ (Bloomberg) Sources: The AI Talent Shortage (Nikolai Yakovenko) https://medium.com/@Moscow25/the-ai-talent-shortage-704d8cf0c4cc Just How Shallow is the Artificial Intelligence Talent Pool (Jeremy Kahn) āØ https://www.bloomberg.com/news/articles/2018-02-07/just-how-shallow-is-the-artificial-intelligence-talent-pool ML Teams - hiring
- 57. Full Stack Deep Learning - UC Berkeley Spring 2021 57 The AI talent gap Fierce competition for AI talent āHiring is crazy right now. ML is a young field that got popular very quickly. Thereās a ton of demand and not a lot of supply.ā āØ (Computer Vision Engineer at Series C startup) ML Teams - hiring
- 58. Full Stack Deep Learning - UC Berkeley Spring 2021 58 The AI talent gap Fierce competition for AI talent āHiring for ML is really challenging and takes way more time and eļ¬ort than we expected. We have someone working on it full-time and weāre still only able to get a few people per quarterā (Startup Founder) ML Teams - hiring
- 59. Full Stack Deep Learning - UC Berkeley Spring 2021 Hiring for ML - outline ā¢ The AI Talent Gap ā¢ Sourcing ā¢ Interviewing ā¢ Finding a job 59 ML Teams - hiring
- 60. Full Stack Deep Learning - UC Berkeley Spring 2021 Most common ML roles ā¢ ML product manager ā¢ DevOps ā¢ Data engineer ā¢ ML engineer ā¢ ML researcher / ML scientist ā¢ Data scientist 60 ML Teams - hiring
- 61. Full Stack Deep Learning - UC Berkeley Spring 2021 Most common ML roles ā¢ ML product manager ā¢ DevOps ā¢ Data engineer ā¢ ML engineer ā¢ ML researcher / ML scientist ā¢ Data scientist 61 Slightly diļ¬erent mindset required. āØ Helpful to look for demonstrated interest in AI - courses, conferences, re-implementations, etc ML Teams - hiring
- 62. Full Stack Deep Learning - UC Berkeley Spring 2021 Most common ML roles ā¢ ML product manager ā¢ DevOps ā¢ Data engineer ā¢ ML engineer ā¢ ML researcher / ML scientist ā¢ Data scientist 62 Our focus ML Teams - hiring
- 63. Full Stack Deep Learning - UC Berkeley Spring 2021 How to hire MLEs - the wrong way ā¢ Job Description (Unicorn Machine Learning Engineer) ā¢ Duties ā¢ Keep up with the state of the art ā¢ Implement models from scratch ā¢ Deep understanding of mathematics & ability to come up with new models ā¢ Build tooling & infrastructure for the ML team ā¢ Build data pipelines for the ML team ā¢ Deploy & monitor models into production ā¢ Requirements ā¢ PhD ā¢ At least 4 years tensorflow experience ā¢ At least 4 years as a software engineer ā¢ Publications in top ML conference ā¢ Experience building large-scale distributed systems 63 ML Teams - hiring
- 64. Full Stack Deep Learning - UC Berkeley Spring 2021 How to hire MLEs - the right way ā¢ Hire for software engineering skills, interest in ML, and desire to learn. Train to do ML. ā¢ Go more junior. Most undergrad computer science students graduate with ML experience. ā¢ Be more specific about what you need. Not every ML engineer needs to do DevOps. 64 ML Teams - hiring
- 65. Full Stack Deep Learning - UC Berkeley Spring 2021 How to hire MLRs ā¢ Look for quality of publications, not quantity (e.g., originality of ideas, quality of execution) ā¢ Look for researchers with an eye for working on important problems (many researchers focus on trendy problems without considering why they matter) ā¢ Look for researchers with experience outside of academia ā¢ Consider hiring talented people from adjacent fields (physics, statistics, math) ā¢ Consider hiring people without PhDs (e.g., talented undergraduate / masters students, graduates of Google/Facebook/OpenAI fellowship programs, dedicated self-studiers) 65 ML Teams - hiring
- 66. Full Stack Deep Learning - UC Berkeley Spring 2021 How to find MLE/MLR candidates ā¢ Standard sources: LinkedIn, recruiters, on-campus recruiting, etc ā¢ Monitor arXiv and top conferences and flag first authors of papers you like ā¢ Look for good reimplementations of papers you like ā¢ Attend ML research conferences (NeurIPS, ICLR, ICML) 66 ML Teams - hiring
- 67. Full Stack Deep Learning - UC Berkeley Spring 2021 How to attract MLR / MLE candidates 67 What do machine learning practitioners want? How to make your company stand out? ā¢ Work with cutting edge tools & techniques ā¢ Build skills / knowledge in an exciting field ā¢ Work with excellent people ā¢ Work on interesting datasets ā¢ Do work that matters ā¢ Work on research-oriented projects. Publicize them. Invest in tooling for your team & empower employees to try new tools. ā¢ Build team culture around learning (reading groups, learning days, professional development budget, conference budget) ā¢ Hire high-profile people. Help your best people build their profile through publishing blogs & papers. ā¢ Sell the uniqueness of your dataset in recruiting materials. ā¢ Sell the mission of your company and potential impact of machine learning on that mission. Work on projects that have a tangible impact today. ML Teams - hiring
- 68. Full Stack Deep Learning - UC Berkeley Spring 2021 Hiring for ML - outline ā¢ The AI Talent Gap ā¢ Sourcing ā¢ Interviewing ā¢ Finding a job 68 ML Teams - hiring
- 69. Full Stack Deep Learning - UC Berkeley Spring 2021 What to test in an ML interview? ā¢ Hire for strengths ā¢ Meet a minimum bar for everything else 69 ML Teams - hiring
- 70. Full Stack Deep Learning - UC Berkeley Spring 2021 What to test in an ML interview? ā¢ Validate your hypotheses of candidateās strengths ā¢ Researchers: make sure they can think creatively about new ML problems, probe how thoughtful they were about previous projects ā¢ Engineers: make sure they are great generalist SWEs ā¢ Make sure candidates meet a minimum bar on weaker areas ā¢ Researchers: test SWE knowledge and ability to write good code ā¢ SWEs: test ML knowledge 70 ML Teams - hiring
- 71. Full Stack Deep Learning - UC Berkeley Spring 2021 What happens in a ML interview? ā¢ Much less well-defined than software engineering interviews ā¢ Common types of assessments: ā¢ Background & culture fit ā¢ Whiteboard coding (similar to SWE interviews) ā¢ Pair coding (similar to SWE interviews) ā¢ Pair debugging (often ML-specific code) ā¢ Math puzzles (e.g., involving linear algebra) ā¢ Take-home ML project ā¢ Applied ML (e.g., explain how youād solve this problem with ML) ā¢ Previous ML projects (e.g., probing on what you tried, why things did / didnāt work) ā¢ ML theory (e.g., bias-variance tradeoļ¬, overfitting, underfitting, understanding of specific algorithms) 71 ML Teams - hiring
- 72. Full Stack Deep Learning - UC Berkeley Spring 2021 Hiring for ML - outline ā¢ The AI Talent Gap ā¢ Sourcing ā¢ Interviewing ā¢ Finding a job 72 ML Teams - hiring
- 73. Full Stack Deep Learning - UC Berkeley Spring 2021 Where to look for a ML job? ā¢ Standard sources: LinkedIn, recruiters, on-campus recruiting, etc ā¢ ML research conferences (NeurIPS, ICLR, ICML) ā¢ Apply directly (remember, thereās a talent gap!) ā¢ This course ā¢ Those who pass the exam will get access to our recruiting database 73 ML Teams - hiring
- 74. Full Stack Deep Learning - UC Berkeley Spring 2021 How to stand out for ML roles? ā¢ Build software engineering skills (e.g., work at a well-known software company) ā¢ Exhibit interest in ML (e.g., conference attendance, online courses taken) ā¢ Show you have broad knowledge of ML (e.g., write blog posts synthesizing a research area) ā¢ Demonstrate ability to get ML projects done (e.g., create side projects, re- implement papers) ā¢ Prove you can think creatively in ML (e.g., win Kaggle competitions, publish papers) 74 More impressive ML Teams - hiring
- 75. Full Stack Deep Learning - UC Berkeley Spring 2021 How to prepare for the interview? ā¢ Prepare for a general SWE interview (e.g., āCracking the Coding Interviewā) ā¢ Prepare to talk in detail about your past ML projects (remember details, prepare to talk about tradeoļ¬s and decisions you made) ā¢ Review how basic ML algorithms work (linear / logistic regression, nearest neighbor, decision trees, k-means, MLPs, ConvNets, recurrent nets, etc) ā¢ Review ML theory ā¢ Think about the problems the company youāre interviewing with may face and what ML techniques may apply to them 75 ML Teams - hiring
- 76. Full Stack Deep Learning - UC Berkeley Spring 2021 Conclusion 76 Orgs ā¢ ML teams are becoming more standalone, hence more interdisciplinary Roles ā¢ Lots of diļ¬erent skills involved in production ML, so thereās an opportunity for many to contribute Hiring ā¢ Talent is scarce, so be specific about what is must-have. It can be hard to break in as an outsider - use projects to build awareness. Managing ā¢ Managing ML teams is hard. Thereās no silver bullet, but shifting toward probabilistic planning can help ML Teams - hiring
- 77. Full Stack Deep Learning - UC Berkeley Spring 2021 Thank you! 77