Sergey Karayev, profile picture
Uploaded bySergey Karayev
24,564 views

Machine Learning Teams - Full Stack Deep Learning

The document discusses the complexities of managing machine learning (ML) teams, emphasizing the need for effective hiring, organizational structure, and project management methodologies. It outlines various roles within ML teams, such as ML product managers, engineers, and researchers, along with the necessary skills for each role. Additionally, it presents best practices for structuring ML organizations based on maturity levels and highlights the challenges in communicating and aligning expectations within technical and leadership domains.

Technology◦
Related topics:
Deep Learning•

Embed presentation

Downloaded 346 times
Full Stack Deep Learning Josh Tobin, Sergey Karayev, Pieter Abbeel Machine Learning Teams
Full Stack Deep Learning Running ML teams is hard 2ML Teams - overview Running any technical team is hard… • ML talent is expensive and scarce • ML teams have a diverse set of roles • Projects have unclear timelines and high uncertainty • The field is moving fast and ML is the “high- interest credit card of technical debt” • Leadership often doesn’t understand AI • Hiring great people • Managing and developing those people • Managing your team’s output and making sure your vectors are aligned • Making good long-term technical choices & managing technical debt • Managing expectations from leadership … And ML adds complexity
Full Stack Deep Learning Goal of this module • Give you some insight into how to think about building and managing ML teams • Help you get a job in ML 3ML Teams - overview
Full Stack Deep Learning Managing Module overview 4 Hiring • How to manage a ML team • How to hire ML engineers. How to get hired. Exam for the course ML Teams - overview Orgs • How ML teams are organized and how they fit into the broader organization Roles • ML-related roles and the skills they require
Full Stack Deep Learning Managing Module overview 5 Hiring • How to manage a ML team • How to hire ML engineers. How to get hired. Exam for the course Orgs • How ML teams are organized and how they fit into the broader organization Roles • ML-related roles and the skills they require ML Teams - roles
Full Stack Deep Learning Most common ML roles • ML product manager • DevOps • Data engineer • ML engineer • ML researcher / ML scientist • Data scientist 6ML Teams - roles
Full Stack Deep Learning Most common ML roles • ML product manager • DevOps • Data engineer • ML engineer • ML researcher / ML scientist • Data scientist 7 What’s the difference? ML Teams - roles
Full Stack Deep Learning Breakdown of job function by role 8 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc ML Teams - roles
Full Stack Deep Learning Breakdown of job function by role 9 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc DevOps engineer Deploy & monitor production systems Deployed product AWS, etc. ML Teams - roles
Full Stack Deep Learning Breakdown of job function by role 10 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc DevOps engineer Deploy & monitor production systems Deployed product AWS, etc. Data engineer Build data pipelines, aggregation, storage, monitoring Distributed system Hadoop, Kafka, Airflow ML Teams - roles
Full Stack Deep Learning Breakdown of job function by role 11 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc DevOps engineer Deploy & monitor production systems Deployed product AWS, etc. Data engineer Build data pipelines, aggregation, storage, monitoring Distributed system Hadoop, Kafka, Airflow ML engineer Train & deploy prediction models Prediction system running on real data (often in production) Tensorflow, Docker ML Teams - roles
Full Stack Deep Learning Breakdown of job function by role 12 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc DevOps engineer Deploy & monitor production systems Deployed product AWS, etc. Data engineer Build data pipelines, aggregation, storage, monitoring Distributed system Hadoop, Kafka, Airflow ML engineer Train & deploy prediction models Prediction system running on real data (often in production) Tensorflow, Docker ML researcher Train prediction models (often forward looking or not production-critical) Prediction model & report describing it Tensorflow, pytorch, Jupyter ML Teams - roles
Full Stack Deep Learning Breakdown of job function by role 13 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc DevOps engineer Deploy & monitor production systems Deployed product AWS, etc. Data engineer Build data pipelines, aggregation, storage, monitoring Distributed system Hadoop, Kafka, Airflow ML engineer Train & deploy prediction models Prediction system running on real data (often in production) Tensorflow, Docker ML researcher Train prediction models (often forward looking or not production-critical) Prediction model & report describing it Tensorflow, pytorch, Jupyter Data scientist Blanket term used to describe all of the above. In some orgs, means answering business questions using analytics Prediction model or report SQL, Excel, Jupyter, Pandas, SKLearn, Tensorflow ML Teams - roles
Full Stack Deep Learning What skills are needed for the roles? 14 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM
Full Stack Deep Learning What skills are needed for the roles? 15 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM Primarily a software engineering role. Often from standard SWE pipeline.
Full Stack Deep Learning What skills are needed for the roles? 16 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM SWE with ML team as an active customer
Full Stack Deep Learning What skills are needed for the roles? 17 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM Rare mix of ML skills and SWE skills. Often SWEs with significant self- teaching or science / engineering PhDs who worked as traditional SWEs after grad school
Full Stack Deep Learning What skills are needed for the roles? 18 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM ML experts. Usually have MS/PhD in CS or Stats or did an industrial fellowship program
Full Stack Deep Learning What skills are needed for the roles? 19 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM Wide range of backgrounds from undergrad-only to science PhD
Full Stack Deep Learning What skills are needed for the roles? 20 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM Traditional PMs, but with a deep understanding of the ML development process & mindset
Full Stack Deep Learning Questions? 21ML Teams - roles
Full Stack Deep Learning Managing Module overview 22 Hiring • How to manage a ML team • How to hire ML engineers. How to get hired. Exam for the course Orgs • How ML teams are organized and how they fit into the broader organization Roles • ML-related roles and the skills they require ML Teams - orgs
Full Stack Deep Learning ML org structures - lessons learned • No consensus yet on the right way to structure a ML team • This lecture: taxonomy of best practices for different organizational maturity levels 23ML Teams - orgs
Full Stack Deep Learning ML organization archetypes 24 The ML Organization Mountain ML Teams - orgs
Full Stack Deep Learning ML organization archetypes 25 The ML Organization Mountain Nascent / Ad-Hoc ML What it looks like • No one is doing ML, or ML is done on an ad-hoc basis • Little ML expertise in-house Example organizations • Most small-medium businesses • Less technology-forward large companies (education, logistics, etc) Advantages • Often low-hanging fruit for ML Dis- advantages • Little support for ML projects, difficult to hire and retain good talent ML Teams - orgs
Full Stack Deep Learning ML R&D ML organization archetypes 26 What it looks like • ML efforts are centered in the R&D arm of the organization • Often hire researchers / PhDs & write papers Example organizations • Larger Oil & gas, manufacturing, telecom companies Advantages • Often can hire experienced researchers • Can work on long-term business priorities & big wins Dis- advantages • Difficult to get data • Rarely translates into actual business value, so usually the amount of investment remains small The ML Organization Mountain ML Teams - orgs
Full Stack Deep Learning ML embedded into business / product teams ML organization archetypes 27 What it looks like • Certain product teams or business units have ML expertise along-side their software or analytics talent • ML reports up to the team’s engineering lead or tech lead Example organizations • Software / technology companies • Financial services companies Advantages • ML improvements are likely to lead to business value • Tight feedback cycle between idea and product improvement Dis- advantages • Hard to hire and develop top talent • Access to resources (data / compute) can lag • ML project cycles conflict with engineering mgmt • Long-term projects can be hard to justify The ML Organization Mountain ML Teams - orgs
Full Stack Deep Learning Independent ML Function ML organization archetypes 28 What it looks like • ML division reporting to senior leadership (often CEO) • ML PMs work with MLRs, MLEs, and customers to build ML into products • Teams sometimes publish long-term research Example organizations • Large financial services companies Advantages • Talent density allows to hire & train top practitioners • Senior leaders can marshal data / compute resources • Can invest in tooling, practices, and culture around ML development Dis- advantages • Model handoffs to lines of business can be challenging - users need to buy-in and be educated on model use • Feedback cycles can be slow The ML Organization Mountain ML Teams - orgs
Full Stack Deep Learning ML-First Organizations ML organization archetypes 29 What it looks like • CEO buy-in • ML division working on challenging, long-term projects • ML expertise in every line of business focusing on quick wins and working with central ML division Example organizations • Large tech companies • ML-focused startups Advantages • Best data access: data thinking permeates the org • Recruiting: ML team works on hardest problems • Easiest deployment: product teams understand ML Dis- advantages • Hard to implement • Challenging & expensive to recruit enough talent • Culturally difficult to embed ML thinking everywhere ML Teams - orgs
Full Stack Deep Learning ML team structures - design choices 30 Software engineering vs research Data ownership Model ownership Key questions • To what extent is the ML team responsible for building or integrating with software? • How important are SWE skills on the team? • How much control does the ML team have over data collection, warehousing, labeling, and pipelining? • Is the ML team responsible for deploying models into production? • Who maintains deployed models? ML Teams - orgs
Full Stack Deep Learning ML team structures - design choices 31 Software engineering vs research Data ownership Model ownership • Research prioritized over SWE skills • Researcher-SWE collaboration lacking ML R&D • ML team has no control over data • ML team typically will not have data engineering component • Models are rarely deployed into production ML Teams - orgs
Full Stack Deep Learning ML team structures - design choices 32 Software engineering vs research Data ownership Model ownership Embedded ML • SWE skills prioritized over research skills • Often, all researchers need strong SWE as everyone expected to deploy • ML team generally does not own data production / mgmt • Work with data engineers to build pipelines • ML engineers own the models that they deploy into production ML Teams - orgs • Research prioritized over SWE skills • Researcher-SWE collaboration lacking ML R&D • ML team has no control over data • ML team typically will not have data engineering component • Models are rarely deployed into production
Full Stack Deep Learning ML team structures - design choices 33 Software engineering vs research Data ownership Model ownership ML Function • Each team has a strong mix of SWE and research skills • SWE and researchers work closely together within team • ML team has a voice in data governance discussions • ML team has strong internal data engineering function • ML team hands off models to user, but is responsible for maintaining them ML Teams - orgs Embedded ML • Research prioritized over SWE skills • Researcher-SWE collaboration lacking ML R&D • ML team has no control over data • ML team typically will not have data engineering component • Models are rarely deployed into production • SWE skills prioritized over research skills • Often, all researchers need strong SWE as everyone expected to deploy • ML team generally does not own data production / mgmt • Work with data engineers to build pipelines • ML engineers own the models that they deploy into production
Full Stack Deep Learning ML team structures - design choices 34 Software engineering vs research Data ownership Model ownership Embedded ML • Research prioritized over SWE skills • Researcher-SWE collaboration lacking ML R&D ML Function ML First • ML team has no control over data • ML team typically will not have data engineering component • Models are rarely deployed into production • SWE skills prioritized over research skills • Often, all researchers need strong SWE as everyone expected to deploy • Each team has a strong mix of SWE and research skills • SWE and researchers work closely together within team • Different teams are more or less research oriented • Research teams collaborate closely with SWE teams • ML team generally does not own data production / mgmt • Work with data engineers to build pipelines • ML engineers own the models that they deploy into production • ML team has a voice in data governance discussions • ML team has strong internal data engineering function • ML team hands off models to user, but is responsible for maintaining them • ML team often owns company-wide data infrastructure • ML team hands off models to user, who operates and maintains them ML Teams - orgs
Full Stack Deep Learning ML team structures - design choices 35 Software engineering vs research Data ownership Model ownership Embedded ML • Research prioritized over SWE skills • Researcher-SWE collaboration lacking ML R&D ML Function ML First • ML team has no control over data • ML team typically will not have data engineering component • Models are rarely deployed into production • SWE skills prioritized over research skills • Often, all researchers need strong SWE as everyone expected to deploy • Each team has a strong mix of SWE and research skills • SWE and researchers work closely together within team • Different teams are more or less research oriented • Research teams collaborate closely with SWE teams • ML team generally does not own data production / mgmt • Work with data engineers to build pipelines • ML engineers own the models that they deploy into production • ML team has a voice in data governance discussions • ML team has strong internal data engineering function • ML team hands off models to user, but is responsible for maintaining them • ML team often owns company-wide data infrastructure • ML team hands off models to user, who operates and maintains them ML Teams - orgs
Full Stack Deep Learning Questions? 36ML Teams - orgs
Full Stack Deep Learning Managing Module overview 37 Hiring • How to manage a ML team • How to hire ML engineers. How to get hired. Exam for the course Orgs • How ML teams are organized and how they fit into the broader organization Roles • ML-related roles and the skills they require ML Teams - managing
Full Stack Deep Learning Managing ML teams is challenging • It’s hard to tell in advance how hard or easy something is 38ML Teams - managing
Full Stack Deep Learning Managing ML teams is challenging 39 https://medium.com/@l2k/why-are-machine-learning-projects-so-hard-to-manage-8e9b9cf49641 It’s hard to tell in advance how easy or hard something is ML Teams - managing
Full Stack Deep Learning Managing ML teams is challenging • It’s hard to tell in advance how hard or easy something is • ML progress is nonlinear • Very common for projects to stall for weeks or longer • In early stages, difficult to plan project because unclear what will work • As a result, estimating project timelines is extremely difficult • I.e., production ML is still somewhere between “research” and “engineering” 40ML Teams - managing
Full Stack Deep Learning Managing ML teams is challenging • It’s hard to tell in advance how hard or easy something is • ML progress is nonlinear • There are cultural gaps between research and engineering • Different values, backgrounds, goals, norms • In toxic cultures, the two sides often don’t value one another 41ML Teams - managing
Full Stack Deep Learning Managing ML teams is challenging • It’s hard to tell in advance how hard or easy something is • ML progress is nonlinear • There are cultural gaps between research and engineering • Leaders often don’t understand it 42ML Teams - managing
Full Stack Deep Learning How to manage ML teams better • Do ML Project planning probabilistically 43ML Teams - managing
Full Stack Deep Learning How to manage ML teams better • Do ML project planning probabilistically • From:
 
 
 
 
 • To:
 
 44ML Teams - managing Task A Task C Task D Task E Task F Task G Week 1 Week 2 Week 3 Week 4
Full Stack Deep Learning How to manage ML teams better • Do ML project planning probabilistically • From:
 
 
 
 
 • To:
 
 45ML Teams - managing Task A Task C Task D Task E Task F Task G Week 1 Week 2 Week 3 Week 4 Task A (50%) Task B (25%) Task C (50%) Week 1 Week 2 Week 3 Week 4 Task D (75%)
Full Stack Deep Learning How to manage ML teams better • Do ML project planning probabilistically • From:
 
 
 
 
 • To:
 
 46ML Teams - managing Task A Task C Task D Task E Task F Task G Week 1 Week 2 Week 3 Week 4 Task A (50%) Task B (25%) Task C (50%) Week 1 Week 2 Week 3 Week 4 Task D (75%)
Full Stack Deep Learning How to manage ML teams better • Do ML project planning probabilistically • From:
 
 
 
 
 • To:
 
 47ML Teams - managing Task A Task C Task D Task E Task F Task G Week 1 Week 2 Week 3 Week 4 Task A Task B Task C Week 1 Week 2 Week 3 Week 4 Task D
Full Stack Deep Learning How to manage ML teams better • Do ML project planning probabilistically • From:
 
 
 
 
 • To:
 
 48ML Teams - managing Task A Task C Task D Task E Task F Task G Week 1 Week 2 Week 3 Week 4 Task A Task B Task C Week 1 Week 2 Week 3 Week 4 Task E (50%) Task F (50%) Task G (10%) Task D (75%)
Full Stack Deep Learning How to manage ML teams better • Do ML Project planning probabilistically • Attempt a portfolio of approaches • Measure progress based on inputs, not results • Have researchers and engineers work together • Get end-to-end pipelines together quickly to demonstrate quick wins • Educate leadership on ML timeline uncertainty 49ML Teams - managing
Full Stack Deep Learning Resources for educating execs • https://a16z.com/2016/06/10/ai-deep-learning-machines/ • Pieter’s upcoming AI Strategy class:
 https://emeritus-executive.berkeley.edu/artificial-intelligence/ 50ML Teams - managing
Full Stack Deep Learning Questions? 51ML Teams - managing
Full Stack Deep Learning Managing Module overview 52 Hiring • How to manage a ML team • How to hire ML engineers. How to get hired. Exam for the course Orgs • How ML teams are organized and how they fit into the broader organization Roles • ML-related roles and the skills they require ML Teams - hiring
Full Stack Deep Learning Hiring for ML - outline • The AI Talent Gap • Sourcing • Interviewing • Finding a job • Course exam 53ML Teams - hiring
Full Stack Deep Learning Hiring for ML - outline • The AI Talent Gap • Sourcing • Interviewing • Finding a job • Course exam 54ML Teams - hiring
Full Stack Deep Learning The AI Talent Gap 55 How many people know how to build AI systems? 5,000 (actively publishing research [Element AI]) 10,000 (estimated num people with the right skillset [Element AI]) 22,000 (PhD-educated AI researchers [Bloomberg]) 90,000 (upper bound on number of people [Element AI]) 200,000 - 300,000 (Number of AI researcher / practitioners [Tencent]) 3.6M (Number of software developers in the US) 18.2M (Number of software developers in the world) Sources: The AI Talent Shortage (Nikolai Yakovenko) https://medium.com/@Moscow25/the-ai-talent-shortage-704d8cf0c4cc Just How Shallow is the Artificial Intelligence Talent Pool (Jeremy Kahn) 
 https://www.bloomberg.com/news/articles/2018-02-07/just-how-shallow-is-the-artificial-intelligence-talent-pool ML Teams - hiring
Full Stack Deep Learning 56 The AI talent gap Fierce competition for AI talent “Everyone agrees that the competition to hire people who know how to build artificial intelligence systems is intense. It’s  turned once- staid academic conferences into frenzied meet markets for corporate recruiters and driven the salaries of the top researchers to seven- figures.” 
 (Bloomberg) Sources: The AI Talent Shortage (Nikolai Yakovenko) https://medium.com/@Moscow25/the-ai-talent-shortage-704d8cf0c4cc Just How Shallow is the Artificial Intelligence Talent Pool (Jeremy Kahn) 
 https://www.bloomberg.com/news/articles/2018-02-07/just-how-shallow-is-the-artificial-intelligence-talent-pool ML Teams - hiring
Full Stack Deep Learning 57 The AI talent gap Fierce competition for AI talent “Hiring is crazy right now. ML is a young field that got popular very quickly. There’s a ton of demand and not a lot of supply.” 
 (Computer Vision Engineer at Series C startup) ML Teams - hiring
Full Stack Deep Learning 58 The AI talent gap Fierce competition for AI talent “Hiring for ML is really challenging and takes way more time and effort than we expected. We have someone working on it full-time and we’re still only able to get a few people per quarter” (Startup Founder) ML Teams - hiring
Full Stack Deep Learning Hiring for ML - outline • The AI Talent Gap • Sourcing • Interviewing • Finding a job • Course exam 59ML Teams - hiring
Full Stack Deep Learning Most common ML roles • ML product manager • DevOps • Data engineer • ML engineer • ML researcher / ML scientist • Data scientist 60ML Teams - hiring
Full Stack Deep Learning Most common ML roles • ML product manager • DevOps • Data engineer • ML engineer • ML researcher / ML scientist • Data scientist 61 Slightly different mindset required. 
 Helpful to look for demonstrated interest in AI - courses, conferences, re-implementations, etc ML Teams - hiring
Full Stack Deep Learning Most common ML roles • ML product manager • DevOps • Data engineer • ML engineer • ML researcher / ML scientist • Data scientist 62 Our focus ML Teams - hiring
Full Stack Deep Learning How to hire MLEs - the wrong way • Job Description (Unicorn Machine Learning Engineer) • Duties • Keep up with the state of the art • Implement models from scratch • Deep understanding of mathematics & ability to come up with new models • Build tooling & infrastructure for the ML team • Build data pipelines for the ML team • Deploy & monitor models into production • Requirements • PhD • At least 4 years tensorflow experience • At least 4 years as a software engineer • Publications in top ML conference • Experience building large-scale distributed systems 63ML Teams - hiring
Full Stack Deep Learning How to hire MLEs - the right way • Hire for software engineering skills, interest in ML, and desire to learn. Train to do ML. • Go more junior. Most undergrad computer science students graduate with ML experience. • Be more specific about what you need. Not every ML engineer needs to do DevOps. 64ML Teams - hiring
Full Stack Deep Learning How to hire MLRs • Look for quality of publications, not quantity (e.g., originality of ideas, quality of execution) • Look for researchers with an eye for working on important problems (many researchers focus on trendy problems without considering why they matter) • Look for researchers with experience outside of academia • Consider hiring talented people from adjacent fields (physics, statistics, math) • Consider hiring people without PhDs (e.g., talented undergraduate / masters students, graduates of Google/Facebook/OpenAI fellowship programs, dedicated self-studiers) 65ML Teams - hiring
Full Stack Deep Learning How to find MLE/MLR candidates • Standard sources: LinkedIn, recruiters, on-campus recruiting, etc • Monitor arXiv and top conferences and flag first authors of papers you like • Look for good reimplementations of papers you like • Attend ML research conferences (NeurIPS, ICLR, ICML) 66ML Teams - hiring
Full Stack Deep Learning How to attract MLR / MLE candidates 67 What do machine learning practitioners want? How to make your company stand out? • Work with cutting edge tools & techniques
 
 • Build skills / knowledge in an exciting field
 
 • Work with excellent people
 
 • Work on interesting datasets
 
 • Do work that matters • Work on research-oriented projects. Publicize them. Invest in tooling for your team & empower employees to try new tools. • Build team culture around learning 
 (reading groups, learning days, professional development budget, conference budget) • Hire high-profile people. Help your best people build their profile through publishing blogs & papers. • Sell the uniqueness of your dataset in recruiting materials. 
 • Sell the mission of your company and potential impact of machine learning on that mission. Work on projects that have a tangible impact today. ML Teams - hiring
Full Stack Deep Learning Hiring for ML - outline • The AI Talent Gap • Sourcing • Interviewing • Finding a job • Course exam 68ML Teams - hiring
Full Stack Deep Learning What to test in an ML interview? • Hire for strengths • Meet a minimum bar for everything else 69ML Teams - hiring
Full Stack Deep Learning What to test in an ML interview? • Validate your hypotheses of candidate’s strengths • Researchers: make sure they can think creatively about new ML problems, probe how thoughtful they were about previous projects • Engineers: make sure they are great generalist SWEs • Make sure candidates meet a minimum bar on weaker areas • Researchers: test SWE knowledge and ability to write good code • SWEs: test ML knowledge 70ML Teams - hiring
Full Stack Deep Learning What happens in a ML interview? • Much less well-defined than software engineering interviews • Common types of assessments: • Background & culture fit • Whiteboard coding (similar to SWE interviews) • Pair coding (similar to SWE interviews) • Pair debugging (often ML-specific code) • Math puzzles (e.g., involving linear algebra) • Take-home ML project • Applied ML (e.g., explain how you’d solve this problem with ML) • Previous ML projects (e.g., probing on what you tried, why things did / didn’t work) • ML theory (e.g., bias-variance tradeoff, overfitting, underfitting, understanding of specific algorithms) 71ML Teams - hiring
Full Stack Deep Learning Hiring for ML - outline • The AI Talent Gap • Sourcing • Interviewing • Finding a job • Course exam 72ML Teams - hiring
Full Stack Deep Learning Where to look for a ML job? • Standard sources: LinkedIn, recruiters, on-campus recruiting, etc • ML research conferences (NeurIPS, ICLR, ICML) • Apply directly (remember, there’s a talent gap!) • This course • Those who pass the exam will get access to our recruiting database 73ML Teams - hiring
Full Stack Deep Learning How to stand out for ML roles? • Build software engineering skills (e.g., work at a well-known software company) • Exhibit interest in ML (e.g., conference attendance, online courses taken) • Show you have broad knowledge of ML (e.g., write blog posts synthesizing a research area) • Demonstrate ability to get ML projects done (e.g., create side projects, re- implement papers) • Prove you can think creatively in ML (e.g., win Kaggle competitions, publish papers) 74 Moreimpressive ML Teams - hiring
Full Stack Deep Learning How to prepare for the interview? • Prepare for a general SWE interview (e.g., “Cracking the Coding Interview”) • Prepare to talk in detail about your past ML projects (remember details, prepare to talk about tradeoffs and decisions you made) • Review how basic ML algorithms work (linear / logistic regression, nearest neighbor, decision trees, k-means, MLPs, ConvNets, recurrent nets, etc) • Review ML theory • Think about the problems the company you’re interviewing with may face and what ML techniques may apply to them 75ML Teams - hiring
Full Stack Deep Learning Hiring for ML - outline • The AI Talent Gap • Sourcing • Interviewing • Finding a job • Course exam 76ML Teams - hiring
Full Stack Deep Learning Overview of the exam • Designed to help you prepare for ML engineering interviews • Take on your own time 77ML Teams - hiring
Full Stack Deep Learning Outline • Problem setup (e.g., for a particular problem, what data would you look for, what model would you choose, and what metric would you optimize?) • Algorithm knowledge (e.g., how does an LSTM work?) • Understanding of common conventions (e.g., what problem with RNNs does an LSTM solve?) • ML theory (e.g., understanding bias and variance) • ML debugging (e.g., what’s likely to cause the bug here?) 78ML Teams - hiring
Full Stack Deep Learning Example question 1 • Why does the Residual Block in the ResNet architecture help with the vanishing gradient problem? 79ML Teams - hiring
Full Stack Deep Learning Example question 2 • Suppose you see the following learning curve when training on a single batch. Which of the following could be the cause? (Select all that apply) • Shuffled labels • Learning rate too low • Learning rate too high • Numerical instability • Too big a model 80ML Teams - hiring
Full Stack Deep Learning Example question 3 •For each of the following prediction tasks, select the loss function that is best suited for it. •Predict sale price of a house listed for sale. •Predict whether an image contains pornography or not. •Predict the category of an email: personal, promotional, reminder, or spam. •Predict whether a voice sample belongs to the owner of the phone. 81 (a) Mean Squared Error 
 (b) Categorical cross-entropy 
 (c) Binary cross-entropy
 (d) CTC loss
 (e) GAN loss ML Teams - hiring
Full Stack Deep Learning Questions? 82ML Teams - hiring
Full Stack Deep Learning Conclusion 83 Orgs • ML teams are becoming more standalone, hence more interdisciplinary Roles • Lots of different skills involved in production ML, so there’s an opportunity for many to contribute Hiring • Talent is scarce, so be specific about what is must-have. It can be hard to break in as an outsider - use projects to build awareness. Managing • Managing ML teams is hard. There’s no silver bullet, but shifting toward probabilistic planning can help ML Teams - hiring
Full Stack Deep Learning Thank you! 84

More Related Content

PDF
Lecture 11: ML Deployment & Monitoring (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
PDF
AWS 를 활용한 저지연 라이브 (Low Latency Live) 서비스 구현 - 류재춘 컨설턴트/에반젤리스트, GS Neot다 :: AW...
byAmazon Web Services Korea
 
PDF
Architecting an Highly Available and Scalable WordPress Site in AWS
byHarish Ganesan
 
PDF
AWS 머신러닝 솔루션을 활용한 고객 응대 자동화 구축 사례 공유 - 이창명, CTO, 위메이드 플레이 ::: Games on AWS 2022
byAmazon Web Services Korea
 
PPTX
Integrating Apache Spark and NiFi for Data Lakes
byDataWorks Summit/Hadoop Summit
 
PPTX
글로벌 기업들의 효과적인 데이터 분석을 위한 Data Lake 구축 및 분석 사례 - 김준형 (AWS 솔루션즈 아키텍트)
byAmazon Web Services Korea
 
PPT
electron spin resonance spectroscopy ( EPR/ESR)
byshyam sunder pandiya
 
PPTX
Choosing technologies for a big data solution in the cloud
byJames Serra
 
Lecture 11: ML Deployment & Monitoring (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
AWS 를 활용한 저지연 라이브 (Low Latency Live) 서비스 구현 - 류재춘 컨설턴트/에반젤리스트, GS Neot다 :: AW...
byAmazon Web Services Korea
 
Architecting an Highly Available and Scalable WordPress Site in AWS
byHarish Ganesan
 
AWS 머신러닝 솔루션을 활용한 고객 응대 자동화 구축 사례 공유 - 이창명, CTO, 위메이드 플레이 ::: Games on AWS 2022
byAmazon Web Services Korea
 
Integrating Apache Spark and NiFi for Data Lakes
byDataWorks Summit/Hadoop Summit
 
글로벌 기업들의 효과적인 데이터 분석을 위한 Data Lake 구축 및 분석 사례 - 김준형 (AWS 솔루션즈 아키텍트)
byAmazon Web Services Korea
 
electron spin resonance spectroscopy ( EPR/ESR)
byshyam sunder pandiya
 
Choosing technologies for a big data solution in the cloud
byJames Serra
 

What's hot

PDF
AWS 기반 소프트웨어 서비스(SaaS) -김용우 솔루션즈 아키텍트 :: AWS 파트너 테크시프트 세미나
byAmazon Web Services Korea
 
PDF
Ammonia
byyagnesh96
 
PDF
CloudWatch 성능 모니터링과 신속한 대응을 위한 노하우 - 박선용 솔루션즈 아키텍트:: AWS Cloud Track 3 Gaming
byAmazon Web Services Korea
 
PPTX
Chemistry:Bioinorganic Chemistry
bySt Mary's College,Thrissur,Kerala
 
PDF
모바일 게임플랫폼과 인프라 구축 경험기
byif kakao
 
PPTX
Hadoop Infrastructure @Uber Past, Present and Future
byDataWorks Summit
 
PPT
Excess gibbs free energy models
bySunny Chauhan
 
PDF
Aspen Custom Modeler Overview
bySara Alvarez
 
PPTX
Non-equilibrium Thermodynamics
byVarinderKhepar
 
PDF
Lecture 6: Infrastructure & Tooling (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
PDF
AWS re:Invent re:Cap - 비용 최적화: 모범사례와 아키텍처 설계 기초편 - 이종남
byAmazon Web Services Korea
 
PDF
대용량 데이터레이크 마이그레이션 사례 공유 [카카오게임즈 - 레벨 200] - 조은희, 팀장, 카카오게임즈 ::: Games on AWS ...
byAmazon Web Services Korea
 
PPTX
Aem asset optimizations & best practices
byKanika Gera
 
PPTX
Synthetic petrol
byDharmraj Sharma
 
PDF
[4차]넷플릭스 알고리즘 분석(151106)
by고려대학교 정보기술경영학회 : ITS
 
PPT
Stereochemistry of Alkanes and Cycloalkanes
byNilesh Thakare
 
ODP
Batch processing in EDA (Event Driven Architectures)
byÁlvaro Santuy Elena
 
PDF
AWS Summit Seoul 2023 | Global scale service의 중앙 집중식 Observability 구축
byAmazon Web Services Korea
 
PDF
BigQuery implementation
bySimon Su
 
PDF
Amazon Elastic MapReduce with Hive/Presto ハンズオン(講義)
byAmazon Web Services Japan
 
AWS 기반 소프트웨어 서비스(SaaS) -김용우 솔루션즈 아키텍트 :: AWS 파트너 테크시프트 세미나
byAmazon Web Services Korea
 
Ammonia
byyagnesh96
 
CloudWatch 성능 모니터링과 신속한 대응을 위한 노하우 - 박선용 솔루션즈 아키텍트:: AWS Cloud Track 3 Gaming
byAmazon Web Services Korea
 
Chemistry:Bioinorganic Chemistry
bySt Mary's College,Thrissur,Kerala
 
모바일 게임플랫폼과 인프라 구축 경험기
byif kakao
 
Hadoop Infrastructure @Uber Past, Present and Future
byDataWorks Summit
 
Excess gibbs free energy models
bySunny Chauhan
 
Aspen Custom Modeler Overview
bySara Alvarez
 
Non-equilibrium Thermodynamics
byVarinderKhepar
 
Lecture 6: Infrastructure & Tooling (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
AWS re:Invent re:Cap - 비용 최적화: 모범사례와 아키텍처 설계 기초편 - 이종남
byAmazon Web Services Korea
 
대용량 데이터레이크 마이그레이션 사례 공유 [카카오게임즈 - 레벨 200] - 조은희, 팀장, 카카오게임즈 ::: Games on AWS ...
byAmazon Web Services Korea
 
Aem asset optimizations & best practices
byKanika Gera
 
Synthetic petrol
byDharmraj Sharma
 
[4차]넷플릭스 알고리즘 분석(151106)
by고려대학교 정보기술경영학회 : ITS
 
Stereochemistry of Alkanes and Cycloalkanes
byNilesh Thakare
 
Batch processing in EDA (Event Driven Architectures)
byÁlvaro Santuy Elena
 
AWS Summit Seoul 2023 | Global scale service의 중앙 집중식 Observability 구축
byAmazon Web Services Korea
 
BigQuery implementation
bySimon Su
 
Amazon Elastic MapReduce with Hive/Presto ハンズオン(講義)
byAmazon Web Services Japan
 

Similar to Machine Learning Teams - Full Stack Deep Learning

PDF
Lecture 13: ML Teams (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
PDF
DevOps Days Rockies MLOps
byMatthew Reynolds
 
PDF
MLOps – Applying DevOps to Competitive Advantage
byDATAVERSITY
 
PPTX
From Notebook to Production: What Most ML Tutorials Don’t Teach
byvivekbharti311
 
PDF
Using MLOps to Bring ML to Production/The Promise of MLOps
byWeaveworks
 
PDF
Building Your Dream Machine Learning Team with Python Expertise
byriyak40
 
PDF
Rsqrd AI: From R&D to ROI of AI
bySanjana Chowdhury
 
PDF
Fundamental MLOps
bySaripudin Gon
 
PDF
MLSEV Virtual. ML Platformization and AutoML in the Enterprise
byBigML, Inc
 
PDF
Real World End to End machine Learning Pipeline
bySrivatsan Srinivasan
 
PPTX
Expert Machine Learning Solutions for Smarter Business
byDigital Hub Solution
 
PDF
DataScientist Job : Between Myths and Reality.pdf
byJedha Bootcamp
 
PDF
Mykola Mykytenko: MLOps: your way from nonsense to valuable effect (approache...
byLviv Startup Club
 
PDF
ML crash course
bymikaelhuss
 
PPTX
Expert ML Development Services | AI Solutions for Your Business
byDigital Hub Solution
 
PDF
Productionising Machine Learning Models
byTash Bickley
 
PDF
Machine Learning Platformization & AutoML: Adopting ML at Scale in the Enterp...
byEd Fernandez
 
PDF
Azure Engineering MLOps
byMinesh A. Jethva
 
PDF
Key MLOps (Mahine Learning Operational) Features
byAgus Kurniawan
 
PPTX
Machine_Learning_with brief conceps.pptx
byratherx8
 
Lecture 13: ML Teams (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
DevOps Days Rockies MLOps
byMatthew Reynolds
 
MLOps – Applying DevOps to Competitive Advantage
byDATAVERSITY
 
From Notebook to Production: What Most ML Tutorials Don’t Teach
byvivekbharti311
 
Using MLOps to Bring ML to Production/The Promise of MLOps
byWeaveworks
 
Building Your Dream Machine Learning Team with Python Expertise
byriyak40
 
Rsqrd AI: From R&D to ROI of AI
bySanjana Chowdhury
 
Fundamental MLOps
bySaripudin Gon
 
MLSEV Virtual. ML Platformization and AutoML in the Enterprise
byBigML, Inc
 
Real World End to End machine Learning Pipeline
bySrivatsan Srinivasan
 
Expert Machine Learning Solutions for Smarter Business
byDigital Hub Solution
 
DataScientist Job : Between Myths and Reality.pdf
byJedha Bootcamp
 
Mykola Mykytenko: MLOps: your way from nonsense to valuable effect (approache...
byLviv Startup Club
 
ML crash course
bymikaelhuss
 
Expert ML Development Services | AI Solutions for Your Business
byDigital Hub Solution
 
Productionising Machine Learning Models
byTash Bickley
 
Machine Learning Platformization & AutoML: Adopting ML at Scale in the Enterp...
byEd Fernandez
 
Azure Engineering MLOps
byMinesh A. Jethva
 
Key MLOps (Mahine Learning Operational) Features
byAgus Kurniawan
 
Machine_Learning_with brief conceps.pptx
byratherx8
 

More from Sergey Karayev

PDF
Lecture 12: Research Directions (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
PDF
Lecture 10: ML Testing & Explainability (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
PDF
Lecture 9: AI Ethics (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
PDF
Lecture 8: Data Management (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
PDF
Lecture 7: Troubleshooting Deep Neural Networks (Full Stack Deep Learning - S...
bySergey Karayev
 
PDF
Lecture 5: ML Projects (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
PDF
Lecture 4: Transformers (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
PDF
Lecture 3: RNNs - Full Stack Deep Learning - Spring 2021
bySergey Karayev
 
PDF
Lecture 2.B: Computer Vision Applications - Full Stack Deep Learning - Spring...
bySergey Karayev
 
PDF
Lecture 2.A: Convolutional Networks - Full Stack Deep Learning - Spring 2021
bySergey Karayev
 
PDF
Lab 1: Intro and Setup - Full Stack Deep Learning - Spring 2021
bySergey Karayev
 
PDF
Lecture 1: Deep Learning Fundamentals - Full Stack Deep Learning - Spring 2021
bySergey Karayev
 
PDF
Data Management - Full Stack Deep Learning
bySergey Karayev
 
PDF
Testing and Deployment - Full Stack Deep Learning
bySergey Karayev
 
PDF
Troubleshooting Deep Neural Networks - Full Stack Deep Learning
bySergey Karayev
 
PDF
Setting up Machine Learning Projects - Full Stack Deep Learning
bySergey Karayev
 
PDF
Research Directions - Full Stack Deep Learning
bySergey Karayev
 
PDF
Infrastructure and Tooling - Full Stack Deep Learning
bySergey Karayev
 
PDF
AI Masterclass at ASU GSV 2019
bySergey Karayev
 
KEY
Attentional Object Detection - introductory slides.
bySergey Karayev
 
Lecture 12: Research Directions (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
Lecture 10: ML Testing & Explainability (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
Lecture 9: AI Ethics (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
Lecture 8: Data Management (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
Lecture 7: Troubleshooting Deep Neural Networks (Full Stack Deep Learning - S...
bySergey Karayev
 
Lecture 5: ML Projects (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
Lecture 4: Transformers (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
Lecture 3: RNNs - Full Stack Deep Learning - Spring 2021
bySergey Karayev
 
Lecture 2.B: Computer Vision Applications - Full Stack Deep Learning - Spring...
bySergey Karayev
 
Lecture 2.A: Convolutional Networks - Full Stack Deep Learning - Spring 2021
bySergey Karayev
 
Lab 1: Intro and Setup - Full Stack Deep Learning - Spring 2021
bySergey Karayev
 
Lecture 1: Deep Learning Fundamentals - Full Stack Deep Learning - Spring 2021
bySergey Karayev
 
Data Management - Full Stack Deep Learning
bySergey Karayev
 
Testing and Deployment - Full Stack Deep Learning
bySergey Karayev
 
Troubleshooting Deep Neural Networks - Full Stack Deep Learning
bySergey Karayev
 
Setting up Machine Learning Projects - Full Stack Deep Learning
bySergey Karayev
 
Research Directions - Full Stack Deep Learning
bySergey Karayev
 
Infrastructure and Tooling - Full Stack Deep Learning
bySergey Karayev
 
AI Masterclass at ASU GSV 2019
bySergey Karayev
 
Attentional Object Detection - introductory slides.
bySergey Karayev
 

Recently uploaded

PDF
Best 10 Dedicated Servers in Amsterdam, Netherlands (2026 Enterprise Edition)...
byAtal Networks
 
PDF
TrustArc Webinar - Unpacking India’s DPDP Act: What You Should Know
byTrustArc
 
PPTX
Spacewalk 2026 - Presented at the IMAX in Raleigh, NC
byAll Things Open
 
PPTX
Becoming Frontier for Dynamics 365 Finance PPT
byjonbravetti
 
PDF
Just Eat Data Scraping For Restaurant Reviews And Pricing.pdf
byfusiondata2026
 
PDF
Top 10 Dedicated Server Providers in Stockholm (Updated Edition).pdf
byAtal Networks
 
PDF
2006 IEEE International Solid-State Circuits Conference
bySlide_N
 
PDF
Building Voice Agents with Google Agent Development Kit
bySuresh Peiris
 
PDF
Zniber Partners. Audience Foresight for Confident Decisions
bySamuel Zniber
 
PDF
Presentation of Cybersecurity and data protection
bytarikaityahya2
 
PPTX
Design Flaws and Known Attacks in Agentic AI - ITI Business Session
byInformation Technology Institute
 
PPTX
Fortify Your Digital Defenses with ServiceNow Security Operations by Suma Soft
byGavin Ellis
 
PDF
Effective Ai powered mock interview .pdf
byamazingnishusingh766
 
PDF
Special Reeling Cable Specification _ Anhui Feichun Special Cable Co., Ltd_.pdf
byAnhui Feichun Special Cable Co., Ltd.
 
PPTX
congo red stain nd PAS STAIN histopathology
bybushrariaz1240
 
PDF
Unlocking Gen AI Capabilities Within UiPath Document Understanding
byDianaGray10
 
PDF
TopMate ES11 3-Wheel Electric Scooter: Comfort, Stability, and Independence f...
byTopmate
 
PDF
Are AI Hallucinations Getting Better or Worse? We Analyzed the Data.
byScott M. Graffius
 
PDF
Adding Copilot+ PCs with Snapdragon to your HP fleet won’t require IT deploym...
byPrincipled Technologies
 
PDF
Why Enterprises Are Moving to Dedicated Servers in the Netherlands
byAtal Networks
 
Best 10 Dedicated Servers in Amsterdam, Netherlands (2026 Enterprise Edition)...
byAtal Networks
 
TrustArc Webinar - Unpacking India’s DPDP Act: What You Should Know
byTrustArc
 
Spacewalk 2026 - Presented at the IMAX in Raleigh, NC
byAll Things Open
 
Becoming Frontier for Dynamics 365 Finance PPT
byjonbravetti
 
Just Eat Data Scraping For Restaurant Reviews And Pricing.pdf
byfusiondata2026
 
Top 10 Dedicated Server Providers in Stockholm (Updated Edition).pdf
byAtal Networks
 
2006 IEEE International Solid-State Circuits Conference
bySlide_N
 
Building Voice Agents with Google Agent Development Kit
bySuresh Peiris
 
Zniber Partners. Audience Foresight for Confident Decisions
bySamuel Zniber
 
Presentation of Cybersecurity and data protection
bytarikaityahya2
 
Design Flaws and Known Attacks in Agentic AI - ITI Business Session
byInformation Technology Institute
 
Fortify Your Digital Defenses with ServiceNow Security Operations by Suma Soft
byGavin Ellis
 
Effective Ai powered mock interview .pdf
byamazingnishusingh766
 
Special Reeling Cable Specification _ Anhui Feichun Special Cable Co., Ltd_.pdf
byAnhui Feichun Special Cable Co., Ltd.
 
congo red stain nd PAS STAIN histopathology
bybushrariaz1240
 
Unlocking Gen AI Capabilities Within UiPath Document Understanding
byDianaGray10
 
TopMate ES11 3-Wheel Electric Scooter: Comfort, Stability, and Independence f...
byTopmate
 
Are AI Hallucinations Getting Better or Worse? We Analyzed the Data.
byScott M. Graffius
 
Adding Copilot+ PCs with Snapdragon to your HP fleet won’t require IT deploym...
byPrincipled Technologies
 
Why Enterprises Are Moving to Dedicated Servers in the Netherlands
byAtal Networks
 

Machine Learning Teams - Full Stack Deep Learning

  • 1.
    Full Stack DeepLearning Josh Tobin, Sergey Karayev, Pieter Abbeel Machine Learning Teams
  • 2.
    Full Stack DeepLearning Running ML teams is hard 2ML Teams - overview Running any technical team is hard… • ML talent is expensive and scarce • ML teams have a diverse set of roles • Projects have unclear timelines and high uncertainty • The field is moving fast and ML is the “high- interest credit card of technical debt” • Leadership often doesn’t understand AI • Hiring great people • Managing and developing those people • Managing your team’s output and making sure your vectors are aligned • Making good long-term technical choices & managing technical debt • Managing expectations from leadership … And ML adds complexity
  • 3.
    Full Stack DeepLearning Goal of this module • Give you some insight into how to think about building and managing ML teams • Help you get a job in ML 3ML Teams - overview
  • 4.
    Full Stack DeepLearning Managing Module overview 4 Hiring • How to manage a ML team • How to hire ML engineers. How to get hired. Exam for the course ML Teams - overview Orgs • How ML teams are organized and how they fit into the broader organization Roles • ML-related roles and the skills they require
  • 5.
    Full Stack DeepLearning Managing Module overview 5 Hiring • How to manage a ML team • How to hire ML engineers. How to get hired. Exam for the course Orgs • How ML teams are organized and how they fit into the broader organization Roles • ML-related roles and the skills they require ML Teams - roles
  • 6.
    Full Stack DeepLearning Most common ML roles • ML product manager • DevOps • Data engineer • ML engineer • ML researcher / ML scientist • Data scientist 6ML Teams - roles
  • 7.
    Full Stack DeepLearning Most common ML roles • ML product manager • DevOps • Data engineer • ML engineer • ML researcher / ML scientist • Data scientist 7 What’s the difference? ML Teams - roles
  • 8.
    Full Stack DeepLearning Breakdown of job function by role 8 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc ML Teams - roles
  • 9.
    Full Stack DeepLearning Breakdown of job function by role 9 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc DevOps engineer Deploy & monitor production systems Deployed product AWS, etc. ML Teams - roles
  • 10.
    Full Stack DeepLearning Breakdown of job function by role 10 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc DevOps engineer Deploy & monitor production systems Deployed product AWS, etc. Data engineer Build data pipelines, aggregation, storage, monitoring Distributed system Hadoop, Kafka, Airflow ML Teams - roles
  • 11.
    Full Stack DeepLearning Breakdown of job function by role 11 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc DevOps engineer Deploy & monitor production systems Deployed product AWS, etc. Data engineer Build data pipelines, aggregation, storage, monitoring Distributed system Hadoop, Kafka, Airflow ML engineer Train & deploy prediction models Prediction system running on real data (often in production) Tensorflow, Docker ML Teams - roles
  • 12.
    Full Stack DeepLearning Breakdown of job function by role 12 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc DevOps engineer Deploy & monitor production systems Deployed product AWS, etc. Data engineer Build data pipelines, aggregation, storage, monitoring Distributed system Hadoop, Kafka, Airflow ML engineer Train & deploy prediction models Prediction system running on real data (often in production) Tensorflow, Docker ML researcher Train prediction models (often forward looking or not production-critical) Prediction model & report describing it Tensorflow, pytorch, Jupyter ML Teams - roles
  • 13.
    Full Stack DeepLearning Breakdown of job function by role 13 Role Job Function Work product Commonly used tools ML product manager Work with ML team, business, users, data owners to prioritize & execute projects Design docs, wireframes, work plans Jira, etc DevOps engineer Deploy & monitor production systems Deployed product AWS, etc. Data engineer Build data pipelines, aggregation, storage, monitoring Distributed system Hadoop, Kafka, Airflow ML engineer Train & deploy prediction models Prediction system running on real data (often in production) Tensorflow, Docker ML researcher Train prediction models (often forward looking or not production-critical) Prediction model & report describing it Tensorflow, pytorch, Jupyter Data scientist Blanket term used to describe all of the above. In some orgs, means answering business questions using analytics Prediction model or report SQL, Excel, Jupyter, Pandas, SKLearn, Tensorflow ML Teams - roles
  • 14.
    Full Stack DeepLearning What skills are needed for the roles? 14 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM
  • 15.
    Full Stack DeepLearning What skills are needed for the roles? 15 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM Primarily a software engineering role. Often from standard SWE pipeline.
  • 16.
    Full Stack DeepLearning What skills are needed for the roles? 16 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM SWE with ML team as an active customer
  • 17.
    Full Stack DeepLearning What skills are needed for the roles? 17 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM Rare mix of ML skills and SWE skills. Often SWEs with significant self- teaching or science / engineering PhDs who worked as traditional SWEs after grad school
  • 18.
    Full Stack DeepLearning What skills are needed for the roles? 18 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM ML experts. Usually have MS/PhD in CS or Stats or did an industrial fellowship program
  • 19.
    Full Stack DeepLearning What skills are needed for the roles? 19 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM Wide range of backgrounds from undergrad-only to science PhD
  • 20.
    Full Stack DeepLearning What skills are needed for the roles? 20 Machine learning Low High Low High Size of bubble = communication / technical writing ML Researcher ML Engineer Data scientist Data engineer ML DevOps ML Teams - roles ML PM Traditional PMs, but with a deep understanding of the ML development process & mindset
  • 21.
    Full Stack DeepLearning Questions? 21ML Teams - roles
  • 22.
    Full Stack DeepLearning Managing Module overview 22 Hiring • How to manage a ML team • How to hire ML engineers. How to get hired. Exam for the course Orgs • How ML teams are organized and how they fit into the broader organization Roles • ML-related roles and the skills they require ML Teams - orgs
  • 23.
    Full Stack DeepLearning ML org structures - lessons learned • No consensus yet on the right way to structure a ML team • This lecture: taxonomy of best practices for different organizational maturity levels 23ML Teams - orgs
  • 24.
    Full Stack DeepLearning ML organization archetypes 24 The ML Organization Mountain ML Teams - orgs
  • 25.
    Full Stack DeepLearning ML organization archetypes 25 The ML Organization Mountain Nascent / Ad-Hoc ML What it looks like • No one is doing ML, or ML is done on an ad-hoc basis • Little ML expertise in-house Example organizations • Most small-medium businesses • Less technology-forward large companies (education, logistics, etc) Advantages • Often low-hanging fruit for ML Dis- advantages • Little support for ML projects, difficult to hire and retain good talent ML Teams - orgs
  • 26.
    Full Stack DeepLearning ML R&D ML organization archetypes 26 What it looks like • ML efforts are centered in the R&D arm of the organization • Often hire researchers / PhDs & write papers Example organizations • Larger Oil & gas, manufacturing, telecom companies Advantages • Often can hire experienced researchers • Can work on long-term business priorities & big wins Dis- advantages • Difficult to get data • Rarely translates into actual business value, so usually the amount of investment remains small The ML Organization Mountain ML Teams - orgs
  • 27.
    Full Stack DeepLearning ML embedded into business / product teams ML organization archetypes 27 What it looks like • Certain product teams or business units have ML expertise along-side their software or analytics talent • ML reports up to the team’s engineering lead or tech lead Example organizations • Software / technology companies • Financial services companies Advantages • ML improvements are likely to lead to business value • Tight feedback cycle between idea and product improvement Dis- advantages • Hard to hire and develop top talent • Access to resources (data / compute) can lag • ML project cycles conflict with engineering mgmt • Long-term projects can be hard to justify The ML Organization Mountain ML Teams - orgs
  • 28.
    Full Stack DeepLearning Independent ML Function ML organization archetypes 28 What it looks like • ML division reporting to senior leadership (often CEO) • ML PMs work with MLRs, MLEs, and customers to build ML into products • Teams sometimes publish long-term research Example organizations • Large financial services companies Advantages • Talent density allows to hire & train top practitioners • Senior leaders can marshal data / compute resources • Can invest in tooling, practices, and culture around ML development Dis- advantages • Model handoffs to lines of business can be challenging - users need to buy-in and be educated on model use • Feedback cycles can be slow The ML Organization Mountain ML Teams - orgs
  • 29.
    Full Stack DeepLearning ML-First Organizations ML organization archetypes 29 What it looks like • CEO buy-in • ML division working on challenging, long-term projects • ML expertise in every line of business focusing on quick wins and working with central ML division Example organizations • Large tech companies • ML-focused startups Advantages • Best data access: data thinking permeates the org • Recruiting: ML team works on hardest problems • Easiest deployment: product teams understand ML Dis- advantages • Hard to implement • Challenging & expensive to recruit enough talent • Culturally difficult to embed ML thinking everywhere ML Teams - orgs
  • 30.
    Full Stack DeepLearning ML team structures - design choices 30 Software engineering vs research Data ownership Model ownership Key questions • To what extent is the ML team responsible for building or integrating with software? • How important are SWE skills on the team? • How much control does the ML team have over data collection, warehousing, labeling, and pipelining? • Is the ML team responsible for deploying models into production? • Who maintains deployed models? ML Teams - orgs
  • 31.
    Full Stack DeepLearning ML team structures - design choices 31 Software engineering vs research Data ownership Model ownership • Research prioritized over SWE skills • Researcher-SWE collaboration lacking ML R&D • ML team has no control over data • ML team typically will not have data engineering component • Models are rarely deployed into production ML Teams - orgs
  • 32.
    Full Stack DeepLearning ML team structures - design choices 32 Software engineering vs research Data ownership Model ownership Embedded ML • SWE skills prioritized over research skills • Often, all researchers need strong SWE as everyone expected to deploy • ML team generally does not own data production / mgmt • Work with data engineers to build pipelines • ML engineers own the models that they deploy into production ML Teams - orgs • Research prioritized over SWE skills • Researcher-SWE collaboration lacking ML R&D • ML team has no control over data • ML team typically will not have data engineering component • Models are rarely deployed into production
  • 33.
    Full Stack DeepLearning ML team structures - design choices 33 Software engineering vs research Data ownership Model ownership ML Function • Each team has a strong mix of SWE and research skills • SWE and researchers work closely together within team • ML team has a voice in data governance discussions • ML team has strong internal data engineering function • ML team hands off models to user, but is responsible for maintaining them ML Teams - orgs Embedded ML • Research prioritized over SWE skills • Researcher-SWE collaboration lacking ML R&D • ML team has no control over data • ML team typically will not have data engineering component • Models are rarely deployed into production • SWE skills prioritized over research skills • Often, all researchers need strong SWE as everyone expected to deploy • ML team generally does not own data production / mgmt • Work with data engineers to build pipelines • ML engineers own the models that they deploy into production
  • 34.
    Full Stack DeepLearning ML team structures - design choices 34 Software engineering vs research Data ownership Model ownership Embedded ML • Research prioritized over SWE skills • Researcher-SWE collaboration lacking ML R&D ML Function ML First • ML team has no control over data • ML team typically will not have data engineering component • Models are rarely deployed into production • SWE skills prioritized over research skills • Often, all researchers need strong SWE as everyone expected to deploy • Each team has a strong mix of SWE and research skills • SWE and researchers work closely together within team • Different teams are more or less research oriented • Research teams collaborate closely with SWE teams • ML team generally does not own data production / mgmt • Work with data engineers to build pipelines • ML engineers own the models that they deploy into production • ML team has a voice in data governance discussions • ML team has strong internal data engineering function • ML team hands off models to user, but is responsible for maintaining them • ML team often owns company-wide data infrastructure • ML team hands off models to user, who operates and maintains them ML Teams - orgs
  • 35.
    Full Stack DeepLearning ML team structures - design choices 35 Software engineering vs research Data ownership Model ownership Embedded ML • Research prioritized over SWE skills • Researcher-SWE collaboration lacking ML R&D ML Function ML First • ML team has no control over data • ML team typically will not have data engineering component • Models are rarely deployed into production • SWE skills prioritized over research skills • Often, all researchers need strong SWE as everyone expected to deploy • Each team has a strong mix of SWE and research skills • SWE and researchers work closely together within team • Different teams are more or less research oriented • Research teams collaborate closely with SWE teams • ML team generally does not own data production / mgmt • Work with data engineers to build pipelines • ML engineers own the models that they deploy into production • ML team has a voice in data governance discussions • ML team has strong internal data engineering function • ML team hands off models to user, but is responsible for maintaining them • ML team often owns company-wide data infrastructure • ML team hands off models to user, who operates and maintains them ML Teams - orgs
  • 36.
    Full Stack DeepLearning Questions? 36ML Teams - orgs
  • 37.
    Full Stack DeepLearning Managing Module overview 37 Hiring • How to manage a ML team • How to hire ML engineers. How to get hired. Exam for the course Orgs • How ML teams are organized and how they fit into the broader organization Roles • ML-related roles and the skills they require ML Teams - managing
  • 38.
    Full Stack DeepLearning Managing ML teams is challenging • It’s hard to tell in advance how hard or easy something is 38ML Teams - managing
  • 39.
    Full Stack DeepLearning Managing ML teams is challenging 39 https://medium.com/@l2k/why-are-machine-learning-projects-so-hard-to-manage-8e9b9cf49641 It’s hard to tell in advance how easy or hard something is ML Teams - managing
  • 40.
    Full Stack DeepLearning Managing ML teams is challenging • It’s hard to tell in advance how hard or easy something is • ML progress is nonlinear • Very common for projects to stall for weeks or longer • In early stages, difficult to plan project because unclear what will work • As a result, estimating project timelines is extremely difficult • I.e., production ML is still somewhere between “research” and “engineering” 40ML Teams - managing
  • 41.
    Full Stack DeepLearning Managing ML teams is challenging • It’s hard to tell in advance how hard or easy something is • ML progress is nonlinear • There are cultural gaps between research and engineering • Different values, backgrounds, goals, norms • In toxic cultures, the two sides often don’t value one another 41ML Teams - managing
  • 42.
    Full Stack DeepLearning Managing ML teams is challenging • It’s hard to tell in advance how hard or easy something is • ML progress is nonlinear • There are cultural gaps between research and engineering • Leaders often don’t understand it 42ML Teams - managing
  • 43.
    Full Stack DeepLearning How to manage ML teams better • Do ML Project planning probabilistically 43ML Teams - managing
  • 44.
    Full Stack DeepLearning How to manage ML teams better • Do ML project planning probabilistically • From:
 
 
 
 
 • To:
 
 44ML Teams - managing Task A Task C Task D Task E Task F Task G Week 1 Week 2 Week 3 Week 4
  • 45.
    Full Stack DeepLearning How to manage ML teams better • Do ML project planning probabilistically • From:
 
 
 
 
 • To:
 
 45ML Teams - managing Task A Task C Task D Task E Task F Task G Week 1 Week 2 Week 3 Week 4 Task A (50%) Task B (25%) Task C (50%) Week 1 Week 2 Week 3 Week 4 Task D (75%)
  • 46.
    Full Stack DeepLearning How to manage ML teams better • Do ML project planning probabilistically • From:
 
 
 
 
 • To:
 
 46ML Teams - managing Task A Task C Task D Task E Task F Task G Week 1 Week 2 Week 3 Week 4 Task A (50%) Task B (25%) Task C (50%) Week 1 Week 2 Week 3 Week 4 Task D (75%)
  • 47.
    Full Stack DeepLearning How to manage ML teams better • Do ML project planning probabilistically • From:
 
 
 
 
 • To:
 
 47ML Teams - managing Task A Task C Task D Task E Task F Task G Week 1 Week 2 Week 3 Week 4 Task A Task B Task C Week 1 Week 2 Week 3 Week 4 Task D
  • 48.
    Full Stack DeepLearning How to manage ML teams better • Do ML project planning probabilistically • From:
 
 
 
 
 • To:
 
 48ML Teams - managing Task A Task C Task D Task E Task F Task G Week 1 Week 2 Week 3 Week 4 Task A Task B Task C Week 1 Week 2 Week 3 Week 4 Task E (50%) Task F (50%) Task G (10%) Task D (75%)
  • 49.
    Full Stack DeepLearning How to manage ML teams better • Do ML Project planning probabilistically • Attempt a portfolio of approaches • Measure progress based on inputs, not results • Have researchers and engineers work together • Get end-to-end pipelines together quickly to demonstrate quick wins • Educate leadership on ML timeline uncertainty 49ML Teams - managing
  • 50.
    Full Stack DeepLearning Resources for educating execs • https://a16z.com/2016/06/10/ai-deep-learning-machines/ • Pieter’s upcoming AI Strategy class:
 https://emeritus-executive.berkeley.edu/artificial-intelligence/ 50ML Teams - managing
  • 51.
    Full Stack DeepLearning Questions? 51ML Teams - managing
  • 52.
    Full Stack DeepLearning Managing Module overview 52 Hiring • How to manage a ML team • How to hire ML engineers. How to get hired. Exam for the course Orgs • How ML teams are organized and how they fit into the broader organization Roles • ML-related roles and the skills they require ML Teams - hiring
  • 53.
    Full Stack DeepLearning Hiring for ML - outline • The AI Talent Gap • Sourcing • Interviewing • Finding a job • Course exam 53ML Teams - hiring
  • 54.
    Full Stack DeepLearning Hiring for ML - outline • The AI Talent Gap • Sourcing • Interviewing • Finding a job • Course exam 54ML Teams - hiring
  • 55.
    Full Stack DeepLearning The AI Talent Gap 55 How many people know how to build AI systems? 5,000 (actively publishing research [Element AI]) 10,000 (estimated num people with the right skillset [Element AI]) 22,000 (PhD-educated AI researchers [Bloomberg]) 90,000 (upper bound on number of people [Element AI]) 200,000 - 300,000 (Number of AI researcher / practitioners [Tencent]) 3.6M (Number of software developers in the US) 18.2M (Number of software developers in the world) Sources: The AI Talent Shortage (Nikolai Yakovenko) https://medium.com/@Moscow25/the-ai-talent-shortage-704d8cf0c4cc Just How Shallow is the Artificial Intelligence Talent Pool (Jeremy Kahn) 
 https://www.bloomberg.com/news/articles/2018-02-07/just-how-shallow-is-the-artificial-intelligence-talent-pool ML Teams - hiring
  • 56.
    Full Stack DeepLearning 56 The AI talent gap Fierce competition for AI talent “Everyone agrees that the competition to hire people who know how to build artificial intelligence systems is intense. It’s  turned once- staid academic conferences into frenzied meet markets for corporate recruiters and driven the salaries of the top researchers to seven- figures.” 
 (Bloomberg) Sources: The AI Talent Shortage (Nikolai Yakovenko) https://medium.com/@Moscow25/the-ai-talent-shortage-704d8cf0c4cc Just How Shallow is the Artificial Intelligence Talent Pool (Jeremy Kahn) 
 https://www.bloomberg.com/news/articles/2018-02-07/just-how-shallow-is-the-artificial-intelligence-talent-pool ML Teams - hiring
  • 57.
    Full Stack DeepLearning 57 The AI talent gap Fierce competition for AI talent “Hiring is crazy right now. ML is a young field that got popular very quickly. There’s a ton of demand and not a lot of supply.” 
 (Computer Vision Engineer at Series C startup) ML Teams - hiring
  • 58.
    Full Stack DeepLearning 58 The AI talent gap Fierce competition for AI talent “Hiring for ML is really challenging and takes way more time and effort than we expected. We have someone working on it full-time and we’re still only able to get a few people per quarter” (Startup Founder) ML Teams - hiring
  • 59.
    Full Stack DeepLearning Hiring for ML - outline • The AI Talent Gap • Sourcing • Interviewing • Finding a job • Course exam 59ML Teams - hiring
  • 60.
    Full Stack DeepLearning Most common ML roles • ML product manager • DevOps • Data engineer • ML engineer • ML researcher / ML scientist • Data scientist 60ML Teams - hiring
  • 61.
    Full Stack DeepLearning Most common ML roles • ML product manager • DevOps • Data engineer • ML engineer • ML researcher / ML scientist • Data scientist 61 Slightly different mindset required. 
 Helpful to look for demonstrated interest in AI - courses, conferences, re-implementations, etc ML Teams - hiring
  • 62.
    Full Stack DeepLearning Most common ML roles • ML product manager • DevOps • Data engineer • ML engineer • ML researcher / ML scientist • Data scientist 62 Our focus ML Teams - hiring
  • 63.
    Full Stack DeepLearning How to hire MLEs - the wrong way • Job Description (Unicorn Machine Learning Engineer) • Duties • Keep up with the state of the art • Implement models from scratch • Deep understanding of mathematics & ability to come up with new models • Build tooling & infrastructure for the ML team • Build data pipelines for the ML team • Deploy & monitor models into production • Requirements • PhD • At least 4 years tensorflow experience • At least 4 years as a software engineer • Publications in top ML conference • Experience building large-scale distributed systems 63ML Teams - hiring
  • 64.
    Full Stack DeepLearning How to hire MLEs - the right way • Hire for software engineering skills, interest in ML, and desire to learn. Train to do ML. • Go more junior. Most undergrad computer science students graduate with ML experience. • Be more specific about what you need. Not every ML engineer needs to do DevOps. 64ML Teams - hiring
  • 65.
    Full Stack DeepLearning How to hire MLRs • Look for quality of publications, not quantity (e.g., originality of ideas, quality of execution) • Look for researchers with an eye for working on important problems (many researchers focus on trendy problems without considering why they matter) • Look for researchers with experience outside of academia • Consider hiring talented people from adjacent fields (physics, statistics, math) • Consider hiring people without PhDs (e.g., talented undergraduate / masters students, graduates of Google/Facebook/OpenAI fellowship programs, dedicated self-studiers) 65ML Teams - hiring
  • 66.
    Full Stack DeepLearning How to find MLE/MLR candidates • Standard sources: LinkedIn, recruiters, on-campus recruiting, etc • Monitor arXiv and top conferences and flag first authors of papers you like • Look for good reimplementations of papers you like • Attend ML research conferences (NeurIPS, ICLR, ICML) 66ML Teams - hiring
  • 67.
    Full Stack DeepLearning How to attract MLR / MLE candidates 67 What do machine learning practitioners want? How to make your company stand out? • Work with cutting edge tools & techniques
 
 • Build skills / knowledge in an exciting field
 
 • Work with excellent people
 
 • Work on interesting datasets
 
 • Do work that matters • Work on research-oriented projects. Publicize them. Invest in tooling for your team & empower employees to try new tools. • Build team culture around learning 
 (reading groups, learning days, professional development budget, conference budget) • Hire high-profile people. Help your best people build their profile through publishing blogs & papers. • Sell the uniqueness of your dataset in recruiting materials. 
 • Sell the mission of your company and potential impact of machine learning on that mission. Work on projects that have a tangible impact today. ML Teams - hiring
  • 68.
    Full Stack DeepLearning Hiring for ML - outline • The AI Talent Gap • Sourcing • Interviewing • Finding a job • Course exam 68ML Teams - hiring
  • 69.
    Full Stack DeepLearning What to test in an ML interview? • Hire for strengths • Meet a minimum bar for everything else 69ML Teams - hiring
  • 70.
    Full Stack DeepLearning What to test in an ML interview? • Validate your hypotheses of candidate’s strengths • Researchers: make sure they can think creatively about new ML problems, probe how thoughtful they were about previous projects • Engineers: make sure they are great generalist SWEs • Make sure candidates meet a minimum bar on weaker areas • Researchers: test SWE knowledge and ability to write good code • SWEs: test ML knowledge 70ML Teams - hiring
  • 71.
    Full Stack DeepLearning What happens in a ML interview? • Much less well-defined than software engineering interviews • Common types of assessments: • Background & culture fit • Whiteboard coding (similar to SWE interviews) • Pair coding (similar to SWE interviews) • Pair debugging (often ML-specific code) • Math puzzles (e.g., involving linear algebra) • Take-home ML project • Applied ML (e.g., explain how you’d solve this problem with ML) • Previous ML projects (e.g., probing on what you tried, why things did / didn’t work) • ML theory (e.g., bias-variance tradeoff, overfitting, underfitting, understanding of specific algorithms) 71ML Teams - hiring
  • 72.
    Full Stack DeepLearning Hiring for ML - outline • The AI Talent Gap • Sourcing • Interviewing • Finding a job • Course exam 72ML Teams - hiring
  • 73.
    Full Stack DeepLearning Where to look for a ML job? • Standard sources: LinkedIn, recruiters, on-campus recruiting, etc • ML research conferences (NeurIPS, ICLR, ICML) • Apply directly (remember, there’s a talent gap!) • This course • Those who pass the exam will get access to our recruiting database 73ML Teams - hiring
  • 74.
    Full Stack DeepLearning How to stand out for ML roles? • Build software engineering skills (e.g., work at a well-known software company) • Exhibit interest in ML (e.g., conference attendance, online courses taken) • Show you have broad knowledge of ML (e.g., write blog posts synthesizing a research area) • Demonstrate ability to get ML projects done (e.g., create side projects, re- implement papers) • Prove you can think creatively in ML (e.g., win Kaggle competitions, publish papers) 74 Moreimpressive ML Teams - hiring
  • 75.
    Full Stack DeepLearning How to prepare for the interview? • Prepare for a general SWE interview (e.g., “Cracking the Coding Interview”) • Prepare to talk in detail about your past ML projects (remember details, prepare to talk about tradeoffs and decisions you made) • Review how basic ML algorithms work (linear / logistic regression, nearest neighbor, decision trees, k-means, MLPs, ConvNets, recurrent nets, etc) • Review ML theory • Think about the problems the company you’re interviewing with may face and what ML techniques may apply to them 75ML Teams - hiring
  • 76.
    Full Stack DeepLearning Hiring for ML - outline • The AI Talent Gap • Sourcing • Interviewing • Finding a job • Course exam 76ML Teams - hiring
  • 77.
    Full Stack DeepLearning Overview of the exam • Designed to help you prepare for ML engineering interviews • Take on your own time 77ML Teams - hiring
  • 78.
    Full Stack DeepLearning Outline • Problem setup (e.g., for a particular problem, what data would you look for, what model would you choose, and what metric would you optimize?) • Algorithm knowledge (e.g., how does an LSTM work?) • Understanding of common conventions (e.g., what problem with RNNs does an LSTM solve?) • ML theory (e.g., understanding bias and variance) • ML debugging (e.g., what’s likely to cause the bug here?) 78ML Teams - hiring
  • 79.
    Full Stack DeepLearning Example question 1 • Why does the Residual Block in the ResNet architecture help with the vanishing gradient problem? 79ML Teams - hiring
  • 80.
    Full Stack DeepLearning Example question 2 • Suppose you see the following learning curve when training on a single batch. Which of the following could be the cause? (Select all that apply) • Shuffled labels • Learning rate too low • Learning rate too high • Numerical instability • Too big a model 80ML Teams - hiring
  • 81.
    Full Stack DeepLearning Example question 3 •For each of the following prediction tasks, select the loss function that is best suited for it. •Predict sale price of a house listed for sale. •Predict whether an image contains pornography or not. •Predict the category of an email: personal, promotional, reminder, or spam. •Predict whether a voice sample belongs to the owner of the phone. 81 (a) Mean Squared Error 
 (b) Categorical cross-entropy 
 (c) Binary cross-entropy
 (d) CTC loss
 (e) GAN loss ML Teams - hiring
  • 82.
    Full Stack DeepLearning Questions? 82ML Teams - hiring
  • 83.
    Full Stack DeepLearning Conclusion 83 Orgs • ML teams are becoming more standalone, hence more interdisciplinary Roles • Lots of different skills involved in production ML, so there’s an opportunity for many to contribute Hiring • Talent is scarce, so be specific about what is must-have. It can be hard to break in as an outsider - use projects to build awareness. Managing • Managing ML teams is hard. There’s no silver bullet, but shifting toward probabilistic planning can help ML Teams - hiring
  • 84.
    Full Stack DeepLearning Thank you! 84