Managing Machine Learning Projects in Industry: As the use of machine learning techniques to analyze and find value in ‘big data’ is being adopted more broadly by industry, we see an increasing need to build teams that can execute on large and complex projects. It is not possible for a single machine learning expert to cover problems of the scope and magnitude that are encountered. The scale of these projects requires teams of researchers and engineers to coordinate and collaborate to deliver impact. In this talk I will touch on some learnings and considerations when building or expanding such a team. I will cover building a group, framing the problem, finding a solution, and evaluating the results. I will illustrate the points with examples drawn from my experience in large companies and startups. I hope to provoke consideration and discussion for the challenges in this area, as well as to illustrate some of the complexities.

1. Managing Machine
Learning Projects in
Industry
Ewa Dominowska
Facebook, Engineering Manager

2. Agenda
• Building a Team
• Selecting and Framing a Problem
• Problem Solving Approach
• Evaluating Solutions
• Delivering Impact

3. Building a Team
• Engineer + ML Expert + Statistician + IR Expert
• Domain expertise
• Academic vs. industry experience
• Research + engineering + experimentation = applied
research
• Investing (domain) vs. outsourcing (science)

4. Selecting a Lead
ML Expert
EngineerManager
1
8
8
8

5. Organizational Structure
• Centralized Research Team
• Centralized Applied Research Team
• Embedded Researchers
• Team Members
• Academia
• Conferences, competitions, data
releases, benchmarks
MSR, Facebook AI Lab
LiveLabs, FB Applied ML
Source: Bonkers World

6. Motivating
• Intellectual challenge
• Creative work
• Autonomy
• Purpose
• Mastery
• Recognition
• Publishing
• Conferences
Source: Motivationhacker

7. Source: Oreilly
Selecting and Framing a Problem

8. Selecting and Framing a Problem
Start with a
business
problem
Break down the
problem
Understand the
impact
Find the right
data
Select an
objective
function
Build Models
Measure and
Evaluate
Experiment
Productionalize
/ Scale

9. Problem Solving Approach
• Establish a baseline
• Check your assumptions
• Select a modellearning technique
• Select features
• Measure and evaluation
• Experiment
• Stability, scalability and robustness
Source: Sheldoncomics

10. Evaluating Solutions
• Defining the right metrics
• Offline evaluation
• A|B testing
• Meaningful vs. representative
• Representativeness and stability of results
• Offline vs. online metrics

11. • How to split traffic
• user, request, budget effects
• How long to run a test
• statistical significance, power, seasonality, novelty
• Calibration
• Model interactions
• Residue effects from previous experiments
Experimentation – Practical Lessons

12. Delivering Impact
• Plan for valuable failure
• Measure long term/steady state effects
• Engineering improvements
• Re-use of components, tools, models and frameworks
• Durability and robustness
• Data, context changes
• Measurement, monitoring, experimentation

13. Thank you!
We are hiring at Facebook!