The document discusses the challenges and solutions related to data science engineering and machine learning infrastructure at Netflix, focusing on model training, dependencies, and scalability. It highlights specific issues such as slow query performance, memory errors, and the use of Metaflow for managing model operations effectively. The text underscores the importance of user support and provides contact information for those interested in implementing Metaflow.

1. More Data Science with Less Engineering
ML Infrastructure at Netflix
SF BIG ANALYTICS
SEPTEMBER 2019

2. “
Architecture
The model always depends on the
previous day’s model, so it needs
to be processed for consecutive
days regardless when the
upstream data updates.

3. “
Dependencies
“The problem seem to be that a
dependency of fancyimpute,
cvxpy released a broken version
18h ago.

4. “
Data Access
“Any ideas why this query takes
20-30 mins to run on Presto, and
it seems to be similarly slow on
Spark too?

5. “
Scalability
“I’m going to start training a set of
thousands of models. I have many
countries, and I need to estimate a
hundred models per country.
We’re training 28 separate models
and for each model, we want to
run a hyperparameter search in
parallel.

6. Model Operations
“This shouldn’t be happening.
Interestingly, sounds like the
Friday run was successful even
though Thursday and Saturday
both failed.
My first guess is that we’re
increasingly hitting out of memory
errors as our member count rises.

7. Types of Friction Data Scientists Experience

8. Model Development
Feature Engineering
Model Operations
Versioning
Architecture
Job Scheduler
Compute Resources
Data Warehouse
How much
data scientist
cares
How much
infrastructure
is needed

9. Model Development
Feature Engineering
Model Operations
Versioning
Architecture
Job Scheduler
Compute Resources
Data Warehouse

10. def compute(input):
...
return output
Baseline

11. class MyFlow(FlowSpec):
@step
def start(self):
self.next(self.a, self.b)
@step
def a(self):
self.next(self.join)
@step
def b(self):
self.next(self.join)
@step
def join(self, inputs):
self.next(self.end)
Architecture

12. class MyFlow(FlowSpec):
@step
def start(self):
self.next(self.a, self.b)
@conda(libs={‘tensorflow’: ‘1.14’})
@step
def a(self):
self.next(self.join)
@conda(libs={‘tensorflow’: ‘2.0’})
@step
def b(self):
self.next(self.join)
@step
def join(self, inputs):
self.next(self.end)
isolated
tasks
Dependencies

13. @step
def b(self):
df = MetaflowDataFrame
.from_parquet(parquet_files())
self.next(self.end)
Data Access

14. @resources(cpu=16)
@step
def a(self):
self.next(self.join)
@resources(memory=200000)
@step
def b(self):
self.next(self.join)
16 cores
200GB RAM
Vertical Scalability

15. @step
def start(self):
self.grid = [’x’,’y’,’z’]
self.next(self.a, foreach=’grid’)
@step
def a(self):
self.x = ord(self.input)
self.next(self.join)
@step
def join(self, inputs):
self.result=max(i.x for i in inputs)
self.next(self.end)
Horizontal Scalability

16. namespace('user:brostykus’)
run = Flow('MyFlow').latest_run
print(run.id) # -> 234
print(run.tags) # -> [‘unsampled_model’]
namespace('user:dberg’)
run = Flow('MyFlow').latest_run
print(run.id) # -> 184
print(run.tags) # -> [‘sampled_model’]
# global namespace
namespace(None)
run = Flow('MyFlow').runs('sampled_model’)
print(run.id) # -> 184
Model Ops: Versioning

17. run = Flow('MyFlow').latest_run
if run['A'].success:
print(run['A'].data.x)
else:
print(run['A'].stderr)
Model Ops: Monitoring

18. Addressing real pain
points + fanatic user
support = ❤
“The team's bias towards
helpfulness in the slack channel
(even with questions that are
definitely user error) makes
Metaflow the best developer
product at Netflix.

19. Metaflow Adoption at Netflix

20. Time to First Production Deployment

21. Thank you
If you want to give Metaflow a try in
your organization, please contact us!
Ville Tuulos
@vtuulos
vtuulos@netflix.com