The document discusses the application of parallel ablation studies in machine learning using the Maggy framework on Apache Spark. It highlights the significance of ablation studies for deep learning model optimization and presents methodologies for distributed training and hyperparameter tuning. Additionally, it includes a programming model for Maggy, examples of model and feature ablation, and acknowledges contributions from various collaborators.

1. Parallel Ablation Studies for
Machine Learning with Maggy on
Apache Spark
Sina Sheikholeslami
PhD Student, KTH Royal Institute of Technology
Jim Dowling
CEO, Logical Clocks AB
Assoc Prof, KTH Royal Institute of Technology
sinash93
jim_dowling

2. Agenda
Ablation Studies
Why are they important for deep learning?
Asynchronous ML Trials on Spark
Maggy Framework
Parallel Ablation Studies with
Maggy
Programming model with worked-through
example

3. Ablation for Machine Learning
3
Dataset
Machine
Learning Model
Optimizer
Evaluate
Problem Definition
Data Preparation
Model Selection
Repeat if
needed
Model Training
area roomsfloors price

4. Ablation study: Remove, retrain, measure.
0.6 0.17 0.05 0.05 0.1 0.98
Accuracy
=+ ++ +
4

5. Problem: Rewrite ML Code for Ablations, Distribution
Explore
and Design
Experimentation:
Tune and Search
Model Training
(Distributed)
Explainability and
Ablation Studies
5

6. Maggy: Unified code for Distributed ML + Ablations
OBLIVIOUS
TRAINING
FUNCTION
# RUNS ON THE WORKERS
def train():
def input_fn(): # return dataset
model = …
optimizer = …
model.compile(…)
rc = tf.estimator.RunConfig(
‘CollectiveAllReduceStrategy’)
keras_estimator = tf.keras.estimator.
model_to_estimator(….)
tf.estimator.train_and_evaluate(
keras_estimator, input_fn)
Ablation StudiesEDA HParam Tuning Training (Dist)
Apache V2 - https://github.com/logicalclocks/maggy
6

7. Maggy: Programming Model
from maggy import experiment
experiment.set_dataset_generator(gen_dataset)
experiment.set_model_generator(gen_model)
# Hyperparameter optimization
experiment.set_context('optimization', 'randomsearch', searchspace)
result = experiment.lagom(train_fun)
params = result.get('best_hp')
# Distributed Training
experiment.set_context('dist_training', 'MultiWorkerMirroredStrategy', params)
experiment.lagom(train_fun)
# Ablation study
experiment.set_context('ablation', 'loco', ablation_study, params)
experiment.lagom(train_fun)
7

8. Maggy: Distribution and Tracking in One Function*
# RUNS ON THE WORKERS
def train(depth, lr):
from hops import model as mr
def build_data():
..
model = generate_model()
optimizer = …
model.compile(…)
print(…)
mr.export_model(model)
return { ‘accuracy’: acc }
# RUNS ON THE DRIVER
from maggy import experiment
sp=Searchspace(depth=('INTEGER',[2,8]), lr=(..))
experiment.set_context('optimization', 'random’,
sp, direction='max’, num_trials=15)
experiment.lagom(train)
training function & Hparams
save model to Hopsworks Model Registry
track this dict with Experiment results
print to notebook & store in experiment log
define HParams
launch 15 ‘train’ functions on workers
define Trials
https://youtu.be/xora_4iDcQ8
8

9. Maggy vs
*https://www.logicalclocks.com/blog/hopsworks-ml-experiments
def train(depth, weight):
X_train, X_test, y_train, y_test = build_data(..)
...
model.fit(X_train, y_train) # auto-logging
...
hops.export_model(model, "tensorflow",..,model_name)
...
# import matplotlib,create diagram.png
plt.savefig(diagram.png')
return {'accuracy': accuracy, 'diagram': 'diagram.png’}
from maggy import experiment
sp=Searchspace(depth=('INTEGER',[2,8]), weight=('INTEGER’,[2,8]))
experiment.set_context('optimization', 'random’, sp,
direction='max’, num_trials=15)
experiment.lagom(train)
def train(depth, weight):
X_train, X_test, y_train, y_test = build_data(..)
mlflow.set_tracking_uri("jdbc:mysql://uname:pwd@host:3306/db")
mlflow.set_experiment("My Experiment")
with mlflow.start_run() as run:
...
mlflow.log_param("depth", depth)
mlflow.log_param("weight", weight)
with open("test.txt", "w") as f:
f.write("hello world!")
mlflow.log_artifacts("/full/path/to/test.txt")
...
model.fit(X_train, y_train) # auto-logging
...
mlflow.tensorflow.log_model(model, "tensorflow-model",
registered_model_name=model_name)
Maggy Tracking, Model Registry & HParam Tuning MLFlow Tracking & Model Registry (No Hparam Tuning)
10

10. PySpark for Distribution
Worker 1
Worker 5
Worker 3
Worker 2
Worker 4
Worker 7
Worker 8
Worker 6
Driver
TF_CONFIG
Driver
Experiment
Controller
Worker 1 Worker NWorker 2
Single
Host
Explore
and Design
Experimentation:
Tune and Search
Model Training
(Distributed)
Explainability and
Ablation Studies
11

11. Maggy makes transparent
… fixing parameters
… launching
the function
… launching trials (parametrized
instantiations of the function)
… generating new trials
… collecting and logging results
… setting up TF_CONFIG
… wrapping in Distribution Strategy
… launching function as workers
… collecting results
12

12. Maggy: Asynchronous Trials in PySpark for Ablations
Task11
Task12
Task13
Task1N
Driver (Global Optimizer)
Barrier
Metrics New Trial
13

13. Ablation Studies in Maggy
15

14. LOCO: Leave One Component Out
A simple, “natural” ablation policy: an implementation of an ablator
Currently supports Feature, Layer, and Module Ablation
16

15. Feature Ablation
Uses the Feature Store to access the dataset metadata
Generates Python callables that once called, will return modified
datasets
▪ Removes one-feature-at-a-time
17
area roomsfloors price roomsfloors price

16. Model Ablation
Uses a base model function
Generates Python callables that once called, will return modified
models
▪ Uses the model configuration to find and remove layer(s)
▪ Removes one-layer-at-a-time, one-layer-group-at-a-time, or one-module-at-a-time
18

17. Ablation User & Developer API
(Scan for Example Notebooks)

18. Programming Workflow
20

19. User API: Define Dataset Creation
21

20. User API: Define Model Creation
22

21. User API: Define Training Function
23

22. User API: Initialize the Study
24

23. User API: Setup Model Ablation
25

24. User API: Setup Feature Ablation
26

25. User API: Launch Parallel Trials
27

26. Developer API: Policy Implementation (1/2)
28

27. Developer API: Policy Implementation (2/2)
29

28. Maggy is Open-source
Code Repository: https://github.com/logicalclocks/maggy
API Documentation: https://maggy.readthedocs.io/en/latest/
30

29. Acknowledgments
Thanks to our colleagues at Logical Clocks and DC@KTH:
Moritz Meister, Robin Andersson, Kim Hammar,
Kai Jeggle, Alessio Molinari, Alex Ormenisan, Tianze Wang,
Amir Payberah, Vladimir Vlassov
This work is supported by the ExtremeEarth
project funded by European Union’s Horizon
2020 Research and Innovation Programme
under grant agreement No. 825258.

30. Demo: Ablation Study of Common
DL Network Architectures with Maggy

31. Feedback
Your feedback is important to us.
Don’t forget to rate
and review the sessions.