Draft slides for AnthillIn

1. 1
AUTOMATING WORKFLOWS IN AI PROJECTS
JAIDEV DESHPANDE, SENIOR DATA SCIENTIST /@jaidevd

2. DEEP LEARNING HAS BECOME CHEAPER*
*NOT CHEAP ENOUGH FOR REPETITIVE MISTAKES
THE BIG IDEA

3. “FAIL FAST. EITHER DO THE RIGHT THING OR
STOP.”
“WHY DO COMPUTERS STOP AND WHAT CAN BE DONE ABOUT IT?”
JIM GRAY (1985)

4. 4
WHAT’S SUCCESS OR FAILURE IN DEEP LEARNING?
● Neural networks should learn
● Neural networks should learn well
● Neural networks should generalize well

5. WHAT KEEPS A NETWORK FROM DOING ALL THAT?
● The Fundamental Theorem of Statistical Learning: or the
inherent inseparability of data
● The Bias-Variance tradeoff: The inherent property of
least squares systems that plagues all supervised
learning tasks
● No Free Lunch Theorem: All models are, on average,
equally bad!

6. 6
THE TYPICAL CYCLE

7. 7
THE TYPICAL CYCLE

8. 8
THE TYPICAL CYCLE

9. INTRODUCING TEST DRIVEN DEEPLEARNING
DEVELOPMENT

10. 10
SANITY CHECKS CAN BE APPLIED TO:
● The training data
○ Well conditioned problem
○ Redundancy or bias / variance in datasets
○ Stratified train/dev splits
● Training heuristics
○ Reasonable optimizers with learning rates
○ Batch sizes
○ Shuffled training samples
● The model architecture
○ Number of layers
○ Layer arrangement and ordering
● Layers
○ Weight initializations
○ Activations
○ Regularization
● Losses & metrics
○ Consistency across epochs
○ Avoidable bias
Don’t bother
training if
too many of
these checks
fail!

11. 11
BOOKKEEPING FOR DEEP LEARNING EXPERIMENTS
● Don’t repeat yourself
● Keep track of metrics across experiments
● Efficient hyperparameter grid search
● Searching for models
● Transfer Learning - you might already have a model for a
given problem, only trained on a different dataset

12. INTRODUCING KEPLER - A SMART JOURNAL
FOR DEEP LEARNING EXPERIMENTS
HTTPS://GITHUB.COM/JAIDEVD/KEPLER

13. 13
DEEP LEARNING EFFORT

14. 14
KEPLER - A SMART JOURNAL FOR DEEP LEARNING EXPERIMENTS
● Like an IDE, organize models into projects
● Each project has experiments
● An experiment consists of a single training / validation session,
contains
○ Configuration of the model used
○ Basic statistics on the training / validation data like number of features,
samples, etc
○ Error curves
○ Metadata like datetime, path to the related files, etc
● A “check” system (like PEP8) which warns the user at different
stages of the project about various inconsistencies in the model
● A simple search engine for models
● A grid search manager - DRY when doing hyperparameter
search
● A verbose logger that logs all events in all projects

15. 15
SOME OF THE CHECKS IN KEPLER

16. 16
$ cat model.py
import keras
from sklearn.datasets import load_digits
digits = load_digits()
X = digits['data']
y = keras.utils.to_categorical(digits['target'])
model = keras.models.Sequential([
keras.layers.Dense(32, input_shape=(64,)),
keras.layers.Activation('sigmoid'),
keras.layers.Dense(10),
keras.layers.Activation('sigmoid')
])
model.compile(loss='categorical_crossentropy',
optimizer=keras.optimizers.SGD())
$ kepler init
Welcome to Kepler!
$ ipython
>>> from model import *
>>> from kepler import ModelInspector
>>> with ModelInspector(model=model) as mi:
... mi.fit(X, y, epochs=5)
KEPLER EXAMPLES

17. 17
KEPLER EXAMPLES
/Users/jaidevd/src/kepler/kepler/checks.py:51: UserWarning: K102: Training data is not
shuffled. This may slow training down.
warnings.warn(self.code + ": " + self.msg)
/Users/jaidevd/src/kepler/kepler/checks.py:51: UserWarning: K301: Training samples are
correlated. There may be redundancy in the data.
warnings.warn(self.code + ": " + self.msg)
/Users/jaidevd/src/kepler/kepler/checks.py:51: UserWarning: K302: There might be
duplicate training samples.
warnings.warn(self.code + ": " + self.msg)
/Users/jaidevd/src/kepler/kepler/checks.py:51: UserWarning: K303: Training data not
normalized.
warnings.warn(self.code + ": " + self.msg)
Epoch 1/5
1797/1797 [==============================] - 0s 148us/step - loss: 2.3471
Epoch 2/5
1797/1797 [==============================] - 0s 35us/step - loss: 2.2209
Epoch 3/5
1797/1797 [==============================] - 0s 35us/step - loss: 2.1414
Epoch 4/5
1797/1797 [==============================] - 0s 37us/step - loss: 2.0842
Epoch 5/5
1797/1797 [==============================] - 0s 37us/step - loss: 2.0339

18. 18
KEPLER EXAMPLES
$ ipython
>>> from model import *
>>> from kepler import ModelInspector
>>> with ModelInspector(model=model) as mi:
... mi.fit(X, y, epochs=5)
...
There are 1 models similar to this one.
Would you like to see their graphs? [Y/n] : y
Enter location for saving graphs [~/.kepler/models/tf_logs]:
>>>
Graphs written to /Users/jaidevd/.kepler/models/tf_logs
Please point Tensorboard to /Users/jaidevd/.kepler/models/tf_logs
Continue training? [Y/n] : n
>>> exit()
$ tensorboard --logdir ~/.kepler/models/tf_logs
TensorBoard 1.12.0 at http://localhost:6006 (Press CTRL+C to quit)

19. 19
KEPLER EXAMPLES

20. THE KEPLER PHILOSOPHY
1. RIGOROUSLY INSPECT MODELS AND DETECT
INTERNAL INCONSISTENCIES
2. LOG EVERYTHING
3. DON’T REPEAT YOURSELF

21. 21
TECH DETAILS
● Fully written in Python
● So far, supports only Keras with a Tensorflow backend
● Models are guaranteed remain untouched
● Well tested but not well documented :-(
● Beta release scheduled for Q2 2019
● Bug reports & pull requests welcome!

22. 22
THANK YOU!
/@jaidevd
/@jaidevd
deshpande.jaidev@gmail.com