This document discusses designing machine learning models to provide guidance rather than just predictions. It describes how a model was developed to provide guidance for drawing simple line drawings. The model was trained on features extracted from preprocessed images, such as the number of lines and corners. It then generated guidance by recommending ways to modify the drawing, such as adding a horizontal line in the upper left quadrant. The model aims to learn from the user's response to the guidance to refine future guidance.

1. Designing for guidance in machine learning
Olivia Gunton
Data Scientist @ Textio

2. Predictive power,
complexity
Introspection,
validation, trust
Augmentation of
human capabilities
Black-box Explainable Guidance-producing
Machine learning: evolving needs inform new approaches
Olivia Gunton • Designing for Guidance in Machine Learning

3. MNIST Dataset
Olivia Gunton • Designing for Guidance in Machine Learning

4. vs.
Olivia Gunton • Designing for Guidance in Machine Learning

5. Olivia Gunton • Designing for Guidance in Machine Learning

6. Olivia Gunton • Designing for Guidance in Machine Learning

7. Olivia Gunton • Designing for Guidance in Machine Learning

8. Olivia Gunton • Designing for Guidance in Machine Learning

9. “seven”
Adapted from https://cloud.google.com/blog/big-data/2016/07/understanding-neural-networks-with-tensorflow-playgroundOlivia Gunton • Designing for Guidance in Machine Learning

10. Tensorflow: deep convolutional classifier
Accuracy: ~0.992
Olivia Gunton • Designing for Guidance in Machine Learning

11. Olivia Gunton • Designing for Guidance in Machine Learning

12. Softmax pixel weights
Source: https://www.tensorflow.org/get_started/mnist/beginners#softmax_regressionsOlivia Gunton • Designing for Guidance in Machine Learning

13. Decision Tree Classifier
f1 <= s1?
f2 <= s2? f3 <= s3?
T F
T TF F
Source: http://tjo-en.hatenablog.com/entry/2014/01/06/234155
Olivia Gunton • Designing for Guidance in Machine Learning

14. Decision Tree Classifier
[ 0.98955379 0.98873612 0.9923884 0.99102787] 0.990426545399
Olivia Gunton • Designing for Guidance in Machine Learning

15. Decision Tree Classifier
Olivia Gunton • Designing for Guidance in Machine Learning

16. Feature engineering
Source: https://www.jainworld.com/literature/story25.htmOlivia Gunton • Designing for Guidance in Machine Learning

17. pip install opencv-python import cv2
pip install scikit-image import skimage
Olivia Gunton • Designing for Guidance in Machine Learning
Feature engineering

18. Feature engineering
Source: https://tensorflow.rstudio.com/tutorial_mnist_beginners.htmlOlivia Gunton • Designing for Guidance in Machine Learning

19. Feature engineering: imports
Olivia Gunton • Designing for Guidance in Machine Learning

20. Feature engineering: preprocessing
Olivia Gunton • Designing for Guidance in Machine Learning

21. Preprocessing: reduce noise
before after
Olivia Gunton • Designing for Guidance in Machine Learning

22. Preprocessing: black-and-white
before after
Olivia Gunton • Designing for Guidance in Machine Learning

23. Preprocessing: skeletonize
before after
Olivia Gunton • Designing for Guidance in Machine Learning

24. Preprocessing: putting it all together
Olivia Gunton • Designing for Guidance in Machine Learning

25. Line detection
Olivia Gunton • Designing for Guidance in Machine Learning

26. Line detection: filter out redundant lines
Olivia Gunton • Designing for Guidance in Machine Learning

27. Line detection: lines to dicts
Olivia Gunton • Designing for Guidance in Machine Learning

28. Line detection: result
Olivia Gunton • Designing for Guidance in Machine Learning

29. Describe and count lines
Olivia Gunton • Designing for Guidance in Machine Learning

30. Corner detection
Olivia Gunton • Designing for Guidance in Machine Learning

31. Corner detection: result
Olivia Gunton • Designing for Guidance in Machine Learning

32. Describe and count corners
Olivia Gunton • Designing for Guidance in Machine Learning

33. Learn a model based on engineered features
Olivia Gunton • Designing for Guidance in Machine Learning

34. Learn a model based on engineered features
Olivia Gunton • Designing for Guidance in Machine Learning

35. Refining our model: improved training data
before after
Olivia Gunton • Designing for Guidance in Machine Learning

36. Refining our model: improved training data
Olivia Gunton • Designing for Guidance in Machine Learning

37. Refining our model: improved training data
Olivia Gunton • Designing for Guidance in Machine Learning

38. Constructing guidance
+1 num_horizontal_lines
+1 num_corners
+1 num_corners_upper_left
“Add a horizontal line,
ending in the upper left quadrant,
intersecting with your existing line”
Olivia Gunton • Designing for Guidance in Machine Learning

39. Constructing guidance
+1 num_horizontal_lines
+1 num_corners
+1 num_corners_upper_left
“Add a horizontal line,
ending in the upper left quadrant,
intersecting with your existing line”
Olivia Gunton • Designing for Guidance in Machine Learning

40. Communicating low confidence
+1 num_corners_upper_left
“(Optional) extend your horizontal
line into the upper left quadrant”
Olivia Gunton • Designing for Guidance in Machine Learning

41. Learning loop: observe user response to guidance
+1 num_horizontal_lines
+2 num_corners
+1 num_corners_upper_left
“Add a horizontal line,
ending in the upper left quadrant,
intersecting with your existing line”
Olivia Gunton • Designing for Guidance in Machine Learning

43. The Textio Data Team
Chris David Kevin Laurie Matt Nguyen Olivia
Olivia Gunton • Designing for Guidance in Machine Learning

44. The Textio Data Team
Chris David Kevin Laurie Matt Nguyen Olivia
Data
Scientist
ML
Engineer
Senior Data
Scientist
Senior ML
Engineer
Senior NLP
Engineer
(we’re hiring!)
Olivia Gunton • Designing for Guidance in Machine Learning

45. Thank you!
Questions?