Talk for NSLondon Meetup, Q2 2018 May 31, 2018 @ Facebook, London, UK https://www.meetup.com/NSLondon/events/249938395/

1. The State of ML for iOS
On the Advent of WWDC 2018
Meghan Kane, @meghafon
NSLondon
May 2018

2. !
Hey, I'm Meghan!
@meghafon
iOS Engineer @ Novoda Berlin

3. wwdc 2018

7. !
big picture
"
when is it practical to use ML for iOS?
#
what's available to us?
$
end-to-end examples
!

9. barriers to entry?
1. A large dataset
2. Access to high end compute power
3. PhD in machine learning
4. All the time in the world
...nope!

10. Is it practical for my app?
image classification
audio classification
speech recognition
text classification
gesture recognition
optical character recognition (OCR)
translation
voice synthesis

11. embrace idea generation
& experimentation

14. is it just hype?

15. machine learning is a powerful tool
but, it is still just another tool

16. how can we think about
ML as
!
developers?

17. Can this be solved without ML?
if so, choose that

18. ML vs not ML
basic unit of solving problem = function ("model")
ML: enabling a machine to learn function on its own
classify sign language alphabet images
not ML: explicitly defining function
determining if a number is even/odd

19. If you decide to use ML
still go with the simplest solution

20. Why do ML (predictions) on mobile?
→ low latency user experience
→ user privacy

21. What's available from Apple?
image classification of 1000 common categories
→ trees, animals, food, vehicles, people
→ SqueezeNet (5 MB), MobileNet (17 MB), Inception
V3 (95 MB), ResNet50 (103 MB), VGG16 (554 MB)
scene classification of 205 categories
→ airport terminal, bedroom, forest, coast
→ Places205-GoogLeNet (25 MB)

22. If not, train custom ML model
step 1: use framework for training
TensorFlow, keras, Turi Create , Caffe, etc
⚠
warning, there are a lot of them
step 2: convert to .mlmodel format (OSS)
→  coremltools github.com/apple/coremltools
→ tf-coreml github.com/tf-coreml

24. It has been
quite a year

27. beyond the cat/dog classifier (TM)

30. End-to-end Process as a developer?
0. Define problem
1. Collect data
2. Train ML model
3. Convert to coreml .mlmodel
4. Import into Xcode project
5. Predict using Core ML (+Vision) framework

31. Mobile specific concerns
size of model
time it takes to run predictions
supported layers

32. examples!

33. 0.Define problem
American Sign Language (ASL) alphabet classifier

34. 1.Collect data
!

35. 2. Train model

36. Quick Review: Deep Learning
neural network model with many layers
deep = many layers
-> deep neural network
Mobile Machine Learning 101: Glossary Jameson
Toole on Heartbeat blog

38. sometime way back in B.C.
people used to train deep neural network from
scratch

39. still some (more recent) time in B.C.
people stand on the shoulders of giants' work
utilizing transfer learning

40. enter.. transfer learning
!
use knowledge learned from source task (MobileNet)
--> to train target task (ASL classifier)

41. don't reinvent the wheel

42. Why Transfer Learning works
neural networks are universal approximators
in theory, they can approximate any function

43. how much data do i need?
depends on problem
just 100s images per category

44. where can i get it?
kaggle
google for them...
record video + extract frames (using e.g. FFmpeg)

46. what if i don't have enough?
data augmentation!
Deeplearning.ai: C4W2L10 Data Augmentation (~10
min video)

48. Let's start
training...
→ can we use Swift for Tensor
Flow?
→ for now, stick with regular
Tensor Flow

50. confession: this is in
python

56. Performance
so how did our training go?
~20 min to run on my MacBook
95.3% accuracy on the test data

59. 3. Convert
to .mlmodel
tf-coreml

61. It just works
!

62. 4. Import into Xcode project
drag + drop

63. It actually just
works

64. 5. Predict using Core ML (+Vision)
framework
vision + core ml

66. audio classification
0.Define problem
1. Gather data
2. Train ML model

67. 0.Define problem
Audio classifier of urban sounds
air conditioner, car horn, children playing, drilling,
siren, etc

68. 1.Gather data
UrbanSound 8K open dataset
Urban Sound Datasets, NYU CUSP

69. should we use raw audio (.wav)?
no, it's too computationally expensive

70. convert wav to spectrogram
represent audio as image (3 dimensions)
1st dimension: time (x-axis)
2nd dimension: frequency (y-axis)
3rd dimension: sound intensity (color)

76. 2. Train Model

77. Performance
so how did our training go?
~1 hour to run on my MacBook
77.1% accuracy on the test data

81. what to
focus on

82. Where to find inspiration
look at open datasets
read research papers!
follow heartbeat blog, openAI

83. Reproduce results
research papers often include this
make sure to do the same if you publish
check licensing + attribute proper credit

84. Looking forward to the future
ML interpretability
swift for TensorFlow

85. Review
!
big picture
"
when is it practical to use ML for iOS?
#
what's available to us?
$
end-to-end examples

86. Attributions & Mentions (1/4)
Apple Machine Learning
WWDC 2017 Videos
TensorFlow for Poets Google codelabs tutorial
Apple coremltools GitHub repo
tf-coreml GitHub repo: TensorFlow->core ml
converter

87. Attributions & Mentions (2/4)
Heartbeat by fritz.ai blog: Machine Learning at the
edge
ASL Datasets
Kaggle Sign Language MNIST
Urban Sound Datasets, NYU CUSP
deeplearning.ai course: Data Augmentation

88. Attributions & Mentions (3/4)
Swift for TensorFlow GitHub repo
Dockerized Swift for TF GitHub repo, Alexis Gallager
themorningpaper by Adrian Colyer
OpenAI Research
"The Building Blocks of Interpretability" Google: C.
Olah et al

89. Attributions & Mentions (4/4)
"Strategically Ignorant" Devon Zuegel
"Transfer Learning of Temporal Information for Driver
Action Classification" J. Lemley et al
"Transfer Learning for Sound Classification"
TataLab

90. Further Learning (1/3)
fast.ai Deep Learning course
My Udacity Core ML course
machinethink,
!
ML for iOS blog by Matthijs
Hollemans
TensorFlow Dev Summit 2018 Videos
TensorFlow playground

91. Further Learning (2/3)
Building Mobile Apps w/ Tensor Flow Pete Warden
Neural Networks & Deep Learning Michael
Nielsen
Stanford's Computer Vision course (CS231n)

92. Further Learning (3/3)
"Distilling the Knowledge in a Neural Network"
Geoffrey Hinton et al.
"Transfer Learning - Machine Learning's Next
Frontier"
!
Sebastian Ruder
"Transfer learning for music classification and
regression tasks"
!
Keunwoo Choi et al.

93. Thank you
Keep in touch!
twitter: @meghafon