The document outlines the goals of a presentation on deep learning, emphasizing the importance of excitement and understanding of the subject rather than detailing programming techniques. It discusses the significant advancements in deep learning, including applications like image recognition and generation, and the evolution of deep neural networks. The content also highlights various methodologies, tools, and the potential future directions of deep learning technology.

3. Goals
 My goal is not to teach you Cognitive Toolkit programming
 I will have some code samples
 But they are illustrative
 CNTK has great Tutorials for that: https://aka.ms/cntk_tut
 I want to get you excited for Deep Learning
 I want you to think about the possibilities
 I want you to understand how far it’s come
 In so short a time
 And what that means for the future

4. Agenda
 The old “state of the art”
 Deep Learning
 What happened? Why?
 What does Deep Learning enable
 Image Recognition, Transfer Learning
 Object Detection, Image Understanding
 Semantic Segmentation
 Image Generation, Style Transfer, Adversarial Networks
 Issues
 The Future

5. Evolution vs. Revolution
 Image Processing had been evolving for years
 Chaining complex “convolutions” helped
 However, it was only making incremental improvements
 State-of-the-art custom featurizers
 Could not perform complex image recognition tasks
 Still lacked performance close to humans on simple ones

6. Convolutional Filters
 Traditional Filter-based methods still work

8. 2012
WINTER

9. 2012
SUMMER

10. CAT FACES, 12 people, news articles…

14. FUEL + SPARK + ENGINE
COMPUTER
HORSEPOWER
+
NEW
MATH+MASSIVE
DATA

15. Massive Data
 Iris dataset: 150 instances, 4 classes
 ImageNet dataset: 1.2M instances, 1000 classes
 Facebook (as of four years ago): 3.5M images / day
 That’s pre-Snapchat!

16. 1965
1975

17. 1995
1985 5,000,000
2005
160,000,000
2015
7,600,000,000
( 2010 )
1,000,000,000

20. Deep Learning
 Shallow networks
 Deep networks (Inception v3 – ResNet152 wouldn’t fit)

21. Deep Neural Networks
 Deeper Networks
 Represent large non-linear function
 Easy to train by “pushing” weights in the
right direction
 “Smarter” Neurons
 CNNs learn those complex convolution filters
“automagically”
 RNNs learn to pay attention to the right bits
of history

23. Recognition Error – Shallow vs. Deep
shallow shallow
AlexNet, 8 layers 8 layers
GoogLeNet, 22 layers
(VGG @ 19/7.3)
ResNet152, 152 layers
0
20
40
60
80
100
120
140
160
2010 2011 2012 2013 2014 2015
layers error

25. * Feature visualization images from “Visualizing and Understanding Convolutional Neural Networks”, Zeiler and Fergus, ECCV 2014.

27. … in CNTK
def create_basic_model(input, out_dims):
with C.layers.default_options(init=C.glorot_uniform(), activation=C.relu):
net = C.layers.Convolution((5,5), 32, pad=True)(input)
net = C.layers.MaxPooling((3,3), strides=(2,2))(net)
net = C.layers.Convolution((5,5), 32, pad=True)(net)
net = C.layers.MaxPooling((3,3), strides=(2,2))(net)
net = C.layers.Convolution((5,5), 64, pad=True)(net)
net = C.layers.MaxPooling((3,3), strides=(2,2))(net)
net = C.layers.Dense(64)(net)
net = C.layers.Dense(out_dims, activation=None)(net)

30. What is it?
 DNNs are uncommonly good at learning abstractions
 Trained CNNs can be adapted to alternate domains
 Far less data required
 More data depending on how far the domain is from trained
 Custom Vision Service

31. …in CNTK
def create_model(model_uri, num_classes, input_features, feature_node_name, last_hidden_name, new_prediction_node_name='pr
base_model = C.load_model(download_from_uri(model_uri))
feature_node = C.logging.find_by_name(base_model, feature_node_name)
last_node = C.logging.find_by_name(base_model, last_hidden_name)
cloned_layers = C.combine([last_node.owner]).clone(C.CloneMethod.clone,
{feature_node: C.placeholder(name='features')})
feat_norm = input_features - C.Constant(114)
cloned_out = cloned_layers(feat_norm)
return C.layers.Dense(num_classes, activation=None, name=new_prediction_node_name) (cloned_out)

32. Example: Leak Detection
 https://aka.ms/leak_detection
 Convert audio into images
 Use FFT to convert to frequency domain
 Aggregate across time window and frequency into image:

33. Example: Using Custom Vision Service
 Detecting food using
mobile apps
 https://aka.ms/cvs_food
 “Layered” models to
reduce confusion

36. Regions of Interest
 Selection of bounding boxes
 Often random sizes and locations
 Major differentiator between methods
 Region Proposal Network (RPN)
 Initially done with separate SVM – super slow!
 Faster: Train class and RPN simultaneously
 YOLO9000
 Lots of tricks to make it fast

39. Sumo Wrestlers

41. Highlights
 Per-pixel classification
 Multiple Methods
 Mask-RCNN
 Multitask Network Cascades
 Recurrent Attention Networks
 Most involve simultaneous
training (like in RCNNs
RoIs

44. Style Transfer of Mona Lisa

45. How Does It Work?
 Trade off content vs. style
 Loss function
 L(x)=αC(x)+βS(x)+T(x)
 Content weighs more heavily in early layers
 Style weighs more heavily in later layers
 Tutorial on CNTK

46. …in CNTK
y = C.input_variable((3, SIZE, SIZE), needs_gradient=True)
z, intermediate_layers = model(y, layers)
content_activations = ordered_outputs(intermediate_layers, {y: [[content]]})
style_activations = ordered_outputs(intermediate_layers, {y: [[style]]})
style_output = np.squeeze(z.eval({y: [[style]]}))
total = (1-decay**(n+1))/(1-decay) # makes sure that changing the decay does not affect the magnitude of content/style
loss = (1.0/total * content_weight * content_loss(y, content)
+ 1.0/total * style_weight * style_loss(z, style_output)
+ total_variation_loss(y))

48. Generative Adversarial Networks
 Two networks
 Generator takes input, generates images
 Discriminator takes images, determines if fake
 Train both at same time – generator learns to fool discriminator

49. … in CNTK
def convolutional_generator(z):
…
h2 = C.layers.ConvolutionTranspose2D(gkernel,
num_filters=gf_dim*2,
strides=gstride,
pad=True,
output_shape=(s_h2, s_w2),
activation=None)(h1)
…
return C.reshape(h3, img_h * img_w)

50. Super Resolution GANs
 GANs Up-res
images
 Just shown
small and
large versions
so training
data is easy https://arxiv.org/pdf/1609.04802.pdf

51. GANs on Steroids
 CycleGANs is currently the best
 GANs fighting GANs
 Changing so fast it has its own zoo
 https://www.youtube.com/watch?v=IbjF5VjniVE
 Yann LeCun says
 “The most important idea in ML in the last 10 years”

52. How fast are GANs changing?
Credit: Bruno Gavranović

53. CycleGAN Results

55. Neural Networks Can Be Fooled
 https://blog.openai.com/adversarial-example-research/
 Networks are learning a large non-linear equation
 “Intelligent” noise finds “alternate” solution

56. Regulations, Liability, Social Issues
 NNs doing medical diagnoses
 How do you regulate? If it’s wrong, where does the fault lie?
 Garbage in, garbage out
 FaceApp “Whitewashing”
 Economic Displacement
 Job losses
 Privacy
 Better, faster facial recognition, voiceprinting
 #FakeNews
 Autogeneration of fake audio, video

58. Deep Learning Toolkits
 CNTK
 1-bit SGD for better distributed training
 TensorFlow
 Momentum
 Chainer
 Torch
 Caffe
 MXNet
 Azure Linux and Windows GPU VMs

60. What Does the Future Hold?
 Adversaries
 Adversarial networks
 Adversarial examples
 Reinforcement
 Deep RL
 “Training” DL
 Mixture of Networks
 RNNs + CNNs
 Conv + Deconv

61. Call to action
 CNTK Tutorials
 In GitHub: https://aka.ms/cntk_tut
 On Notebooks.azure.com/cntk
 Finding the latest papers
 arXiv (https://arxiv.org/) …
 And even more important arXiv Sanity Preserver: http://www.arxiv-
sanity.com/
 Re-visit de:code session recordings on Channel 9.
 Continue your education at
Microsoft Virtual Academy online.