8. THEORY
▸ A way to compute a function inspired by the human brain
▸ Much simpler than the brain!
▸ Input -> Network -> Output
▸ Comprised of many neurons
▸ These neurons are interconnected
▸ Some are “inputs” and some are “outputs”
9. THEORY
DEEP NETWORKS
▸ Nowadays we have so many neurons that we no longer describe the networks
with them
▸ Instead, we have layers that contain many neurons
▸ When layers are connected to each other, the constituent neurons get
connected
▸ Neurons in different types of layers behave differently from those in other
types
10. TEXT
CONVOLUTIONAL NEURAL NETWORKS
▸ CNNs use a layer type called a Spatial Convolution to operate on images
▸ This is a way of connecting neurons so that nearby pixels in 2D images are
connected to nearby neurons
▸ The operation the neurons perform is called convolution and is a generalized
technique for manipulating images
▸ The benefits are that local information gets local connections and the
operation itself is very powerful
11. TEXT
RECURRENT NEURAL NETWORKS
▸ RNNs use feedback to predict the future!
▸ Most networks are pure functions that process inputs and produce outputs
▸ RNNs feed outputs back into the network to model time
▸ It’s scarily effective
13. THEORY
TRAINING
▸ We train the network by showing it a bunch of inputs and desired outputs
▸ The training algorithm is called back propagation and involves a lot of
number crunching
▸ Neurons assign weights to the neurons they’re connected to
▸ These weights control how much the neighbors influence that neuron
▸ Training is the process of determining these weights
▸ The number of times a pair have been reused is called an epoch
15. THEORY
GENERALIZING
▸ What’s the point of training a network if we already have a bunch of inputs and
outputs?
▸ The hope is that the network will learn how to solve problems it hasn’t yet seen
▸ When we train, we always reserve a batch of validation input and outputs that it
never sees
▸ We then use those inputs and outputs to rate the network
17. PRACTICE
NEURAL NETWORK DISTRIBUTION
▸ While the concepts are universal…
▸ Neural networks are released as source code that can train and execute the network
▸ The code can be in any number of languages and use any number of support libraries
▸ Python with TensorFlow
▸ Lua with Torch
▸ Networks may only work with some hardware
▸ NVIDIA CUDA is used a lot (they invest in academia)
▸ Cloud solutions exist varying from virtualized hardware to proprietary languages
21. PRACTICE
INSTALLATION
▸ Install NVIDIA drivers
▸ CUDA - GPU programming SDK
▸ CUDNN - GPU libraries to help writing nets
▸ I did this on Mac and Linux
▸ Install Torch
▸ Lua libraries that can use CUDNN
▸ Install pix2pix
24. PRACTICE
TRAINING COMPLETE
▸ We get a trained
model file
▸ Contains the
structured of the
network
▸ Along with the
learned weights
25. PRACTICE
RUNNING THE NETWORK
▸ Now that the network is trained, we can run it against new inputs
▸ Put images you want to test in a special folder
▸ Run test.lua instead of train.lua
▸ Now you have outputs!
28. ZOOM AND ENHANCE
GOOD TRAINING EXAMPLES
▸ It some fooling around to learn what the network can and can’t do
▸ You can’t just throw images at it and hope for the best
▸ You must spend time to give it good training examples
▸ Inputs and outputs should only differ by what you want the network to
learn
▸ Other differences will cause slow or impossible learning
▸ Examples: aspect ratio, cropping / region of interest, backgrounds
29. ZOOM AND ENHANCE
FACE ZOOMING EXAMPLE GENERATION
▸ Narrowed the task down to zooming in on faces
▸ Wrote an app that extracts faces from images using Apple’s CoreImage
framework
▸ Wrote another app that down samples those faces by 8x, then 16x
▸ This only simulates the problem of Z&E since noise is drastically reduced by
this downsampling
▸ Ideally, my inputs and outputs would be taken with the same camera using
two different zoom levels. But alas…
37. FUTURE
MOBILE LIBRARIES
▸ TensorFlow is a C++ library that runs on Android & iOS
▸ Miguel de Icaza is binding TensorFlow to .NET (hope it works on mobile!)
▸ https://github.com/migueldeicaza/TensorFlowSharp
▸ Apple includes Metal Performance Shaders that contains basic CNN routines
▸ I’m porting Torch, for now…