This document outlines methods for passive stereo vision, from traditional to deep learning-based approaches. It discusses modeling from multiple views, stereo matching techniques like dense correspondence search and cost aggregation. Traditional methods include semi-global matching and energy minimization using graph cuts or belief propagation. Deep learning has also been applied to learn sparse depth representations and end-to-end stereo matching. The document provides an overview of techniques and challenges in passive stereo vision.

1. Passive Stereo Vision: From Traditional
to Deep Learning-based Methods
YU HUANG
SUNNYVALE, CALIFORNIA
YU.HUANG07@GMAIL.COM

2. Outline
• Modeling from multiple views
• Stereo matching
• constraints in stereo vision
• difficulties in stereo vision
• pipeline of stereo matching
• state of art methods
• Quality metric of stereo matching
• census transform and hamming distance
• guided filter in cost aggregation (volume)
• semi-global matching
• ELAS: efficient large scale stereo
• stereo matching as energy minimization
• dynamic programming/graph cut/belief propagation
• phase matching for stereo vision
• disparity refinement
• Multiple cameras/views
• Learning sparse represent. of depth maps
• Stereopsis via deep learning
• Deep learning of depth (and motion)
• Stereo matching by CNN
• Constant Highway Networks and Reflective
Confidence Learning;
• Efficient Deep Learning for Stereo Matching;
• E2e Learning of Geometry and Context;
• Appendix A: Depth from an image by learning
• Appendix B: Learning and optimization

3. Modeling from Multiple Views in Computer Vision
time
# cameras
photograph
binocular stereo
trinocular stereo
multi-baseline stereo
camcorder
human vision
camera dome
two frames ...
...

4. Binocular Stereo
• Given a calibrated binocular stereo pair, fuse it to produce a depth image
image 1 image 2
Dense depth map
Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923

5. Basic Stereo Matching Algorithm
• For each pixel in the first image
Find corresponding epipolar line in the right image
Examine all pixels on the epipolar line and pick the best match
Triangulate the matches to get depth information
• Simplest case: epipolar lines are corresponding
scanlines;
• If necessary, rectify the two stereo images to transform
epipolar lines into scanlines

6. Depth from Disparity
f
x x’
Baseline
B
z
O O’
X
f
z
fB
xxdisparity


Disparity is inversely proportional to depth!
• Stereo camera calibration (focal length and baseline are known)
• Image planes of cameras parallel to each other and to the baseline
• Camera centers are at same height
• Focal lengths are the same
• Then, epipolar lines fall along the horizontal scan lines of the images

7. Calibration
• Find the intrinsic and extrinsic parameters of a camera
◦ Extrinsic parameters: the camera’s location and orientation in the world.
◦ Intrinsic parameters: the relationships between pixel coordinates and camera coordinates.
• Work of Roger Tsai and work of Zhengyou Zhang are influential: 3-D node setting and 2-d plane
• Basic idea:
◦ Given a set of world points Pi and their image coordinates (ui,vi)
◦ find the projection matrix M
◦ And then find intrinsic and extrinsic parameters.
• Calibration Techniques
◦ Calibration using 3D calibration object
◦ Calibration using 2D planer pattern
◦ Calibration using 1D object (line-based calibration)
◦ Self Calibration: no calibration objects
◦ Vanishing points from for orthogonal direction

8. Calibration
• Calibration using 3D calibration object:
◦ Calibration is performed by observing a calibration object whose geometry in 3D space is known with very good precision.
◦ Calibration object usually consists of two or three planes orthogonal to each other, e.g. calibration cube
◦ Calibration can also be done with a plane undergoing a precisely known translation (Tsai approach)
◦ (+) most accurate calibration, simple theory
◦ (-) more expensive, more elaborate setup
• 2D plane-based calibration (Zhang approach)
◦ Require observation of a planar pattern at few different orientations
◦ No need to know the plane motion
◦ Set up is easy, most popular approach
◦ Seems to be a good compromise.
• 1D line-based calibration:
◦ Relatively new technique.
◦ Calibration object is a set of collinear points, e.g., two points with known distance, three collinear points with known
distances, four or more…
◦ Camera can be calibrated by observing a moving line around a fixed point, e.g. a string of balls hanging from the ceiling!
◦ Can be used to calibrate multiple cameras at once. Good for network of cameras.

9. Fundamental Matrix
Let p be a point in left image, p’ in right image
Epipolar relation
◦ p maps to epipolar line l’
◦ p’ maps to epipolar line l
Epipolar mapping described by a 3x3 matrix F
It follows that
l’l
p p’
This matrix F is called
• the “Essential Matrix” E
– when image intrinsic parameters are known
• the “Fundamental Matrix”
– more generally (uncalibrated case)
Can solve for F from point correspondences
• Each (p, p’) pair gives one linear equation in entries of F
• 8/5 points give enough to solve for F/E (8/5-point algo)

10. Planar Rectification
Bring two views
to standard stereo setup
(moves epipole to )
(not possible when in/close to image)
~ image size
(calibrated)
Distortion minimization
(uncalibrated)

12. Polar re-parameterization around epipoles
Requires only (oriented) epipolar geometry
Preserve length of epipolar lines
Choose  so that no pixels are compressed
original image rectified image
Polar Rectification
Works for all relative motions
Guarantees minimal image size
Determine the common region from the extremal
epipolar lines and the location of epiole: e’F=0
Select half epipolar lines moving around the epipole
Construct rectified image line by line

14. Matching cost
disparity
Left Right
scanline
Correspondence Search
• Slide a window along the right scanline and compare contents of that window with the
reference window in the left image
• Matching cost: SSD or normalized correlation

15. Constraints in Stereo Vision
• Color constancy
• Lambertian surface assumption;
•Epipolar geometry
• Scanline as epipolar line for rectifed pair;
• Uniqueness
• For any point in one image, there should be at
most one matching point in the other image;
• Ordering
• Corresponding points should be in the same
order in both views;
• Smoothness
• Disparities to change slowly (the most part).
Epipolar plane
Epipolar line for pEpipolar line for p’
Uniqueness Ordering

16. Difficulties in Stereo Vision
• Photometric distortions and noise;
• Foreshortening;
• Perspective distortions;
• Uniform/ambiguous regions;
• Repetitive/ambiguous patterns;
• Transparent objects;
• Occlusions and discontinuities.

17. Pipeline of Stereo Matching Methods
• Pre-processing: compensate for photometric distortion;
• LoG, Census transform, phase only(DCT or WT), histogram equalization/matching, isotropic diffusion, …
• Cost computation:
• Absolute difference, squared difference, weighted difference, SAD, SSD, SWD, ZMNCC, …
• Cost aggregation:
• Bilateral filter, guided filter, non local, segment tree,...
• Disparity computation/optimization
• Integral image, box filtering, …
• Local (fast), global (slow), semi-global, …
• Disparity refinement
• Sub pixel interpolation, median filter, cross check (left-right consistency check) and occlusion filling.

18. State-of-Art Stereo Matching Methods
• Local method
• Look at one image patch at at time
• Solve many small problems independently
• Faster, less accurate, usually works for high texture
• Needs enough texture in a patch to disambiguate
• Global method
• Look at the whole image
• Solve one large problem
• Slower, more accurate, works up to medium texture
• Propagates estimates from textured to untextured regions
• Sparse point-based method
• Still works for low textured regions, hard to handle ambiguous regions
• Semi-global method
• SGM (semi-global-matching), 2-d search to 1-d search along 8/16 directions.

19. Quality Metrics in Stereo Matching (Passive)
• General objective approaches:
• Compute error statistics w.r.t. some ground truth data;
• RMS (root-mean-squared) error (in disparity units) btw. computed disparity dC (x, y) and ground truth dT (x, y);
• Percentage of bad matching pixels;
• Select the following areas support the analysis of matching results
• textureless regions;
• occluded regions;
• depth discontinuity regions.
• Evaluate synthetic image by warping the reference with disparity map;
• Forward warp the reference image by the computed disparity map;
• Inverse warp a new view by the computed disparity map.
• Subjective evaluation

20. Census Transform and Hamming Distance
• Census transform converts relative intensity difference to 0 or 1 and deforms 1 dimensional vector as much as
window size of census transform;
• Census transform makes data of (image size * vector size).
• Modified CTW: compared with the mean rather than the central pixel;
• Hamming distance of CT vectors with correlation windows used to find matched patches;
• Advantage: robustness to radiometric distortion, vignetting, lighting, boundaries and noise.
210159998639
198170326747
45677810298
304033115109
393126130121
11111
11000
00X11
00011
00011
111111100000110001100011
Census transform window (CTW)
Height
Width
Height
Width
(Square size of CTW)-1

21. CTonintensity
&gradientresp.
Originalgradient

22. Guided Filter in Cost Aggregation for Stereo Matching
• Idea: stereo match as labeling, a spatially smooth labeling with label transitions aligned with color edges;
• Edge preserving filter: WLS, Anisotropic diffusion, bilateral filter, total variation filter, guided filter, ...
• Guided filter works better than bilateral filter;
•
• Cost volume filtering with guided filter works like segmentation implicitly;
Wi,j : The filter weights depend on the guidance image IC’ : the filtered cost volume

24. PatchMatch Stereo
• Idea: First a random initialization of disparities and plane para.s for each pix. and
update the estimates by propagating info. from the neighboring pix.s;
• Spatial propagation: Check for each pix. the disparities and plane para.s for left and
upper neighbors and replace the current estimates if matching costs are smaller;
• View propagation: Warp the point in the other view and check the corresponding
etimates in the other image. Replace if the matching costs are lower;
• Temporal propagation: Propagate the information analogously by considering the
etimates for the same pixel at the preceding and consecutive video frame;
• Plane refinement: Disparity and plane para.s for each pix. refined by generat. random
samples within an interval and updat. estimates if matching costs reduced;
• Post-processing: Remove outliers with left/right consistency checking and weighted
median filter; Gaps are filled by propagating information from the neighborhood.

25. PatchMatch Stereo

26. Semi-Global Matching for Stereo Computation
• Semi-global matching approximates a global optimization by combining several local optimization steps;
• Minimizing E(D) in a two-dimensional manner would be very costly, while SGM simplifies it by traversing
one-dimensional paths and ensures the constraints with respect to these explicit directions;
• At least 8 paths (16 suggested), like horizontal, vertical and diagonal orientations;
• For instance, cost aggregation along a horizontal path as
• Pixel-based cost computation by mutual information as
• Left-right consistency check for occlusion detection and disparity propagation for hole filling.
• To accelerate the process, down-sampled image pairs are used for disparity estimation.
a small penalty P1 a large penalty P2 for large disparity changes

28. ELAS: Efficient Large Scale Stereo Matching

29. ELAS: Efficient Large Scale Stereo Matching
• Prior model:
• Likelihood model:
• Posterior can be factorized by the Bayes rule as
• Likelihood calculated along the epipolar line as
• Disparity estimation as MAP:
• To minimize an energy function
A mean function linking the support points and the observations

31. Stereo as Energy Minimization
• Find disparities d that minimize an energy function
• Simple pixel / window matching
= SSD distance between windows I(x, y) and J(x, y + d(x,y))
I(x, y) J(x, y)
y = 141
C(x, y, d); the disparity space image (DSI)
x
d
• Choose the minimum of each column in the DSI independently:

32. Dynamic Programming (DP) in Stereo Matching
• Can minimize E(d) independently per scanline using dynamic programming (DP);
leftS
rightS
Left
occlusion
t
q
Right
occlusion
s p
occlC
occlC corrC
Three cases:
• Sequential – cost of match
• Left occluded – cost of no match
• Right occluded – cost of no match
Left image
Right image I
I
• DP yields the optimal path through grid,
the best set of matches for the ordering
constraint in scan-line stereo.

33. d1
d2
d3
• Graph Cut
• Delete enough edges so that
• each pixel is connected to exactly one label node
• Cost of a cut: sum of deleted edge weights
• Finding min cost cut equivalent to finding global minimum of energy
function
Energy Minimization via Graph Cuts Labels
(disparities)
edge weight
edge weight
• What defines a good stereo correspondence?
• 1. Match quality
• Want each pixel to find a good match in the other image
• 2. Smoothness
• If two pixels are adjacent, they should (usually) move
about the same amount
{
{
match cost smoothness cost
“Potts model”
L1 distance
Graph Cut: convert multi-way cut into a seq. of binary cut

35. Model Stereo Vision by MRF and Solution by Belief Propagation
• Allows rich probabilistic models for images.
• But built in a local, modular way. Learn local relationships, get global effects out.
disparity
images
Disparity-disparity
compatibility
function
neighboring
disparity nodes
local
observationsImages-disparity
compatibility
function
 FY
i
ii
ji
ji
yxxx
Z
yxP ),(),(1
),(
,
BELIEFS: Approximate posterior marginal distributions
neighborhood of node i
MESSAGES: Approximate sufficient statistics
I. Belief Update (Message Product)
II. Message Propagation (Convolution)

36. Hierarchical Belief Propagation (HBP) and Constant Space HBP
• HBP works in a coarse-to-fine manner;
• (a) initialize the messages at the coarsest level to all zeros;
• (b) apply BP at the coarsest level to iteratively refine the messages;
• (c) use refined messages from the coarser level to initialize the messages for the next level.
• Constant space HBP relies on that, only a small number of disparity levels and the corresponding
message values are needed at each pixel to losslessly reconstruct the BP messages;
• Apply the coarse-to-fine (CTF) scheme to both spatial and depth domain, i.e. gradually reduce the number of
disparity levels as the messages propagate in CTF;
• Re-computes the data term at each level (not each iter.);
• Slower 9/8, but memory does not grow with max disp;
• Energy computed only once at the finest level;
• Gradually reduce the disparity levels in CTF.
• The closer the messages are to the fixed points, the fewer the required disparity levels; Then, CSBP refines the
messages hierarchically to approach the fixed points.

38. Phase Matching in Frequency or WT Domain
• Phase reflects the structure information of the signal and inhibit the HF noise effect;
• Phase singularity is a problem;
• Local phase information as the primitive;
• Wavelet transform builds a hierarchical framework for multi-level coarse-to-fine processing;
• Stereo matching (disparity) with phase separation and instantaneous frequency of signals:
• Dynamic programming (DP) used for global optimization (occlusion handling) in stereo matching ;
• Phase is not uniformly stable;
• Smoothness constraints;
• Discontinuities detection;
• Multiple resolution solution:
• 1. top level: control points with feature matching, apply DP;
• 2. middle level: interpolation, apply DP;
• 3. bottom level: sub-pixel precision.
Local phaseDisparity Left/Right images

39. Original Phase matching Phase matching with DP

40. Disparity/Depth Refinement
• Sub-pixel refinement: real valued disparities may be obtained by approximating the cost
function locally using a parabola;
• Left-Right Consistency Check: outlier detection by difference;
• By computing a disparity for every pixel of the left image (left to right);
• by computing a disparity for every pixel of the right image (right to left);
• Segmentation can be used for outlier identification.
• Occlusion filling:
• Occlusion detection;
• Background expansion;
• Inpainting.
• Discontinuities smoothing:
• Bilateral filtering.

41. Multiple Cameras
Multi-baseline stereo
use the third view to verify depth estimates

42. Spatial Temporal Video Disparity Estimation
• The important problem of extending to video is flickering;
• Typical methods:
• Spatial temporal consistency: smoothing in the space-time volume;
• Post-processing of disparity maps by applying a median filter along the flow fields;
• Spatial-temporal cost aggregation and solved by local/global optimization methods;
• Joint disparity and flow estimation;
• SGM-based, as an instance;
• Modeled with MRF and solved by global optimization.
• Scene flow: 2D motion field along with 1D disparity change field.
• Dense method is very computationally expensive;
• Sparse method relies on heavily initial sparse correspondence success.

43. Sparse Coding
Sparse coding (Olshausen & Field, 1996).
Originally developed to explain early visual processing in the brain (edge detection).
Objective: Given a set of input data vectors learn a dictionary of
bases such that:
Each data vector is represented as a sparse linear combination of bases.
Sparse: mostly zeros

44. Predictive Sparse Coding
Recall the objective function for sparse coding:
Modify by adding a penalty for prediction error:
◦ Approximate the sparse code with an encoder
PSD for hierarchical feature training
◦ Phase 1: train the first layer;
◦ Phase 2: use encoder + absolute value as 1st feature extractor
◦ Phase 3: train the second layer;
◦ Phase 4: use encoder + absolute value as 1st feature extractor
◦ Phase 5: train a supervised classifier on top layer;
◦ Phase 6: optionally train the whole network with supervised BP.

45. Methods of Solving Sparse Coding
Greedy methods: projecting the residual on some atom;
◦ Matching pursuit, orthogonal matching pursuit;
L1-norm: Least Absolute Shrinkage and Selection Operator (LASSO);
◦ The residual is updated iteratively in the direction of the atom;
Gradient-based finding new search directions
◦ Projected Gradient Descent
◦ Coordinate Descent
Homotopy: a set of solutions indexed by a parameter (regularization)
◦ LARS (Least Angle Regression)
First order/proximal methods: Generalized gradient descent
◦ solving efficiently the proximal operator
◦ soft-thresholding for L1-norm
◦ Accelerated by the Nesterov optimal first-order method
Iterative reweighting schemes
◦ L2-norm: Chartand and Yin (2008)
◦ L1-norm: Cand`es et al. (2008)

46. Strategyof Dictionary Selection
• What D to use?
• A fixed overcomplete set of basis: no adaptivity.
• Steerable wavelet;
• Bandlet, curvelet, contourlet;
• DCT Basis;
• Gabor function;
• ….
• Data adaptive dictionary – learn from data;
• K-SVD: a generalized K-means clustering process for Vector Quantization (VQ).
• An iterative algorithm to effectively optimize the sparse approximation of signals in a learned
dictionary.
• Other methods of dictionary learning:
• non-negative matrix decompositions.
• sparse PCA (sparse dictionaries).
• fused-lasso regularizations (piecewise constant dictionaries)
• Extending the models: Sparsity + Self-similarity=Group Sparsity

47. Learning Sparse Representation in Depth Maps
• Sparse representations learned from
Middlebury database disparity maps;
• Then they are exploited in a two-layer
graphical model for inferring depth from
stereo, by including a sparsity prior on
the learned features;
 The first layer solved using an existing MRF-
based stereo matching algorithm;
 The second layer is solved using the non-
stationary sparse coding algorithm.

48. Learning Sparse Representation in Depth Maps
(c) Graph cut (d) GC + Sparse coding

49. Deep Learning
Representation learning attempts to automatically learn good features or representations;
Deep learning algorithms attempt to learn multiple levels of representation of increasing
complexity/abstraction (intermediate and high level features);
Become effective via unsupervised pre-training + supervised fine tuning;
◦ Deep networks trained with back propagation (without unsupervised pre-training) perform worse than
shallow networks.
Deal with the curse of dimensionality (smoothing & sparsity) and over-fitting (unsupervised, regularizer);
Semi-supervised: structure of manifold assumption;
◦ labeled data is scarce and unlabeled data is abundant.

50. Why Deep Learning?
Supervised training of deep models (e.g. many-layered Nets) is too hard (optimization
problem);
◦ Learn prior from unlabeled data;
Shallow models are not for learning high-level abstractions;
◦ Ensembles or forests do not learn features first;
◦ Graphical models could be deep net, but mostly not.
Unsupervised learning could be “local-learning”;
◦ Resemble boosting with each layer being like a weak learner
Learning is weak in directed graphical models with many hidden variables;
◦ Sparsity and regularizer.
Traditional unsupervised learning methods aren’t easy to learn multiple levels of
representation.
◦ Layer-wised unsupervised learning is the solution.
Multi-task learning (transfer learning and self taught learning);
Other issues: scalability & parallelism with the burden from big data.

51. Multi Layer Neural Network
A neural network = running several logistic regressions at the same time;
◦ Neuron=logistic regression or…
Calculate error derivatives (gradients) to refine: back propagate the error derivative through model
(the chain rule)
◦ Online learning: stochastic/incremental gradient descent
◦ Batch learning: conjugate gradient descent

52. Problems in MLPs
Multi Layer Perceptrons (MLPs), one feed-forward neural network, were popularly used for decades.
Gradient is progressively getting more scattered
◦ Below the top few layers, the correction signal is minimal
Gets stuck in local minima
◦ Especially start out far from ‘good’ regions (i.e., random initialization)
In usual settings, use only labeled data
◦ Almost all data is unlabeled!
◦ Instead the human brain can learn from unlabeled data.

53. Convolutional Neural Networks
CNN is a special kind of multi-layer NNs applied to 2-d arrays (usually images), based on spatially localized
neural input;
◦ local receptive fields(shifted window), shared weights (weight averaging) across the hidden units, and often, spatial
or temporal sub-sampling;
◦ Related to generative MRF/discriminative CRF:
◦ CNN=Field of Experts MRF=ML inference in CRF;
◦ Generate ‘patterns of patterns’ for pattern recognition.
Each layer combines (merge, smooth) patches from previous layers
◦ Pooling /Sampling (e.g., max or average) filter: compress and smooth the data.
◦ Convolution filters: (translation invariance) unsupervised;
◦ Local contrast normalization: increase sparsity, improve optimization/invariance.
C layers convolutions,
S layers pool/sample

54. Convolutional Neural Networks
Convolutional Networks are trainable multistage architectures composed of multiple stages;
Input and output of each stage are sets of arrays called feature maps;
At output, each feature map represents a particular feature extracted at all locations on input;
Each stage is composed of: a filter bank layer, a non-linearity layer, and a feature pooling layer;
A ConvNet is composed of 1, 2 or 3 such 3-layer stages, followed by a classification module;
◦ A fully connected layer: softmax transfer function for posterior distribution.
Filter: A trainable filter (kernel) in filter bank connects input feature map to output feature map;
Nonlinearity: a pointwise sigmoid tanh() or a rectified sigmoid abs(gi•tanh()) function;
◦ In rectified function, gi is a trainable gain parameter, might be followed a contrast normalization N;
Feature pooling: treats each feature map separately -> a reduced-resolution output feature map;
Supervised training is performed using a form of SGD to minimize the prediction error;
◦ Gradients are computed with the back-propagation method.
Unsupervised pre-training: predictive sparse decomposition (PSD), then supervised fine-tuning.
* is discrete convolution operator

56. LeNet (LeNet-5)
A layered model composed of convolution and subsampling operations followed by a holistic representation
and ultimately a classifier for handwritten digits;
Local receptive fields (5x5) with local connections;
Output via a RBF function, one for each class, with 84 inputs each;
Learning by Graph Transformer Networks (GTN);

57. AlexNet
A layered model composed of convol., subsample., followed by a holistic
representation and all-in-all a landmark classifier;
Consists of 5 convolutional layers, some of which followed by max-pooling
layers, 3 fully-connected layers with a final 1000-way softmax;
Fully-connected layers: linear classifiers/matrix multiplications;
ReLU are rectified-linear nonlinearities on layer output, can be trained
several times faster;
Local (contrast) normalization scheme aids generalization;
Overlapping pooling slightly less prone to overfitting;
Data augmentation: artificially enlarge the dataset using label-preserving
transformations;
Dropout: setting to zero output of each hidden neuron with prob. 0.5;
Trained by SGD with batch # 128, momentum 0.9, weight decay 0.0005.

58. The network’s input is 150,528-dimensional, and the number of neurons in the network’s
remaining layers is given by 253,440–186,624–64,896–64,896–43,264-4096–4096–1000.

59. MattNet
Matthew Zeiler from the startup company “Clarifai”, winner of ImageNet Classification in 2013;
Preprocessing: subtracting a per-pixel mean;
Data augmentation: downsampled to 256 pixels and a random 224 pixel crop is taken out of the image and
randomly flipped horizontally to provide more views of each example;
SGD with min-batch # 128, learning rate annealing, momentum 0.9 and dropout to prevent overfitting;
65M parameters trained for 12 days on a single Nvidia GPU;
Visualization by layered DeconvNets: project the feature activations back to the input pixel space;
◦ Reveal input stimuli exciting individual feature maps at any layer;
◦ Observe evolution of features during training;
◦ Sensitivity analysis of the classifier output by occluding portions to reveal which parts of scenes are important;
DeconvNet attached to each of ConvNet layer, unpooling uses locations of maxima to preserve structure;
Multiple such models were averaged together to further boost performance;
Supervised pre-training with AlexNet, then modify it to get better performance (error rate 14.8%).

60. Architecture of an eight layer ConvNet model. Input: 224 by 224 crop of an image (with 3 color planes). # 1-5
layers Convolution: 96 filters, 7x7, stride of 2 in both x and y. Feature maps: (i) via a rectified linear function, (ii)
3x3 max pooled (stride 2), (iii) contrast normalized 55x55 feature maps. # 6-7 layers: fully connected, input in
vector form (6x6x256 = 9216 dimensions). The final layer: a C-way softmax function, C - number of classes.

61. Top: A deconvnet layer (left) attached to
a convnet layer (right). The deconvnet
will reconstruct approximate version of
convnet features from the layer beneath.
Bottom: Unpooling operation in the
deconvnet, using switches which record
the location of the local max in each
pooling region (colored zones) during
pooling in the convnet.

62. Oxford VGG Net: Very Deep CNN
Networks of increasing depth using an architecture with very small (3×3) convolution filters;
◦ Spatial pooling is carried out by 5 max-pooling layers;
◦ A stack of convolutional layers followed by three Fully-Connected (FC) layers;
◦ All hidden layers are equipped with the rectification ReLU non-linearity;
◦ No Local Response Normalisation!
Trained by optimising the multinomial logistic regression objective using SGD;
Regularised by weight decay and dropout regularisation for the first two fully-connected layers;
The learning rate was initially set to 10−2, and then decreased by a factor of 10;
For random initialisation, sample the weights from a normal distribution;
Derived from the publicly available C++ Caffe toolbox, allow training and evaluation on multiple GPUs
installed in a single system, and on full-size (uncropped) images at multiple scales;
Combine the outputs of several models by averaging their soft-max class posteriors.

63. The depth of the configurations increases from the left (A) to the
right (E), as more layers are added (the added layers are shown in
bold). The convolutional layer parameters are denoted as
“conv<receptive field size> - <number of channels>”. The ReLU
activation function is not shown for brevity.

64. GoogleNet
Questions:
◦ Vanishing gradient?
◦ Exploding gradient?
◦ Tricky weight initialization?
Deep convolutional neural network architecture codenamed Inception;
◦ Finding out how an optimal local sparse structure in a convolutional vision network can be approximated and
covered by readily available dense components;
◦ Judiciously applying dimension reduction and projections wherever the computational requirements would increase
too much otherwise;
Increasing the depth and width of the network but keeping the computational budget constant;
◦ Drawbacks: Bigger size typically means a larger number of parameters, which makes the enlarged network more
prone to overfitting and the dramatically increased use of computational resources;
◦ Solution: From fully connected to sparsely connected architectures, analyze the correlation statistics of the
activations of the last layer and clustering neurons with highly correlated outputs.
◦ Based on the well known Hebbian principle: neurons that fire together, wire together;
Trained using the DistBelief: distributed machine learning system.

65. Inception module (with dimension reductions)

66. Convolution
Pooling
Softmax
Other
Problems with training deep architectures?
Network in a network in a network
9 Inception modules

67. PReLU Networks at MSR
A Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit;
◦ PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk;
◦ Allow negative activations on the ReLU function with a control parameter a learned Adaptively;
◦ Resolve diminishing gradient problem for very deep neural networks (> 13 layers) ;
Derive a robust initialization method better than “Xavier” (normalization) initialization;
Also use Spatial Pyramid Pooling (SPP) layer just before the fully connected layers;
Can train extremely deep rectified models and investigate deeper or wider network architectures;
ReLU vs. PReLU Note: μ is momemtum, ϵ is learning rate.

68. PReLU Networks at MSR
Performance: 4.94% top-5 test error on the ImageNet 2012 Classification dataset;
◦ ILSVRC 2014 winner (GoogLeNet, 6.66%);
Adopt the momentum method in BP training;
Mostly initialized by random weights from Gaussian distr.;
Investigate the variance of the FP responses in each layer;
Consider a sufficient condition in BP:
◦ The gradient is not exponentially large/small.

69. Architectures of large models
PReLU Networks

70. Batch Normalization at Google
Normalizing layer inputs for each mini-batch to handle saturating
nonlinearities and covariate shift;
◦ Internal Covariate Shift (ICS): the change in the distribution of network activations
due to the change in network parameters during training;
◦ Whitening to reduce ICS: linear transform to have zero means and unit variances, and
decorrelated;
◦ Fix the means and variance of layer inputs (instead of whitening jointly the features in
both I/O);
◦ Batch normalizing transform applied for activation over a mini-batch;
◦ BN transform is differentiable transform introducing normalized activations into the
network;
Batch normalized networks
◦ Unbiased variance estimate;
◦ Moving average;
Batch normalized ConvNets
◦ Effective mini-batch size;
◦ Per feature, not per activation.

71. Batch Normalization at Google
Reduce the dependence of gradients on the scale of the parameters or of the initial values;
◦ Prevent small changes from amplifying into larger and suboptimal changes in activation in gradients;
◦ Stabilize the parameter growth and make gradient propagation better behaved in BN training;
In some cases, eliminate the need of dropout as a regularizer;
◦ In ImageNet Classification, remove local response normalization and reduce photometric distortions;
◦ Reach 4.9% in top-five validation error and 4.8% test error (human raters only 5.1%).
Accelerating BN network:
◦ Enable larger learning rate and less care about initialization, which accelerates the training;
◦ Reduce L2 weight regularization;
◦ Accelerate the learning rate decay.

72. Batch Normalization at Google
Inception architecture

73. Neural Turing Machines
A Neural Turing Machine (NTM) architecture contains two basic components: a neural
network controller and a memory bank;
◦ During each update cycle, the controller network receives inputs from an external
environment and emits outputs in response;
◦ It also reads to and writes from a memory matrix via a set of parallel read and write heads.
These weightings arise by combining two addressing mechanisms with complementary
facilities;
◦ “content-based addressing”: focuses attention on locations based on the similarity between their
current values and values emitted by the controller;
◦ “location-based addressing”: the content of a variable is arbitrary, but the variable still needs a
recognizable name or addresses, by location, not by content;
Controller network: feed forward or recurrent.

74. Neural Turing Machines
Neural Turing Machine Architecture.
Flow Diagram of the Addressing Mechanism.

75. Highway Networks: Information Highway
Ease gradient-based training of very deep networks;
Allow unimpeded info. flow across several layers on information highways;
Use gating units to learn regulating the flow of info. through a network;
A highway network consists of multiple blocks such that the ith block computes a block
state Hi(x) and transform gate output Ti(x);
Highway networks with hundreds of layers can be trained directly using SGD and with a
variety of activation functions.
the transform gate the carry gate
C = 1 - T

76. Deep Residual Learning for Image Recognition
Reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning
unreferenced functions;
◦ The desired underlying mapping as H(x), then let the stacked nonlinear layers fit another mapping of F(x) = H(x) - x;
◦ The formulation of F(x)+x can be realized by feed forward NN with “shortcut connections” (such as “Highway
Network” and “Inception”);
These residual networks are easier to optimize, and can gain accuracy from considerably increased depth;
An ensemble of 152 layers residual nets achieves 3.57% error on the ImageNet test set;
◦ 224x224 crop, per-pixel mean subtracted, color augmentation, batch normalization;
◦ SGD with a mini-batch size of 256, learning rate from 0.1 then by 10;
◦ Weight decay of 0.0001 and a momentum of 0.9, no drop-out;

77. Rethink Inception Architecture for Computer Vision
Scale up networks in ways that aim at utilizing the added computation efficiently by factorized convolutions
and aggressive regularization;
Design principles in Inception:
◦ Avoid representational bottlenecks, especially early in the network;
◦ Higher dimensional representations are easier to process locally within a network;
◦ Spatial aggregation over lower dim embeddings w/o loss in representational power;
◦ Balance the width and depth of the network.
Factorizing convolutions with large filter size: asymmetric convolutions;
Auxiliary classifiers: act as regularizer, esp. batch normalized or dropout;
Grid size reduction: two parallel stride 2 blocks (pooling and activation) ;
Model regularization via label smoothing: marginalized effect of dropout;
Trained with Tensorflow: SGD with 50 replicas, batch size 32 for 100 epochs, learning rate of 0.045,
exponential rate of 0.94, a wei decay of 0.9.

78. Rethink Inception Architecture for Computer Vision
Inception modules after the factorization of the nxn
convolutions. In the proposed architecture, it choses
n = 7 for the 17x17 grid.
Inception modules with expanded
the filter bank outputs.
Inception modules where
each 5x5 convolution is
replaced by two 3x3
convolution.

79. Rethink Inception Architecture for Computer Vision
Auxiliary classifier on top
of the last 17x17 layer Inception module that reduces the grid-size while
expands the filter banks. It is both cheap and avoids
the representational bottleneck.
The outline of the proposed
network architecture

80. Belief Nets
Belief net is directed acyclic graph composed of stochastic var.
Can observe some of the variables and solve two problems:
◦ inference: Infer the states of the unobserved variables.
◦ learning: Adjust the interactions between variables to more likely generate the observed data.
stochastic
hidden
cause
visible
effect
Use nets composed of layers
of stochastic variables with
weighted connections.

81. Boltzmann Machines
Energy-based model associate a energy to each configuration of stochastic variables of interests (for
example, MRF, Nearest Neighbor);
◦ Learning means adjustment of the low energy function’s shape properties;
Boltzmann machine is a stochastic recurrent model with hidden variables;
◦ Monte Carlo Markov Chain, i.e. MCMC sampling (appendix);
Restricted Boltzmann machine is a special case:
◦ Only one layer of hidden units;
◦ factorization of each layer’s neurons/units (no connections in the same layer);
Contrastive divergence: approximation of gradient (appendix).
probability
Energy Function
Learning rule

82. Deep Belief Networks
A hybrid model: can be trained as generative or
discriminative model;
Deep architecture: multiple layers (learn features
layer by layer);
◦ Multi layer learning is difficult in sigmoid belief
networks.
◦ Top two layers are undirected connections, RBM;
◦ Lower layers get top down directed connections
from layers above;
Unsupervised or self-taught pre-learning provides
a good initialization;
◦ Greedy layer-wise unsupervised training for
RBM
Supervised fine-tuning
◦ Generative: wake-sleep algorithm (Up-down)
◦ Discriminative: back propagation (bottom-up)

83. Deep Boltzmann Machine
Learning internal representations that become increasingly complex;
High-level representations built from a large supply of unlabeled inputs;
Pre-training consists of learning a stack of modified RBMs, which are composed to create a deep Boltzmann
machine (undirected graph);
Generative fine-tuning: different from DBN
◦ Positive and negative phase (appendix)
Discriminative fine-tuning: the same to DBN
◦ Back propagation.

84. Denoising Auto-Encoder
Multilayer NNs with target output=input;
Reconstruction=decoder(encoder(input));
◦ Perturbs the input x to a corrupted version;
◦ Randomly sets some of the coordinates of input to zeros.
◦ Recover x from encoded perturbed data.
Learns a vector field towards higher probability regions;
Pre-trained with DBN or regularizer with perturbed training data;
Minimizes variational lower bound on a generative model;
◦ corresponds to regularized score matching on an RBM;
PCA=linear manifold=linear Auto Encoder;
Auto-encoder learns the salient variation like a nonlinear PCA.

85. Stacked Denoising Auto-Encoder
Stack many (may be sparse) auto-encoders in succession and train them using greedy layer-wise
unsupervised learning
◦ Drop the decode layer each time
◦ Performs better than stacking RBMs;
Supervised training on the last layer using final features;
(option) Supervised training on the entire network to fine- tune all weights of the neural net;
Empirically not quite as accurate as DBNs.

86. Stereopsis via Deep Learning
• Learn a binocular cross correlation model: use two quadrature pairs to detect disparity;
 Various filters correspond to phases, positions and frequencies;
• Disparity as latent variable: a pattern of matching filter responses;
 A joint probabilistic model over patch pairs and disparity defined as a Boltzmann machine.
 Training amounts to finding the parameters for max the log probability for pairs
 RBM used for this case;
 During inference, each latent variable receives activity
from exactly two products of matched filter-responses.
pooling

87. Stereopsis via Deep Learning
Example training data: Row 1, row 2 and row 3
show rendered image planes for the left/right
camera, where in row 3 the right camera has
been rotated by 45 around the z axis. Images
are rendered by depth maps shown in row 4 and
a randomly selected texture map from the
Berkeley Segmentation Database.
Example pairs from NORB-cluttered dataset. Learned binocular filter pairs.

88. Unsupervised Learning of Depth (and Motion)
• Learning about the interrelations between images from multiple cameras, multiple
frames in a video, or the combination of both;
• Depth and motion in a feature learning architecture based on the energy model;
• An AutoEncoder single-layer model uses multiplicative interactions to detect synchrony,
and a pooling layer independently trained on the hidden responses to achieve content
invariance;
• Depth as a latent variable in learning:
• Reconstruction error:
• Contraction as regularization:
• Complete objective function:
Note: there is no need for rectification, since
the model can learn any transformation
between the frames not just horizontal shift

89. Unsupervised Learning of Depth (and Motion)
• Extension to stereo sequences: both depth and motion;
 Encoding depth:
 Encoding motion:
 Multiview disparity:
Representation of depth
Representation of motion
Representation of disparity
Products of frame responses
Products of frame responses
Products of frame responses

90. Unsupervised Learning of Depth (and Motion)
Filters learned on stereo patch pairs from KITTI dataset.
Example of a filter pair learned on sequences by the
SAE-D model from the Hollywood3D dataset.

91. Stereo Matching by CNN
• Train a convolutional neural network on pairs of small image patches;
• The network output is used to initialize the matching cost btw a pair of patches;
• Eight layers, L1 through L8 with input as 9x9 gray patch and matching cost as output;
• 1st layer as convolutional only and other layers are fully connected.
• Rectified linear units follow each layer, except L8, but NO pooling!
• Trained with SGD (batch size as 128), by194 image pairs, 45 million extracted examples.
• Matching costs are combined between neighboring pixels with similar image
intensities using cross-based cost aggregation;
• Smoothness constraints are enforced by semi-global matching (SGM) and a left-right
consistency check is used to detect and eliminate errors in occluded regions;
• sub-pixel enhancement and median filter + bilateral filter -> final disparity map;
• Achieve the error rate of 2.61% on the KITTI stereo database ( < 2.83% before).

92. Stereo Matching by CNN
Support region

93. A Deep Embedding Model for Stereo Matching Costs
This deep embedding model leverages appearance data to learn visual similarity relationships btw corresp.
image patches, and maps intensity values into an embedding feature space to measure pixel dissimilarities;
Features are extracted in a pair of patches at different scales, followed
by an inner product to obtain the matching scores, then the scores
from different scales are then merged for an ensemble.
The deployed network architecture of our testing model for deep embedding.
Features are extracted in two images only once respectively. The sliding-
window style inner product can be grouped for matrix operation.

94. Improved Stereo Matching with Constant Highway
Networks and Reflective Confidence Learning
A 3-step pipeline for the stereo matching problem and a highway network architecture for computing the
matching cost at each possible disparity, based on multilevel weighted residual shortcuts, trained with a
hybrid loss that supports multilevel comparison of image patches.
A post-processing step employs a second deep convolutional neural network for pooling global
information from multiple disparities.
It outputs both the image disparity map, which replaces the conventional “winner takes all” strategy, and a
confidence in the prediction.
The confidence score is achieved by training the network with a reflective loss.
The learned confidence is employed to better detect outliers in the refinement.

95. Improved Stereo Matching with Constant Highway
Networks and Reflective Confidence Learning
The λ-ResMatch architecture of the matching cost network

96. Improved Stereo Matching with Constant Highway
Networks and Reflective Confidence Learning
The Global disparity network model for representing disparity patches

97. Efficient Deep Learning for Stereo Matching
 A matching network which is able to
produce very accurate results in less than a
second of GPU computation.
A product layer which simply computes the
inner product btw the two representations of
a Siamese architecture.
 Treat the disparity estimation problem as
multi-class classification, where the classes are
all possible disparities.
A Siamese network extracts marginal distributions
over all possible disparities for each pixel.
four-layer Siamese network
architecture
The code and data can is online at:
http://www.cs.toronto.edu/deepLowLevelVision.

98. End-to-End Learning of Geometry and
Context for Deep Stereo Regression
A deep learning architecture for regressing disparity from a rectified pair
of stereo images.
Leverage knowledge of the problem’s geometry to form a cost volume
using deep feature representations.
Learn to incorporate contextual information using 3-D convolutions over
this volume.
Disparity values regressed from the cost volume using a differentiable soft
argmin operation, which allows to train end-to-end to sub-pixel accuracy
without any additional post-processing or regularization.

99. End-to-End Learning of Geometry and
Context for Deep Stereo Regression
End-to-end deep stereo regression architecture, GC-Net (Geometry and Context Network)

100. End-to-End Training of Hybrid
CNN-CRF Models for Stereo
convolutional neural networks (CNNs) + optimization-based approaches for stereo estimation;
 The optimization, posed as a conditional random field (CRF), takes local matching costs and consistency-
enforcing (smoothness) costs as inputs, both estimated by CNN blocks.
 To perform inference in CRF, based on linear programming relaxation with a fixed number of iterations.
Training end-to-end: in the discriminative formulation (structured SVM), the training is practically feasible.
 The optimization part efficiently replaces post-processing steps by a trainable, well-understood model.
A CNN, called Unary-CNN, computes features of the two
images for each pixel. The features are compared using a
Correlation layer. The resulting matching cost volume becomes
the unary cost of the CRF. The pairwise costs of the CRF are
parametrized by edge weights, which can either follow a usual
contrast sensitive model or estimated by the Pairwise-CNN.

101. End-to-End Training of Hybrid
CNN-CRF Models for Stereo
The cross-correlation of features φ0 and φ1 respectively
The CRF model optimizes the cost of disparity labelings
matching costs
pairwise terms

102. Appendix A:
Depth from Single Image by Learning

103. Learning-based Depth from Image
Initial over-segmentation (super pixels);
Markov Random Field (MRF) to infer patch’s orientation and location from image features (texture, color and
gradient);
◦ Connected, co-planar or colinear as prior;
◦ Occlusion boundaries /folds indication;
◦ Multi-conditional learning; solved by linear program;
MRF overlaid on “super pixels”
Occlusion/fold
Coplanarity and Colinearity

105. Single Image Depth Estimation From Predicted Semantic Labels
Semantic segmentation to guide the 3D reconstruction;
Works like holistic scene understanding:
◦ 1. Multi-class image labeling MRF for scene segmentation;
◦ 2. Depth estimation for each semantic class by learning (logistic regression);
3. Scene depth estimation by MRF (pixel or super-pixel) with potential (learned boosted decision tree classifiers ) and prior of
geometry (horizon prediction, vertical objects), pixel’s smoothness, super-pixel ‘s soft connectivity, co-planarity and
orientation.
semantically derived geometric constraints
Smoothed per-pixel log-depth prior for each semantic class with horizon rotated to center of image

106. Image semantic overlay ground truth depth measurements

107. Learning Depth from Examples
Two similar images are likely to have similar 3D structure (depth).
Nearest-neighbor (kNN) search: finding k image+depth pairs that are most similar to the query (histograms of
oriented gradients as feature);
Depth fusion: median filtering of the k depth fields;
Joint-bilateral depth filtering: smoothing of the median-fused depth.

108. K-NN
query
Depth Fusion and Smoothing
Depth output
Note: depth (disparity) warping via SIFT-flow in aligning with the query is omitted.

109. Depth Transfer for Monocular Video
K-NN Search for candidates of query frames;
Depth changes are gradual frame-to-frame;
Moving objects are usually on the ground;
Warped with SIFT flow and regularized with smoothness and prior
Computational cost is worth?

110. Depth Inference with MRF

112. To form a basis (dictionary) over the RGB and depth spaces, and represent depth maps by
a sparse linear combination of weights.
A prediction function is estimated between weight vectors in RGB to depth space to
recover depth maps from query images.
A final super-pixel post processor aligns depth maps with occlusion boundaries, creating
physically plausible results.
Scalable Exemplar Based Depth Transfer

113. images with similar global depth profile clustered together in 2D utilizing RGB pairwise
features (left) and sparse positive descriptors on depth (right) effective in grouping
images with similar depths profiles together.
estimate a transformation T, that maps points from one space to another.
Scalable Exemplar Based Depth Transfer

114. Learning to be a Depth Camera (Active Near-IR)
• Use hybrid classification-regression forests to learn how to map from near infrared
intensity images to absolute, metric depth in real-time;
• Simplify the problem by dividing it into sub-problems in the first layer, and then applies models
trained for these sub-problems in the second layer to solve the main problem efficiently;
• Restrict the depths of the object to a certain range for significant simplification;
• The first layer learns to infer a coarsely quantized depth range for each pixel, and optionally
pools these predictions across all pixels to obtain a more reliable distribution over these depth
ranges;
• The second layer then applies one or more expert repressor trained specifically on the inferred
depth ranges.
• Note: the forests do not need to explicitly model scene illumination, surface geometry and
reflectance, or complex inter-reflections, required by traditional SFS methods.

115. Learning to be a Depth Camera (Active Near-IR)
• Comparable to high-quality consumer depth cameras with a reduced cost, power
consumption, and form-factor.

116. Learning to be a Depth Camera (Active Near-IR)
• Applied for specific hand and face objects.

117. Appendix B:
Machine Learning and Optimization

118. Graphical Models
• Graphical Models: Powerful framework for representing dependency
structure between random variables.
• The joint probability distribution over a set of random variables.
• The graph contains a set of nodes (vertices) that represent random variables, and a set
of links (edges) that represent dependencies between those random variables.
• The joint distribution over all random variables decomposes into a product of
factors, where each factor depends on a subset of the variables.
• Two type of graphical models:
• Directed (Bayesian networks)
• Undirected (Markov random fields, Boltzmann machines)
• Hybrid graphical models that combine directed and undirected models, such as Deep
Belief Networks, Hierarchical-Deep Models.

119. Generative Model: MRF
Random Field: F={F1,F2,…FM} a family of random variables on set S in which each Fi takes
value fi in a label set L.
Markov Random Field: F is said to be a MRF on S w.r.t. a neighborhood N if and only if it
satisfies Markov property.
◦ Generative model for joint probability p(x)
◦ allows no direct probabilistic interpretation
◦ define potential functions Ψ on maximal cliques A
◦ map joint assignment to non-negative real number
◦ requires normalization
MRF is undirected graphical models

120. A flow network G(V, E) defined as a fully connected directed graph
where each edge (u,v) in E has a positive capacity c(u,v) >= 0;
The max-flow problem is to find the flow of maximum value on a
flow network G;
A s-t cut or simply cut of a flow network G is a partition of V into S
and T = V-S, such that s in S and t in T;
A minimum cut of a flow network is a cut whose capacity is the
least over all the s-t cuts of the network;
Methods of max flow or mini-cut:
◦ Ford Fulkerson method;
◦ "Push-Relabel" method.

121. Mostly labeling is solved as an energy minimization problem;
Two common energy models:
◦ Potts Interaction Energy Model;
◦ Linear Interaction Energy Model.
Graph G contain two kinds of vertices: p-vertices and i-vertices;
◦ all the edges in the neighborhood N, called n-links;
◦ edges between the p-vertices and the i-vertices called t-links.
In the multiple labeling case, the multi-way cut should leave each p-vertex connected to one i-vertex;
The minimum cost multi-way cut will minimize the energy function where the severed n-links would
correspond to the boundaries of the labeled vertices;
The approximation algorithms to find this multi-way cut:
◦ "alpha-expansion" algorithm;
◦ "alpha-beta swap" algorithm.

122.  A simplified Bayes Net: it propagates info. throughout a graphical model via a series
of messages between neighboring nodes iteratively; likely to converge to a consensus that
determines the marginal prob. of all the variables;
 messages estimate the cost (or energy) of a configuration of a clique given all other cliques;
then the messages are combined to compute a belief (marginal or maximum probability);
Two types of BP methods:
◦ max-product;
◦ sum-product.
BP provides exact solution when there are no loops in graph!
Equivalent to dynamic programming/Viterbi in these cases;
Loopy Belief Propagation: still provides approximate (but often good) solution;

123. Generalized BP for pairwise MRFs
◦ Hidden variables xi and xj are connected through a compatibility function;
◦ Hidden variables xi are connected to observable variables yi by the local “evidence” function;
The joint probability of {x} is given by
To improve inference by taking into account higher-order interactions among the
variables;
◦ An intuitive way is to define messages that propagate between groups of nodes rather than just single nodes;
◦ This is the intuition in Generalized Belief Propagation (GBP).

124. Stochastic Gradient Descent (SGD)
• The general class of estimators that arise as minimizers of sums are called M-
estimators;
• Where are stationary points of the likelihood function (or zeroes of its derivative, the score
function)?
• Online gradient descent samples a subset of summand functions at every step;
• The true gradient is approximated by a gradient at a single example;
• Shuffling of training set at each pass.
• There is a compromise between two forms, often called "mini-batches", where the
true gradient is approximated by a sum over a small number of training examples.
• STD converges almost surely to a global minimum when the objective function
is convex or pseudo-convex, and otherwise converges almost surely to a local
minimum.

125. Back Propagation
E (f(x0,w),y0) = -log (f(x0,w)- y0).

126. Variable Learning Rate
Too large learning rate
◦ cause oscillation in searching for the minimal point
Too slow learning rate
◦ too slow convergence to the minimal point
Adaptive learning rate
◦ At the beginning, the learning rate can be large when the current point is far from the
optimal point;
◦ Gradually, the learning rate will decay as time goes by.
Should not be too large or too small:
◦ annealing rate 𝛼(𝑡)=𝛼(0)/(1+𝑡/𝑇)
◦ 𝛼(𝑡) will eventually go to zero, but at the beginning it is almost a constant.

127. Variable Momentum

128. AdaGrad/AdaDelta

129. Dropout and Maxout for Overfitting
Dropout: set the output of each hidden neuron to zero w.p. 0.5.
◦ Motivation: Combining many different models that share parameters succeeds in reducing test
errors by approximately averaging together the predictions, which resembles the bagging.
◦ The units which are “dropped out” in this way do not contribute to the forward pass and do not
participate in back propagation.
◦ So every time an input is presented, the NN samples a different architecture, but all these
architectures share weights.
◦ This technique reduces complex co-adaptations of units, since a neuron cannot rely on the presence
of particular other units.
◦ It is, therefore, forced to learn more robust features that are useful in conjunction with many
different random subsets of the other units.
◦ Without dropout, the network exhibits substantial overfitting.
◦ Dropout roughly doubles the number of iterations required to converge.
Maxout takes the maximum across multiple feature maps;

130. Weight Decay for Overfitting
Weight decay or L2 regularization adds a penalty term to the error function, a term called the
regularization term: the negative log prior in Bayesian justification,
◦ Weight decay works as rescaling weights in the learning rule, but bias learning still the same;
◦ Prefer to learn small weights, and large weights allowed if improving the original cost function;
◦ A way of compromising btw finding small weights and minimizing the original cost function;
In a linear model, weight decay is equivalent to ridge (Tikhonov) regression;
L1 regularization: the weights not really useful shrink by a constant amount toward zero;
◦ Act like a form of feature selection;
◦ Make the input filters cleaner and easier to interpret;
L2 regularization penalizes large values strongly while L1 regularization ;
Markov Chain Monte Carlo (MCMC): simulating a Markov chain whose equilibrium distr. is the
posterior distribution for weights & hyper-parameters;
Hybrid Monte Carlo: gradient and sampling.

131. Early Stopping for Overfitting
Steps in early stopping:
◦ Divide the available data into training and validation sets.
◦ Use a large number of hidden units.
◦ Use very small random initial values.
◦ Use a slow learning rate.
◦ Compute the validation error rate periodically during training.
◦ Stop training when the validation error rate "starts to go up".
Early stopping has several advantages:
◦ It is fast.
◦ It can be applied successfully to networks in which the number of weights far exceeds the sample size.
◦ It requires only one major decision by the user: what proportion of validation cases to use.
Practical issues in early stopping:
◦ How many cases do you assign to the training and validation sets?
◦ Do you split the data into training and validation sets randomly or by some systematic algorithm?
◦ How do you tell when the validation error rate "starts to go up"?

132. MCMC Sampling for Optimization
Markov Chain: a stochastic process in which future states are independent of past states but the
present state.
◦ Markov chain will typically converge to a stable distribution.
Monte Carlo Markov Chain: sampling using ‘local’ information
◦ Devise a Markov chain whose stationary distribution is the target.
◦ Ergodic MC must be aperiodic, irreducible, and positive recurrent.
◦ Monte Carlo Integration to get quantities of interest.
Metropolis-Hastings method: sampling from a target distribution
◦ Create a Markov chain whose transition matrix does not depend on the normalization term.
◦ Make sure the chain has a stationary distribution and it is equal to the target distribution (accept ratio).
◦ After sufficient number of iterations, the chain will converge the stationary distribution.
Gibbs sampling is a special case of M-H Sampling.
◦ The Hammersley-Clifford Theorem: get the joint distribution from the complete conditional distribution.
Hybrid Monte Carlo: gradient sub step for each Markov chain.

133. Mean Field for Optimization
Variational approximation modifies the optimization problem to be tractable, at the price of
approximate solution;
Mean Field replaces M with a (simple) subset M(F), on which A* (μ) is a closed form (Note: F is
disconnected graph);
◦ Density becomes factorized product distribution in this sub-family.
◦ Objective: K-L divergence.
Mean field is a structured variation approximation approach:
◦ Coordinate ascent (deterministic);
Compared with stochastic approximation (sampling):
◦ Faster, but maybe not exact.

134. Contrastive Divergence for RBMs
Contrastive divergence (CD) is proposed for training PoE first, also being a quicker way to learn
RBMs;
◦ Contrastive divergence as the new objective;
◦ Taking gradients and ignoring a term which is usually very small.
Steps:
◦ Start with a training vector on the visible units.
◦ Then alternate between updating all the hidden units in parallel and updating all the visible units in parallel.
Can be applied using any MCMC algorithm to simulate the model (not limited to just Gibbs
sampling);
CD learning is biased: not work as gradient descent
Improved: Persistent CD explores more modes in the distribution
◦ Rather than from data samples, begin sampling from the mode samples, obtained from the last gradient
update.
◦ Still suffer from divergence of likelihood due to missing the modes.
Score matching: the score function does not depend on its normal. factor. So, match it b.t.w. the
model with the empirical density.

135. “Wake-Sleep” Algorithm for DBN
Pre-trained DBN is a generative model;
Do a stochastic bottom-up pass (wake phase)
◦ Get samples from factorial distribution (visible first, then generate hidden);
◦ Adjust the top-down weights to be good at reconstructing the feature activities in the layer below.
Do a few iterations of sampling in the top level RBM
◦ Adjust the weights in the top-level RBM.
Do a stochastic top-down pass (sleep phase)
◦ Get visible and hidden samples generated by generative model using data coming from nowhere!
◦ Adjust the bottom-up weights to be good at reconstructing the feature activities in the layer above.
◦ Any guarantee for improvement? No!
The “Wake-Sleep” algorithm is trying to describe the representation economical (Shannon’s coding
theory).

136. Greedy Layer-Wise Training
Deep networks tend to have more local minima problems than shallow networks during
supervised training
Train first layer using unlabeled data
◦ Supervised or semi-supervised: use more unlabeled data.
Freeze the first layer parameters and train the second layer
Repeat this for as many layers as desire
◦ Build more robust features
Use the outputs of the final layer to train the last supervised layer (leave early weights frozen)
Fine tune the full network with a supervised approach;
Avoid problems to train a deep net in a supervised fashion.
◦ Each layer gets full learning
◦ Help with ineffective early layer learning
◦ Help with deep network local minima

137. Why Greedy Layer-Wise Training Works?
Take advantage of the unlabeled data;
Regularization Hypothesis
◦ Pre-training is “constraining” parameters in a region relevant to unsupervised
dataset;
◦ Better generalization (representations that better describe unlabeled data are
more discriminative for labeled data) ;
Optimization Hypothesis
◦ Unsupervised training initializes lower level parameters near localities of better
minima than random initialization can.
Only need fine tuning in the supervised learning stage.

138. Two-Stage Pre-training in DBMs
Pre-training in one stage
◦ Positive phase: clamp observed, sample hidden, using variational approximation (mean-field)
◦ Negative phase: sample both observed and hidden, using persistent sampling (stochastic approximation:
MCMC)
Pre-training in two stages
◦ Approximating a posterior distribution over the states of hidden units (a simpler directed deep model as DBNs
or stacked DAE);
◦ Train an RBM by updating parameters to maximize the lower-bound of log-likelihood and correspond.
posterior of hidden units.
◦ Options (CAST, contrastive divergence, stochastic approximation…).