Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Implementing Deep
Learning using cuDNN
이예하

VUNO Inc.
1
Contents
1. Deep Learning Review

2. Implementation on GPU using cuDNN

3. Optimization Issues

4. Introduction to VUNO-Ne...
1.

Deep Learning Review
3
Brief History of Neural Network
1940 1950 1960 1970 1980 1990 2000
Golden Age Dark Age (“AI Winter”)
Electronic Brain
1943...
Machine/Deep Learning is eating the world!
5
• Restricted Boltzmann machine
• Auto-encoder
• Deep belief network

• Deep Boltzmann machines

• Generative stochastic ne...
• Receptive Field(Huber & Wiesel, 1959)



• Neocognitron (Fukushima, 1980)
Convolutional Neural Networks
7
• LeNet-5 (Yann LeCun, 1998)
Convolutional Neural Networks
8
• Alex Net (Alex Krizhevsky et. al., 2012)





• Clarifai (Matt Zeiler and Rob Fergus, 2013)
Convolutional Neural Network...
• Detection/Classification and Localization

• 1.2M for Training/50K for Validation

• 1000 Classes
ImageNet Challenge
10
...
Convolutional Neural Networks
Network
• Softmax Layer (Output)

• Fully Connected Layer

• Pooling Layer

• Convolution La...
Neural Network
12
Forward Pass
Softmax Layer
FC Layer
Conv Layer
Backward Pass
Softmax Layer
FC Layer
Conv Layer
Convolution Layer - Forward
x1 x4 x7
x2 x5 x8
x3 x6 x9
w1 w3
w2 w4
y1 y3
y2 y4
Filter Input
13
w1
w2
w4
w3
Output
Convolution Layer - Forward
How to evaluate the convolution layer efficiently?
1. Lower the convolutions into a matrix mul...
Fully Connected Layer - Forward
15
x2x1
y1 y2
x3
w11 w12 w13
w w w
• Matrix calculation is very fast on GPU

• cuBLAS libr...
Softmax Layer - Forward
16
x2x1
y1 y2
Issues
1. How to efficiently evaluate denominator? 

2. Numerical instability due to...
Learning on Neural Network
Update network (i.e. weights) to minimize loss function
• Popular loss function for classificat...
Learning on Neural Network
• Due to the complexity of the loss, a closed-form solution is usually not possible

• Using it...
Softmax Layer - Backward
x2x1
y1 y2
Loss function:
• Errors back-propagated from softmax Layer
can be calculated
• Subtrac...
Fully Connected Layer - Backward
20
x2x1
y1 y2
x3
w11 w w
w21 w w
Forward pass:
Error:
Gradient:
• Matrix calculation is v...
Convolution Layer - Backward
x1 x4 x7
x2 x5 x8
x3 x6 x9
w1 w3
w2 w4
y1 y3
y2 y4
Filter Input
21
w1
w2
w4
w3
Output
w4 w2
w...
Convolution Layer - Backward
Filter Input
22
Output
x1 x4 x7
x2 x5 x8
x3 x6 x9
w1 w3
w2 w4
Gradient
• The error and gradie...
2.

Implementation on GPU
using cuDNN
23
Introduction to cuDNN
cuDNN is a GPU-accelerated library of primitives for deep
neural networks
• Convolution forward and ...
Introduction to cuDNN (version 2)
cuDNN's convolution routines aim for performance
competitive with the fastest GEMM
•Lowe...
Introduction to cuDNN
Benchmarks
26
https://github.com/soumith/convnet-benchmarks
https://developer.nvidia.com/cudnn
Goal
Learning VGG model using cuDNN
• Data Layer
• Convolution Layer
• Pooling Layer
• Fully Connected Layer
• Softmax Lay...
Preliminary
Common data structure for Layer
• Device memory & tensor description for input/output data & error
• Tensor De...
Data Layer
29
create & set Tensor Descriptor
cudnnCreateTensorDescriptor();
cudnnSetTensor4dDescriptor();
cudnnSetTensor4d...
Convolution Layer
1. Initialization
30
1.1 create & set Filter Descriptor
1.2 create & set Conv Descriptor
1.3 create & se...
Convolution Layer
1.1 Set Filter Descriptor
31
cudnnSetFilter4dDescriptor(

filterDesc,

CUDNN_FLOAT,

filterCnt,

input_c...
Convolution Layer
1.2 Set Convolution Descriptor
32
CUDNN_CROSS_CORRELATION
Convolution Layer
1.3 Set output Tensor Descriptor
33
• n, c, h and w indicate output dimension
• Tensor Description defin...
Convolution Layer
1.4 Get Convolution Algorithm
34
inputDesc
outputDesc
CUDNN_CONVOLUTION_FWD_PREFER_FASTEST
Convolution Layer
2. Forward Pass
2.1 Convolution
2.2 Activation
cudnnConvolutionForward(…);
cudnnActivationForward(…);
35
Convolution Layer
2.1 Convolution
36
d_input

inputDesc
d_output

outputDesc
Convolution Layer
2.2 Activation
37
sigmoid

tanh

ReLU
outputDesc

d_output
Convolution Layer
3. Backward Pass
3.1 Activation Backward
3.2 Calculate Gradient
cudnnActivationBackward(…);
cudnnConvolu...
• Errors back-propagated from l+1 layer (d_outputDelta) is multiplied by
• See 22 slide (Convolution Layer - Backward)
Con...
Convolution Layer
3.2 Calculate Gradient
40
inputDesc

d_input
outputDesc

d_outputDelta
filterDesc

d_filterGradient
Convolution Layer
3.2 Error Backpropagation
41
outputDesc

d_outputDelta
inputDesc

d_inputDelta
Pooling Layer / Softmax Layer
42
1. Initilization
2. Forward Pass
3. Backward Pass
cudnnCreatePoolingDescriptor(&poolingDe...
3.

Optimization Issues
43
Optimization
Learning Very Deep ConvNet
We know the Deep ConvNet can be trained without pre-training

• weight sharing

• ...
Optimization
VGG (K. Simonyan and A. Zisserman, 2015) GoogleNet (C. Szegedy et. al., 2014)
45
Optimization
Initialization of Weights for Rectifier (Kaiming He et. al., 2015)
•The variance of the response in each laye...
Optimization
Case study
47
The filter number
64 0.059
128 0.042
256 0.029
512 0.021
3x3 filter
• When using 0.01, the std ...
Speed
Data loading & model learning
•Reducing data loading and augmentation time

•Data provider thread (dp_thread)

•Mode...
Speed
Multi-GPU
• Data parallelization v.s. Model parallelization
• Distribute the model, use the same data : Model Parall...
Parallelism
50
Data Parallelization Model Parallelization
The good : Easy to implement

The bad : Cost of sync increases w...
Parallelism
51
Mixing Data Parallelization and Model Parallelization
(Krizhevsky, 2014)
Speed
Data Parallelization & Gradient Average
Forward Pass
Backward Pass
Gradient
Update
data (256)
Forward Pass
Backward ...
4.

Introduction to VUNO-
Net
53
The Team
54
VUNO-Net
Theano
Caffe
Cuda-

ConvNet
Pylearn
RNNLIB
DL4J
Torch VUNO-Net
55
VUNO-Net
56
Structure Output Learning
Convolution
LSTM
MD-LSTM(2D)
Pooling
Spatial Pyramid Pooling
Fully Connection
Concat...
Performance
The state-of-the-art performance at image & speech
84.3%82.2%
VUNOWR1
Speech Recognition

(TIMIT*)
82.1%79.3%
...
Application
We’ve achieved record breaking performance on medical
image analysis.
58
Application
Whole Lung Quantification
• Sliding window: Pixel level classification

• But, context information is more imp...
Application
Whole Lung Quantification
60
Original Image (CT) VUNO Golden Standard
Example #1
Application
Whole Lung Quantification
61
Original Image (CT) VUNO Golden Standard
Example #2
Visualization
SVM Features (22 features)
62
Histogram, Gradient, Run-length, Co-occurrence matrix, Cluster analysis, 

Top...
Visualization
Activation of top hidden layer
63
200 hidden nodes
Visualization
64
We Are Hiring!!
65
Algorithm Engineer CUDA Programmer Application Developer
Staff MemberBusiness Developer
Q&A
66
Thank You
67
Upcoming SlideShare
Loading in …5
×

[251] implementing deep learning using cu dnn

15,418 views

Published on

DEVIEW2015 DAY2. implementing deep learning using cu dnn

Published in: Technology
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • how to lose belly fat naturally without exercise ■■■ http://t.cn/AiugoW6M
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Enough is a enough! Is this going to be the day you finally do something about your health? It is a lot easier than you think to be able to shed off unwanted weight. See how you can get started today with 1 minute weight loss routines! ◆◆◆ https://tinyurl.com/1minweight4u
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Which food is best for weight loss? ▲▲▲ https://tinyurl.com/bkfitness4u
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

[251] implementing deep learning using cu dnn

  1. 1. Implementing Deep Learning using cuDNN 이예하 VUNO Inc. 1
  2. 2. Contents 1. Deep Learning Review 2. Implementation on GPU using cuDNN 3. Optimization Issues 4. Introduction to VUNO-Net 2
  3. 3. 1. Deep Learning Review 3
  4. 4. Brief History of Neural Network 1940 1950 1960 1970 1980 1990 2000 Golden Age Dark Age (“AI Winter”) Electronic Brain 1943 S. McCulloch - W. Pitts • Adjustable Weights • Weights are not Learned 1969 XOR Problem M. Minsky - S. Papert • XOR Problem Multi-layered Perceptron (Backpropagation) 1986 D. Rumelhart - G. Hinton - R. Wiliams • Solution to nonlinearly separable problems • Big computation, local optima and overfitting Backward Error Foward Activity V. Vapnik - C. Cortes 1995 SVM • Limitations of learning prior knowledge • Kernel function: Human Intervention Deep Neural Network (Pretraining) 2006 G. Hinton - S. Ruslan • Hierarchical feature Learning 2010 Perceptron 1957 F. Rosenblatt • Learnable Weights and Threshold ADALINE 1960 B. Widrow - M. Hoff 4
  5. 5. Machine/Deep Learning is eating the world! 5
  6. 6. • Restricted Boltzmann machine • Auto-encoder • Deep belief network • Deep Boltzmann machines • Generative stochastic networks • Convolutional neural networks • Recurrent neural networks Building Blocks 6
  7. 7. • Receptive Field(Huber & Wiesel, 1959)
 
 • Neocognitron (Fukushima, 1980) Convolutional Neural Networks 7
  8. 8. • LeNet-5 (Yann LeCun, 1998) Convolutional Neural Networks 8
  9. 9. • Alex Net (Alex Krizhevsky et. al., 2012)
 
 
 • Clarifai (Matt Zeiler and Rob Fergus, 2013) Convolutional Neural Networks 9
  10. 10. • Detection/Classification and Localization • 1.2M for Training/50K for Validation • 1000 Classes ImageNet Challenge 10 Error rate Remark Human 5.1% Karpathy google (2014/8) 6.67% Inception baidu (2015/1) 5.98% Data Augmentation ms (2015/2) 4.94% SPP, PReLU, 
 Weight Initialization google (2015/2) 4.82% Batch Normalization baidu (2015/5) 4.58% Cheating
  11. 11. Convolutional Neural Networks Network • Softmax Layer (Output) • Fully Connected Layer • Pooling Layer • Convolution Layer Layer • Input / Output • Weights • Neuron activationVGG (K. Simonyan and A. Zisserman, 2015) 11
  12. 12. Neural Network 12 Forward Pass Softmax Layer FC Layer Conv Layer Backward Pass Softmax Layer FC Layer Conv Layer
  13. 13. Convolution Layer - Forward x1 x4 x7 x2 x5 x8 x3 x6 x9 w1 w3 w2 w4 y1 y3 y2 y4 Filter Input 13 w1 w2 w4 w3 Output
  14. 14. Convolution Layer - Forward How to evaluate the convolution layer efficiently? 1. Lower the convolutions into a matrix multiplication (cuDNN) • There are several ways to implement convolutions efficiently 2. Fast Fourier Transform to compute the convolution (cuDNN_v3) 3. Computing the convolutions directly (cuda-convnet) 14
  15. 15. Fully Connected Layer - Forward 15 x2x1 y1 y2 x3 w11 w12 w13 w w w • Matrix calculation is very fast on GPU • cuBLAS library
  16. 16. Softmax Layer - Forward 16 x2x1 y1 y2 Issues 1. How to efficiently evaluate denominator? 2. Numerical instability due to the too large or too small xk Fortunately, softmax Layer is supported by cuDNN
  17. 17. Learning on Neural Network Update network (i.e. weights) to minimize loss function • Popular loss function for classification • Necessary criterion Sum-of-squared error Cross-entropy error 17
  18. 18. Learning on Neural Network • Due to the complexity of the loss, a closed-form solution is usually not possible • Using iterative approach • How to evaluate the gradient • Error backpropagation Gradient descent Newton’s method 18
  19. 19. Softmax Layer - Backward x2x1 y1 y2 Loss function: • Errors back-propagated from softmax Layer can be calculated • Subtract 1 from the output value with target index • This can be done efficiently using GPU threads 19
  20. 20. Fully Connected Layer - Backward 20 x2x1 y1 y2 x3 w11 w w w21 w w Forward pass: Error: Gradient: • Matrix calculation is very fast on GPU • Element-wise multiplication can be done efficiently using GPU thread Error: Gradient:
  21. 21. Convolution Layer - Backward x1 x4 x7 x2 x5 x8 x3 x6 x9 w1 w3 w2 w4 y1 y3 y2 y4 Filter Input 21 w1 w2 w4 w3 Output w4 w2 w3 w1 Forward Error
  22. 22. Convolution Layer - Backward Filter Input 22 Output x1 x4 x7 x2 x5 x8 x3 x6 x9 w1 w3 w2 w4 Gradient • The error and gradient can be computed with convolution scheme • cuDNN supports this operation
  23. 23. 2. Implementation on GPU using cuDNN 23
  24. 24. Introduction to cuDNN cuDNN is a GPU-accelerated library of primitives for deep neural networks • Convolution forward and backward • Pooling forward and backward • Softmax forward and backward • Neuron activations forward and backward: • Rectified linear (ReLU) • Sigmoid • Hyperbolic tangent (TANH) • Tensor transformation functions 24
  25. 25. Introduction to cuDNN (version 2) cuDNN's convolution routines aim for performance competitive with the fastest GEMM •Lowering the convolutions into a matrix multiplication 25(Sharan Chetlur et. al., 2015)
  26. 26. Introduction to cuDNN Benchmarks 26 https://github.com/soumith/convnet-benchmarks https://developer.nvidia.com/cudnn
  27. 27. Goal Learning VGG model using cuDNN • Data Layer • Convolution Layer • Pooling Layer • Fully Connected Layer • Softmax Layer 27
  28. 28. Preliminary Common data structure for Layer • Device memory & tensor description for input/output data & error • Tensor Description defines dimensions of data 28 cuDNN initialize & release cudaSetDevice( deviceID ); cudnnCreate( &cudnn_handle ); cudnnDestroy( cudnn_handle ); cudaDeviceReset(); float *d_input, *d_output, *d_inputDelta, *d_outputDelta cudnnTensorDescriptor_t inputDesc; cudnnTensorDescriptor_t outputDesc;
  29. 29. Data Layer 29 create & set Tensor Descriptor cudnnCreateTensorDescriptor(); cudnnSetTensor4dDescriptor(); cudnnSetTensor4dDescriptor( outputDesc, CUDNN_TENSOR_NCHW, CUDNN_FLOAT, sampleCnt, channels, height, width ); 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 sample #1 sample #2 channel #1 channel #2 Example: 2 images (3x3x2)
  30. 30. Convolution Layer 1. Initialization 30 1.1 create & set Filter Descriptor 1.2 create & set Conv Descriptor 1.3 create & set output Tensor Descriptor 1.4 Get Convolution Algorithm cudnnCreateFilterDescriptor(&filterDesc); cudnnSetFilter4dDescriptor(…); cudnnCreateConvolutionDescriptor(&convDesc); cudnnSetConvolution2dDescriptor(…); cudnnGetConvolution2dForwardOutputDim(…); cudnnCreateTensorDescriptor(&dstTensorDesc); cudnnSetTensor4dDescriptor(); cudnnGetConvolutionForwardAlgorithm(…); cudnnGetConvolutionForwardWorkspaceSize(…);
  31. 31. Convolution Layer 1.1 Set Filter Descriptor 31 cudnnSetFilter4dDescriptor( filterDesc, CUDNN_FLOAT, filterCnt, input_channelCnt, filter_height, filter_width ); 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Filter #1 Filter #2 channel #1 channel #2 Example: 2 Filters (2x2x2)
  32. 32. Convolution Layer 1.2 Set Convolution Descriptor 32 CUDNN_CROSS_CORRELATION
  33. 33. Convolution Layer 1.3 Set output Tensor Descriptor 33 • n, c, h and w indicate output dimension • Tensor Description defines dimensions of data
  34. 34. Convolution Layer 1.4 Get Convolution Algorithm 34 inputDesc outputDesc CUDNN_CONVOLUTION_FWD_PREFER_FASTEST
  35. 35. Convolution Layer 2. Forward Pass 2.1 Convolution 2.2 Activation cudnnConvolutionForward(…); cudnnActivationForward(…); 35
  36. 36. Convolution Layer 2.1 Convolution 36 d_input inputDesc d_output outputDesc
  37. 37. Convolution Layer 2.2 Activation 37 sigmoid tanh ReLU outputDesc d_output
  38. 38. Convolution Layer 3. Backward Pass 3.1 Activation Backward 3.2 Calculate Gradient cudnnActivationBackward(…); cudnnConvolutionBackwardFilter(…); 3.2 Error Backpropagation cudnnConvolutionBackwardData(…); 38
  39. 39. • Errors back-propagated from l+1 layer (d_outputDelta) is multiplied by • See 22 slide (Convolution Layer - Backward) Convolution Layer 3.1 Activation Backward 39 outputDesc d_output outputDesc d_outputDelta outputDesc d_output outputDesc d_outputDelta
  40. 40. Convolution Layer 3.2 Calculate Gradient 40 inputDesc d_input outputDesc d_outputDelta filterDesc d_filterGradient
  41. 41. Convolution Layer 3.2 Error Backpropagation 41 outputDesc d_outputDelta inputDesc d_inputDelta
  42. 42. Pooling Layer / Softmax Layer 42 1. Initilization 2. Forward Pass 3. Backward Pass cudnnCreatePoolingDescriptor(&poolingDesc); cudnnSetPooling2dDescriptor(…); cudnnPoolingForward(…); cudnnPoolingBackward(…); Pooling Layer Softmax Layer Forward Pass cudnnSoftmaxForward(…);
  43. 43. 3. Optimization Issues 43
  44. 44. Optimization Learning Very Deep ConvNet We know the Deep ConvNet can be trained without pre-training • weight sharing • sparsity • Recifier unit But, “With fixed standard deviations very deep models have difficulties to converges” (Kaiming He et. al., 2015) • e.g. random initialization from Gaussian dist. with 0.01 std • >8 convolution layers 44
  45. 45. Optimization VGG (K. Simonyan and A. Zisserman, 2015) GoogleNet (C. Szegedy et. al., 2014) 45
  46. 46. Optimization Initialization of Weights for Rectifier (Kaiming He et. al., 2015) •The variance of the response in each layer •Sufficient condition that the gradient is not exponentially large/small •Standard deviation for initialization (spatial filter size)2 x (filter Cnt) 46
  47. 47. Optimization Case study 47 The filter number 64 0.059 128 0.042 256 0.029 512 0.021 3x3 filter • When using 0.01, the std of the gradient propagated from conv10 to convey Error vanishing
  48. 48. Speed Data loading & model learning •Reducing data loading and augmentation time •Data provider thread (dp_thread) •Model learning thread (worker_thread) readData() { if(is_dp_thread_running) pthread_join(dp_thread) … if(is_data_remain) pthread_create(dp_thread) } readData(); for(…) { readData(); pthread_create(worker_thread) … pthread_join(worker_thread) } 48
  49. 49. Speed Multi-GPU • Data parallelization v.s. Model parallelization • Distribute the model, use the same data : Model Parallelism • Distribute the data, use the same model : Data Parallelism • Data parallelization & Gradient Average • One of the easiest way to use Multi-GPU • The result is same with using single GPU 49
  50. 50. Parallelism 50 Data Parallelization Model Parallelization The good : Easy to implement The bad : Cost of sync increases with the number of GPU The good : Larger network can be trained
 The bad : Sync is necessary in all layers
  51. 51. Parallelism 51 Mixing Data Parallelization and Model Parallelization (Krizhevsky, 2014)
  52. 52. Speed Data Parallelization & Gradient Average Forward Pass Backward Pass Gradient Update data (256) Forward Pass Backward Pass Gradient Update data1 (128) Forward Pass Backward Pass Gradient data2(128) Average 52
  53. 53. 4. Introduction to VUNO- Net 53
  54. 54. The Team 54
  55. 55. VUNO-Net Theano Caffe Cuda-
 ConvNet Pylearn RNNLIB DL4J Torch VUNO-Net 55
  56. 56. VUNO-Net 56 Structure Output Learning Convolution LSTM MD-LSTM(2D) Pooling Spatial Pyramid Pooling Fully Connection Concatenation Softmax Regression Connectionist Temporal Classification Multi-GPU Support Batch Normalization Parametric Rectifier Initialization for Rectifier Stochastic Gradient Descent Dropout Data Augmentation Open Source VUNO-net Only Features
  57. 57. Performance The state-of-the-art performance at image & speech 84.3%82.2% VUNOWR1 Speech Recognition (TIMIT*) 82.1%79.3% VUNOWR2 Image Classification (CIFAR-100) Kaggle Top 1
 (2014) * TIMIT is a one of most popular benchmark dataset for speech recognition task (Texas Instrument - MIT) WR1 (World Record) - “Speech Recognitino with Deep Recurrent Neural Networks”, Alex Graves, ICCASP(2013) WR2 (World Record) - “kaggle competition: https://www.kaggle.com/c/cifar-10” 57
  58. 58. Application We’ve achieved record breaking performance on medical image analysis. 58
  59. 59. Application Whole Lung Quantification • Sliding window: Pixel level classification • But, context information is more important • Ongoing works MD-LSTM Recurrent CNN normal Emphysemav.s. 59
  60. 60. Application Whole Lung Quantification 60 Original Image (CT) VUNO Golden Standard Example #1
  61. 61. Application Whole Lung Quantification 61 Original Image (CT) VUNO Golden Standard Example #2
  62. 62. Visualization SVM Features (22 features) 62 Histogram, Gradient, Run-length, Co-occurrence matrix, Cluster analysis, Top-hat transformation
  63. 63. Visualization Activation of top hidden layer 63 200 hidden nodes
  64. 64. Visualization 64
  65. 65. We Are Hiring!! 65 Algorithm Engineer CUDA Programmer Application Developer Staff MemberBusiness Developer
  66. 66. Q&A 66
  67. 67. Thank You 67

×