• Save
Simple Introduction to AutoEncoder
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Simple Introduction to AutoEncoder

on

  • 6,791 views

Simple Introduction to AutoEncoder

Simple Introduction to AutoEncoder

Statistics

Views

Total Views
6,791
Views on SlideShare
6,772
Embed Views
19

Actions

Likes
9
Downloads
0
Comments
0

4 Embeds 19

http://www.slashdocs.com 9
http://www.docshut.com 6
http://deepl4ml-lit.blogspot.com 2
https://www.blogger.com 2

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Simple Introduction to AutoEncoder Presentation Transcript

  • 1. Simple Introduction to AutoEncoder Lang JunDeep Learning Study Group, HLT, I2R 17 August, 2012
  • 2. Outline1. What is AutoEncoder? Input = decoder(encoder(input))2. How to train AutoEncoder? pre-training3. What can it be used for? reduce dimensionality 2/34
  • 3. 1. What is AutoEncoder?➢ Multilayer neural net simple review 3/34
  • 4. 1. What is AutoEncoder?➢ Multilayer neural net simple review 4/34
  • 5. 1. What is AutoEncoder?➢ Multilayer neural net simple review 5/34
  • 6. 1. What is AutoEncoder?➢ Multilayer neural net simple review 6/34
  • 7. 1. What is AutoEncoder?➢ Multilayer neural net simple review 7/34
  • 8. 1. What is AutoEncoder?➢ Multilayer neural net simple review 8/34
  • 9. 1. What is AutoEncoder?➢ Multilayer neural net simple review 9/34
  • 10. 1. What is AutoEncoder?➢ Multilayer neural net simple review 10/34
  • 11. 1. What is AutoEncoder?➢ Multilayer neural net simple review 11/34
  • 12. 1. What is AutoEncoder?➢ Multilayer neural net simple review 12/34
  • 13. 1. What is AutoEncoder?➢ Multilayer neural net simple review 13/34
  • 14. 1. What is AutoEncoder?➢ Multilayer neural net simple review 14/34
  • 15. 1. What is AutoEncoder?➢ Multilayer neural net simple review 15/34
  • 16. 1. What is AutoEncoder?➢ Multilayer neural net simple review 16/34
  • 17. 1. What is AutoEncoder?➢ Multilayer neural net simple review 17/34
  • 18. 1. What is AutoEncoder?➢ Multilayer neural net simple review 18/34
  • 19. 1. What is AutoEncoder?➢ Multilayer neural net simple review 19/34
  • 20. 1. What is AutoEncoder?➢ Multilayer neural net simple review 20/34
  • 21. 1. What is AutoEncoder?➢ Multilayer neural net simple review 21/34
  • 22. 1. What is AutoEncoder?➢ Multilayer neural net simple review 22/34
  • 23. 1. What is AutoEncoder?➢ Multilayer neural net simple review 23/34
  • 24. 1. What is AutoEncoder?➢ Multilayer neural net with target output = input➢ Reconstruction=decoder(encoder(input))➢ Minimizing reconstruction error➢ Probable inputs have small reconstruction error 24/34
  • 25. 2. How to train AutoEncoder? Hinton (2006) Science PaperRestricted Boltzmann Machine(RBM) 25/34
  • 26. 2. How to train AutoEncoder? Hinton (2006) Science Paperrestricted Boltzmann machine 26/34
  • 27. Effective deep learning becamepossible through unsupervised pre- training Purely supervised neural net With unsupervised pre‐training (with RBMs and Denoising Auto-Encoders) 27/34 0–9 handwritten digit recognition error rate (MNIST data)
  • 28. Why is unsupervised pre-training working so well?Regularization hypothesis: Representations goodfor P(x) are good for P(y|x)Optimization hypothesis: Unsupervised initializationsstart near better local minimum of supervised training error Minima otherwise notachievable by randominitializationErhan, Courville, Manzagol, Vincent, Bengio (JMLR, 2010) 28/34
  • 29. 3. What can it be used for? illustration for images 29/34
  • 30. 3. What can it be used for? document retrieval output2000 reconstructed counts vector • We train the neural network 500 neurons to reproduce its input vector as its output • This forces it to compress as 250 neurons much information as possible into the 10 numbers in the central bottleneck. 10 • These 10 numbers are then a good way to compare documents. 250 neurons – See Ruslan Salakhutdinov’s talk 500 neurons input 30/34 2000 word counts vector
  • 31. 3. What can it be used for? visualize documents output 2000 reconstructed counts vector• Instead of using codes to retrieve documents, we can 500 neurons use 2-D codes to visualize sets of documents. – This works much better 250 neurons than 2-D PCA 2 250 neurons 500 neurons input 31/34 2000 word counts vector
  • 32. First compress all documents to 2 numbers using a type of PCA Then use different colors for differentdocument categories 32/34
  • 33. First compress all documents to 2 numbers with an autoencoder Then use different colors for different documentcategories 33/34
  • 34. 3. What can it be used for? transliteration 34/34
  • 35. Thanks for your attendance Looking forward to present Recursive AutoEncoder 35/34