Your SlideShare is downloading. ×
Simple Introduction to AutoEncoder
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Simple Introduction to AutoEncoder

10,101

Published on

Simple Introduction to AutoEncoder

Simple Introduction to AutoEncoder

Published in: Education
0 Comments
20 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
10,101
On Slideshare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
0
Comments
0
Likes
20
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Simple Introduction to AutoEncoder Lang JunDeep Learning Study Group, HLT, I2R 17 August, 2012
  • 2. Outline1. What is AutoEncoder? Input = decoder(encoder(input))2. How to train AutoEncoder? pre-training3. What can it be used for? reduce dimensionality 2/34
  • 3. 1. What is AutoEncoder?➢ Multilayer neural net simple review 3/34
  • 4. 1. What is AutoEncoder?➢ Multilayer neural net simple review 4/34
  • 5. 1. What is AutoEncoder?➢ Multilayer neural net simple review 5/34
  • 6. 1. What is AutoEncoder?➢ Multilayer neural net simple review 6/34
  • 7. 1. What is AutoEncoder?➢ Multilayer neural net simple review 7/34
  • 8. 1. What is AutoEncoder?➢ Multilayer neural net simple review 8/34
  • 9. 1. What is AutoEncoder?➢ Multilayer neural net simple review 9/34
  • 10. 1. What is AutoEncoder?➢ Multilayer neural net simple review 10/34
  • 11. 1. What is AutoEncoder?➢ Multilayer neural net simple review 11/34
  • 12. 1. What is AutoEncoder?➢ Multilayer neural net simple review 12/34
  • 13. 1. What is AutoEncoder?➢ Multilayer neural net simple review 13/34
  • 14. 1. What is AutoEncoder?➢ Multilayer neural net simple review 14/34
  • 15. 1. What is AutoEncoder?➢ Multilayer neural net simple review 15/34
  • 16. 1. What is AutoEncoder?➢ Multilayer neural net simple review 16/34
  • 17. 1. What is AutoEncoder?➢ Multilayer neural net simple review 17/34
  • 18. 1. What is AutoEncoder?➢ Multilayer neural net simple review 18/34
  • 19. 1. What is AutoEncoder?➢ Multilayer neural net simple review 19/34
  • 20. 1. What is AutoEncoder?➢ Multilayer neural net simple review 20/34
  • 21. 1. What is AutoEncoder?➢ Multilayer neural net simple review 21/34
  • 22. 1. What is AutoEncoder?➢ Multilayer neural net simple review 22/34
  • 23. 1. What is AutoEncoder?➢ Multilayer neural net simple review 23/34
  • 24. 1. What is AutoEncoder?➢ Multilayer neural net with target output = input➢ Reconstruction=decoder(encoder(input))➢ Minimizing reconstruction error➢ Probable inputs have small reconstruction error 24/34
  • 25. 2. How to train AutoEncoder? Hinton (2006) Science PaperRestricted Boltzmann Machine(RBM) 25/34
  • 26. 2. How to train AutoEncoder? Hinton (2006) Science Paperrestricted Boltzmann machine 26/34
  • 27. Effective deep learning becamepossible through unsupervised pre- training Purely supervised neural net With unsupervised pre‐training (with RBMs and Denoising Auto-Encoders) 27/34 0–9 handwritten digit recognition error rate (MNIST data)
  • 28. Why is unsupervised pre-training working so well?Regularization hypothesis: Representations goodfor P(x) are good for P(y|x)Optimization hypothesis: Unsupervised initializationsstart near better local minimum of supervised training error Minima otherwise notachievable by randominitializationErhan, Courville, Manzagol, Vincent, Bengio (JMLR, 2010) 28/34
  • 29. 3. What can it be used for? illustration for images 29/34
  • 30. 3. What can it be used for? document retrieval output2000 reconstructed counts vector • We train the neural network 500 neurons to reproduce its input vector as its output • This forces it to compress as 250 neurons much information as possible into the 10 numbers in the central bottleneck. 10 • These 10 numbers are then a good way to compare documents. 250 neurons – See Ruslan Salakhutdinov’s talk 500 neurons input 30/34 2000 word counts vector
  • 31. 3. What can it be used for? visualize documents output 2000 reconstructed counts vector• Instead of using codes to retrieve documents, we can 500 neurons use 2-D codes to visualize sets of documents. – This works much better 250 neurons than 2-D PCA 2 250 neurons 500 neurons input 31/34 2000 word counts vector
  • 32. First compress all documents to 2 numbers using a type of PCA Then use different colors for differentdocument categories 32/34
  • 33. First compress all documents to 2 numbers with an autoencoder Then use different colors for different documentcategories 33/34
  • 34. 3. What can it be used for? transliteration 34/34
  • 35. Thanks for your attendance Looking forward to present Recursive AutoEncoder 35/34

×