PhDThesis, Dr Shen Furao

47,848
-1

Published on

Published in: Technology, Education
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
47,848
On Slideshare
0
From Embeds
0
Number of Embeds
19
Actions
Shares
0
Downloads
56
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

PhDThesis, Dr Shen Furao

  1. 1. 1/39An Algorithm for IncrementalUnsupervised Learning andTopology Representation Shen Furao Hasegawa Lab Department of Computational Intelligence and Systems Science
  2. 2. 2/39Contents Chapter 1: Introduction Chapter 2: Vector Quantization Chapter 3: Adaptive Incremental LBG Chapter 4: Experiment of adaptive incremental LBG Chapter 5: Self-organizing incremental neural network Chapter 6: Experiment with artificial data Chapter 7: Application Chapter 8: Conclusion and discussion
  3. 3. 3/39 Introduction Clustering: Construct decision boundaries based on unlabeled data. Topology learning: find a topology structure that closely reflects the topology of the data distribution Online incremental learning: Adapt to new information without corrupting previously learned information
  4. 4. 4/39 Vector Quantization Targets  To minimize the average distortion through a suitable choice of codewords Application  Data compression, speech recognition  Separate the data set to Voronoi regions, find the centroid of the Voronoi regions LBG method (Linde, Buzo & Gray, 1980)  Dependence on initial starting conditions  Tendency to result in local minima
  5. 5. 5/39 Adaptive incremental LBG (Shen & Hasegawa, 2005) To solve the problem caused by poorly chosen initial conditions  independent of initial conditions With fixed number of codewords, to find a suitable codebook to minimize the distortion error MQE.  It can work better than or same as ELBG (Patane & Russo, 2001) With fixed distortion error, to minimize the number of codewords and find a suitable codebook.  Meaning: To get the same reconstruction quality for different vector set, the codebook will have different size and thus can save plenty of storage.
  6. 6. 6/39 Test Image Lena (512*512*8) is separated to 4*4 blocks. Such blocks are the input vectors. There are totally 16384 vectors. Peak Signal to Noise Ratio (PSNR) is used to evaluate the resulting images after the quantization process. 2552PSNR  10 log10 1  N i 1 ( f (i )  g (i )) 2 N Lena (512*512*8)
  7. 7. 7/39 Improvement I: Incrementally inserting codewords The optimal solution of k- clustering problem can be reachable from the (k- 1)-clustering problem.
  8. 8. 8/39 Improvement II: Distance measure function Within cluster distance must be significantly less than between cluster distance. ld ( x, c)  ( ( xi  ci ) 2 ) p i 1p  log10 q  1
  9. 9. 9/39 Improvement III: Delete and insert codewordDelete codewordwith lowest localdistortion errorInsert codewordnear the codewordwith highest localdistortion error
  10. 10. 10/39 Experiment 1 PSNRNumber ofcodewords LBG (Linde Mk (Lee et ELBG(Pata AILBG et al.,1980) al., 1997) ne, 2001) 256 31.60 31.92 31.94 32.01 512 32.49 33.09 33.14 33.22 1024 33.37 34.42 34.59 34.71 Meaning: With the same number of codewords, proposed method can get highest PSNR, i.e., with the same compression ratio, proposed method can get best reconstruction quality.
  11. 11. 11/39 Experiment 2 Number of codewords PSNR ELBG (Patane, AILBG 2001) 31.94 256 244 33.14 512 488 34.59 1024 988Meaning:• With a predefined reconstruction quality, proposed method can find a good codebook with reasonable number of codewords.
  12. 12. 12/39Experiment 3: Original Images Boat Gray21
  13. 13. 13/39 Results of experiment 3 PSNR Number of codewords (dB) Gray21 Lena Boat 28.0 9 22 54 30.0 12 76 199 33.0 15 454 1018Meaning:1. For different images, with the same PSNR, number of codewords will be different.2. Proposed method can be used to set up an image database with same reconstruction quality (PSNR)
  14. 14. 14/39 Unsupervised learning Clustering  K-means (King, 1967), ELBG (Patane, 2001), Global k-means (Likas, 2003), AILBG (Shen, 2005)  Determine the number of clusters k in advance  data sets consisting only of isotropic clusters  Single-link (Sneath, 1973), complete-link (King, 1967), CURE (Guha, 1998)  Computation overload, much memory space  Unsuitable for large data sets or online data Topology Learning: Reflects topology of high-dimension data distribution  SOM (Kohonen, 1982): predetermined structure and size  CHL+NG (Martinetz, 1994): a priori decision about the network size  GNG (Fritzke, 1995): permanent increase in the number of nodes Online Learning  GNG-U (Frutzke, 1998): destroy learned knowledge  LLCS (Hamker, 2001): supervised learning
  15. 15. 15/39 Self-organizing incremental neural network (Shen & Hasegawa, 2005)1. To process the on-line non-stationary data.2. To do the unsupervised learning without any priori condition such as: • suitable number of nodes • a good initial codebook • how many classes there are3. Report a suitable number of classes4. Represent the topological structure of the input probability density.5. Separate the classes with some low-density overlaps6. Detect the main structure of clusters polluted by noises
  16. 16. 16/39 The Proposed algorithm First Layer Second LayerInput Growing First Growing Secondpattern Network Output Network Output Insert Delete Classify Node Node
  17. 17. 17/39 Algorithms Insert new nodes  Criterion: nodes with high errors serve as a criterion to insert a new node  error-radius is used to judge if the insert is successful Delete nodes  Criterion: remove nodes in low probability density regions  Realize: delete nodes with no or only one direct topology neighbor Classify  Criterion: all nodes linked with edges will be one cluster
  18. 18. 18/39First-layer Second-layer Input signals== Initialize multiple of  Input signal Within-class Insertion Find winner Judge if insertion and second winner is successful Delete overlap and Y Between-class noise nodes Insertion N N Input signals== Connect winner multiple of LT and second winner Y Update weight of First-layer Y winner and neighbor N Output results
  19. 19. 19/39 Experiment Environment I II III IV V VI VII A 1 0 1 0 0 0 0 B 0 1 0 1 0 0 0 C 0 0 1 0 0 1 0 D 0 0 0 1 1 0 0 E1 0 0 0 0 1 0 0 E2 0 0 0 0 0 1 0Original Data Set E3 0 0 0 0 0 0 1
  20. 20. 20/39 Experiment: Stationary environmentOriginal Data Set GNG (Fritzke, 1995)
  21. 21. 21/39 Experiment: Stationary environmentProposed method: first layer Proposed method: final results
  22. 22. 22/39 Experiment: Non-stationary environmentGNG (Fritzke, 1995) GNG-U (Fritzke, 1998)
  23. 23. 23/39Experiment:Non-stationary environment Proposed method: first layer
  24. 24. 24/39Experiment:Non-stationary environment Proposed method: first layer
  25. 25. 25/39Experiment:Non-stationary environment Proposed method: first layer
  26. 26. 26/39 Experiment: Non-stationary environmentProposed method: first layer Proposed method: Final output
  27. 27. 27/39Application: Face recognition(ATT_FACE)Facial Image (a) 10 classes (b) 10 samples of class 1
  28. 28. 28/39Face recognition: Feature Vector Vector of (a) Vector of (b)
  29. 29. 29/39Face Recognition: results 10 clusters Stationary Correct Recognition Ratio: 90% Non-Stationary Correct Recognition Ratio: 86%
  30. 30. 30/39Application: Vector Quantization Stationary Environment: DecodingOriginal Lena (512*512*8) image, 130 nodes, 0.45bpp, PSNR = 30.79dB
  31. 31. 31/39 Vector Quantization: Compare with GNG Stationary Environment Number of bpp PSNR Nodes First-layer 130 0.45 30.79GNG (Fritzke, 130 0.45 29.98 1995)Second-layer 52 0.34 29.29 GNG 52 0.34 28.61
  32. 32. 32/39 Vector Quantization: Non-stationary EnvironmentFirst-layer: 499 nodes, 0.56bpp, Second-layer: 64 nodes, 0.375bpp,PSNR = 32.91dB PSNR = 29.66dB
  33. 33. 33/39 Application: Handwritten character recognition Optical Recognition of Handwritten Digits database (optdigits) (UCI repository, 1996)  10 classes (handwritten digits) from a total of 43 people  30 contributed to the training set, 3823 samples  Different 13 to the test set, 1797 samples  Dimension of the samples is 64 Method:  Train: A separate SOINN to describe each class of data  Test: Classify an unknown data point according to whichever model gives the best match (nearest neighbor)
  34. 34. 34/39 Optdigits: Comparison with 1-NN Proposed method 1-NN (1) (2) (3) (4)Recognition 98% 98.5% 97.1% 96.5% 96.0% ratio No. of 3823 845 544 415 334 prototype Speed up 1 4.53 7.02 9.21 11.45 (times) Memory 100% 22.1% 14.2% 10.8% 8.7%
  35. 35. 35/39 Optdigits: Comparison with SVM Improved SVM Traditional SVM (Passerini, 2002) Proposed method One-vs-All All-pairs One-vs-All All-pairsRecognition 97.2 97.4 98.2 98.1 98.5 ratioGaussian Kernel
  36. 36. 36/39Application: others Humanoid robot Scene recognition Texture recognition Semi-supervised learning
  37. 37. 37/39 Journal papers (2003~2005)1. Shen Furao & Osamu Hasegawa, “An adaptive incremental LBG for vector quantization,” Neural Networks, accepted.2. Shen Furao & Osamu Hasegawa, “An incremental network for on- line unsupervised classification and topology learning,” Neural Networks, accepted.3. Shen Furao & Osamu Hasegawa, Fractal image coding with simulated annealing search, Journal of Advanced Computational Intelligence and Intelligent Informatics, Vol.9, No.1, pp.80-88, 2005.4. Shen Furao & Osamu Hasegawa, A fast no search fractal image coding method, Signal Processing: Image Communication, vol.19, pp.393-404, (2004)5. Shen Furao & Osamu Hasegawa, A growing neural network for online unsupervised learning, Journal of Advanced Computational Intelligence and Intelligent Informatics, Vol.8, No.2, pp.121-129, (2004)
  38. 38. 38/39 Refereed International Conference (2003~2005)1. Shen Furao, Youki Kamiya & Osamu Hasegawa, “An incremental neural network for online supervised learning and topology representation,” 12th International Conference on Neural Information Processing (ICONIP 2005), Taipei, Taiwan, October 30 - November 2, 2005, accepted.2. Shen Furao & Osamu Hasegawa, “An incremental k-means clustering algorithm with adaptive distance measure,” 12th International Conference on Neural Information Processing (ICONIP 2005), Taipei, Taiwan, October 30 - November 2, 2005, accepted.3. Shen Furao & Osamu Hasegawa, “An on-line learning mechanism for unsupervised classification and topology representation,” IEEE Computer Society International Conference on Computer Vision and Pattern Recognition (CVPR 2005), San Diego, CA, USA, June 21-26, 2005.4. Shen Furao & Osamu Hasegawa, “An incremental neural network for non-stationary unsupervised learning,” 11th International Conference on Neural Information Processing (ICONIP 2004), Calcutta, India, November 22-25, 2004.5. Shen Furao & Osamu Hasegawa, “An effective fractal image coding method without search,” IEEE International Conference on Image Processing (ICIP 2004), Singapore, October 24-27, 2004.6. Youki Kamiya, Shen Furao & Osamu Hasegawa, “Non-stop learning : a new scheme for continuous learning and recognition,” Joint 2nd SCIS and 5th ISIS, Keio University, Yokohama, Japan, September 21-24, 2004.7. Osamu Hasegawa & Shen Furao, “A self-structurizing neural network for online incremental learning,” CD-ROM SICE Annual Conference in Sapporo, FAII-5-2, August 4-6, 2004.8. Shen Furao & Osamu Hasegawa, “A self-organized growing network for on-line unsupervised learning,” 2004 International Joint Conference on Neural Networks (IJCNN 2004), Budapest, Hungary, CD-ROM ISBN 0-7803-8360-5, Vol.1, pp.11-16, 2004.9. Shen Furao & Osamu Hasegawa, “A fast and less loss fractal image coding method using simulated annealing,” 7th Joint Conference on Information Science (JCIS 2003), Cary, North Carolina, USA, September 26-30, 2003.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×