Downloaded 21 times
Uploaded bySafaa Alnabulsi
Comparison of Learning Algorithms for Handwritten Digit Recognition
This document compares different machine learning algorithms for handwritten digit recognition on the MNIST dataset. Convolutional neural networks achieved the best results, with LeNet5 achieving 0.9% error and boosted LeNet4 achieving the lowest error rate of 0.7%. Neural networks required more training time but had faster recognition times and lower memory requirements compared to nearest neighbor classifiers. Overall, convolutional neural networks were best suited for handwritten digit recognition due to their ability to handle variations in size, position and orientation of digits.
More Related Content
Similar to Comparison of Learning Algorithms for Handwritten Digit Recognition
Comparison of Learning Algorithms for Handwritten Digit Recognition
- 1. Comparison of learningalgorithms for handwritten digit recognition Y LeCun, L Jackel, L Bottou, A Brunot, C Cortes, J Denker, H, Drucker, I Guyon, U Muller, E Sackinger, P Simard, and V Vapnik 1995 Author | Safaa Alnabulsi
- 2. Index ďľ Introduction ďľ Database ďľThe Classifiers ďś Linear Classifiers ďś Nearest Neighbor Classifiers ďś Neural Networks Classifiers ďś Convolutional Neural Networks Classifiers ďľ Discussion ďľ Conclusions Author | Safaa Alnabulsi
- 3. Introduction ďľ This papercompares the relative merits of several classification algorithms develop ed at Bell Laboratories and elsewhere for the purpose of recognizing handwritten digits. ďľ It is an excellent benchmark for comparing shapes not only digits. ďľ They consider: o Raw accuracy o Rejection training time o Recognition time o Memory requirement Author | Safaa Alnabulsi
- 4. Database ďľ The MNISTdatabase of handwritten digits was constructed from NIST's Special Database 3 and Special Database 1 which contain binary images of handwritten digits: ⢠Training set was composed of 60,000 pattern contained examples from approximately 250 disjoint writes. ⢠Test set was composed of 10,000 patterns. ďľ All the images were size-normalized to fit in a 20x20 pixel box while preserving the aspect ratio. Author | Safaa Alnabulsi
- 5. The Classifiers Author |Safaa Alnabulsi LINEAR NEAREST NEIGHBOR NEURAL NETWORK CONVOLUTIONAL NEURAL NETWORK
- 6. Linear Classifiers ďľ Baseline LinearClassifier ďľ Pairwise Linear Classifier ďľ PCA and Polynomial Classifier ďľ Optimal Margin OMC Author | Safaa Alnabulsi
- 7. Baseline Linear Classifier ďľThe simplest classifier. Each input pixel value contributes to a weighted sum for each output unit. ďľ The output unit with the highest sum indicates the class of the input character. ďľ Thus, as we can see, the image is treated as a 1D vector and connected to a 10-output vector. ďľ The test error rate is 8.4%. Author | Safaa Alnabulsi
- 8. Pairwise Linear Classifier ďľA simple improvement of the basic linear classifier. The idea is to train each unit of a single-layer network to classify one class from one other class. ďľ The final score for class i is : the sum of the outputs all the units labelled i/z minus the sum of the output of all the units labelled y/i, for all z and y. ďľ Error rate on the test set was 7.6%, only slightly better than a linear classifier. Author | Safaa Alnabulsi 45
- 9. PCA and PolynomialClassifier ďľ This classifier can be seen as a linear classifier with 821 inputs, preceded by a stage which computes the projection of the input pattern on the 40 principal components of the set of training vectors. ďľ The 40.dimensional feature vector was used as the input of a second degree polynomial classifier. ďľ Error on the test set was 3.3%. Author | Safaa Alnabulsi From âHandbook Of Character Recognition And Document Image Analysisâ Page 111
- 10. Optimal Margin Classifier(OMC) ďľ OMC is called SVM now, which constructs a hyperplane or set of hyperplanes in a high or infinite-dimensional space, which can be used for classification. ďľ Best hyperplane is the one that represents the largest separation, or margin, between the two classes ďľ Using Regular SVM, a test error of 1.4% was reached. ďľ Whereas, using a slighlty different techinqe, Soft Margin Classifier (Cortes & Vapnik ) with a 4-th degree decision surface, a test error of 1.1% was reached. Author | Safaa Alnabulsi
- 11. Nearest neighbor Classifiers ďľ Baseline NearestNeighbor Classifier ďľ Tangent Distance Classifier (TDCs) Author | Safaa Alnabulsi
- 12. Baseline Nearest NeighborClassifier ďľ Another simple classifier with a Euclidean distance measure between input images. ďľ It would operate on feature vectors rather than directly on the pixels ďľ No training time and no brain on the part of the designer ďľ The memory requirement and recognition time are large ďľ Deslanted 20x20 images were used. ďľ The test error for k = 3 is 2.4%. Author | Safaa Alnabulsi
- 13. Tangent Distance Classifier(TDC) Author | Safaa Alnabulsi
- 14. Tangent Distance Classifier(TDC) ďľ It is a nearest-neighbor method where the distance function is made insensitive to small distortions and translations of the input image. ďľ Tangent plane ? If we consider an image as a point in a high dimensional pixel space then an evolving distortion traces out a curve in pixel space.Taken together, all these distortions define a low-dimensional manifold in pixel space which can be approximated by a tangent plane. ďľ An excellent measure of âclosenessâfor character images is the distance between their tangent planes ďľ A test error rate of 1.1% was achieved using 16x16 pixel images. Author | Safaa Alnabulsi
- 15. Neural Networks Classifiers ďľ Radial BasisFunction Network ďľ Large Fully Connected Multi-Layer Neural Network Author | Safaa Alnabulsi
- 16. Radial Basis FunctionNetwork ďľ Architecture: ⢠The first layer was composed of 1000 Gaussian RBF units with 400 inputs (20x20). The RBF units were divided into 10 groups of 100. ⢠The second layer was a simple 100 linear classifier. ďľ Training: ⢠Each group of units was trained on all the training examples of one of the 10 classes using the adaptive K-means algorithm. ďľ Error rate on the test set was 3.6% Author | Safaa Alnabulsi
- 17. Large Fully ConnectedMulti-Layer Neural Network ďľ Architecture: ⢠Two layers of weights (one hidden layer) ďľ Training: ⢠Each network trained with various numbers of hidden units. ⢠Deslanted 20x20 images were used. ⢠As the learning proceeds, the weights grow, which progressively increases the effective capacity of the network. ďľ The best result was 1.6% on the test set. Author | Safaa Alnabulsi
- 18. Convolutional Neural Networks Classifiers ďľ Letet1 ďľ LeNet4 ďľLeNet5 ďľ Boosted LeNet4 Author | Safaa Alnabulsi
- 19. Motiviation Behind CNN ďľTo solve the dilemma between small networks that cannot learn the training set, and large networks that seem overparameterized, one can design specialized network architectures that are specifically designed to recognize two-dimensional shapes such as digits, while eliminating irrelevant distortions and variability. These considerations lead us to the idea of convolutional network. Author | Safaa Alnabulsi
- 20. LeNet1 ďľ Because ofLeNet 1âs small input field, the images were down-sampled to 16x16 pixels and centered in the 28x28 input layer. ďľ Small number of free parameters, only about 3000. ďľ LeNet 1 achieved 1.7% test error. Author | Safaa Alnabulsi
- 21. LeNet4 ďľ LeNet 4was designed to address the problem of large size of the training. ďľ It is an expanded version of LeNet 1 that has a 32x32 input layer in which the 20x20 images (not deslanted) were centered by center of mass. ďľ It includes more feature maps and an additional layer of hidden units that is fully connected to both the last layer of features maps and to the output units. ďľ LeNet 4 contains about 260,000 connections and has about 17,000 free parameters. ďľ Test error was 1.1%. Author | Safaa Alnabulsi
- 22. LeNet5 ďľ LeNet 5,has an architecture similar to LeNet 4, but has more feature maps, a larger fully-connected layer. ďľ LeNet 5 has a total of about 340,000 connections, and 60,000 free parameters, most of them in the last two layers. ďľ the training procedure included a module that distorts the input images during training using randomly picked affine transformations (shift, scaling, rotation, and procedureing small skewing). ďľ It achieved 0.9% error. Author | Safaa Alnabulsi
- 23. Boosted LeNet4 Author |Safaa Alnabulsi
- 24. Boosted LeNet4 ďľ ThreeLeNet 4 are combined: ⢠The first one is trained the usual way. ⢠The second one is trained on a mix of patterns that are filtered by the first net (50% of which the first net got right, and 50% of which it got wrong). ⢠The third net is trained on new patterns on which the first and the second nets disagree. ďľ During testing, the outputs of the three nets are simply added. ďľ The test error rate was 0.7%, the best of any of our classifiers. ďľ At first glance, bossting appears to be three times more expensive as a single net. In fact, when the first net produces a high confidence answer, the other nets are not called. The cost is bout 1.75 times that of a single net. Author | Safaa Alnabulsi
- 25. Discussion Author | SafaaAlnabulsi ERROR RATE REJECTION TRAINING TIME TRAINING TIME MEMORY
- 26. Discussion â ErrorRate Author | Safaa Alnabulsi ďľ Boosted LeNet 4 is clearly the best, achieving score of 0.7%, closely followed by LeNet 5 at 0.9%. ďľ This can be compared to our estimate of human performance , 0.2%
- 27. Discussion â RejectionTraining Time Author | Safaa Alnabulsi ďľ In many applications, rejection performance is more significant than raw error rate. ďľ Again Boosted LeNet 4 has the best score.
- 28. Discussion â TrainigTime Author | Safaa Alnabulsi ďľ K-nearest neighbors and TDC have essentially zero training time. ďľ While the single-layer net, the pairwise net, and PCA+quadratic net could be trained in less than an hour, ďľ the multilayer net training times were expectedly much longer: 3 days for LeNet 1, 7 days for the fully connected net, 2 weeks for LeNet 4 and 5, and about a month for boosted LeNet 4. Training the Soft Margin classifier took about 10 days.
- 29. Discussion â Memory Author| Safaa Alnabulsi ďľ Memory requirements for the neural networks assume 4 bytes. ďľ Of the high-accuracy classifiers, LeNet 4 requires the least memory.
- 30. Conclusions ďľ Performance dependson many factors including high accuracy, low run time, and low memory requirements. ďľ Furture: As computer technology improves, larger capacity recognizers become feasible. The neural nets advantage will become more striking as training databases continue to increase in size. ďľ Boosting: We find that boosting gives a substantial improvement in accuracy, with a relatively modest penalty in memory and computing expense. ďľ Training Data: When plenty of data is available, many methods can attain respectable accuracy. ďľ Optimal margin classifier: it has excellent accuracy, which is most remarkable, because unlike the other high performance classifiers, it does not include a priori knowledge about the problem. It is still much slower and memory hungry than the convolutional nets. ďľ Convolutional networks: are particularly well suited for recognizing or rejecting shapes with widely varying size, position, and orientation. ďľ Trained neural networks can run much faster and require much less space than memory-based techniques. Author | Safaa Alnabulsi
- 31. Author | SafaaAlnabulsi
Editor's Notes
- #8Â The simplest classifier
- #9Â For the n(=10) classes you build all n(n-1)/2 = 45 Binary classifiers, denoted by i/j where i and j are different classes. The i/z classifier output tells what makes i favorable over class z. On the other hand x/i tells what speaks against i compared to class x. Then you add up all 9 unique comparisons where i is either left or right of the dash. If i is right, you should note that x/i so to say equals -i/x.
- #10Â To compute the principal components: the mean of each input component was first computed and subtracted from the training vectors. The covariance matrix of the resulting vectors was then computed, and diagonalized using Singular Value Decomposition (SVD).
- #11 Challange: Polynomial classifiers are well studied methods for generating complex decision surfaces. Unfortunately, they are impractical for high-dimensional problems. One reasonable choice as the best hyperplane is the one that represents the largest separation, or margin, between the two classes. So we choose the hyperplane so that the distance from it to the nearest data point on each side is maximized. If such a hyperplane exists, it is known as the maximum-margin hyperplane and the linear classifier it defines is known as a maximum-margin classifier; or equivalently, the perceptron of optimal stability.[citation needed] SVM? More formally, a support-vector machine constructs a hyperplane or set of hyperplanes in a high- or infinite-dimensional space, which can be used for classification, regression, or other tasks like outliers detection. Intuitively, a good separation is achieved by the hyperplane that has the largest distance to the nearest training-data point of any class (so-called functional margin), since in general the larger the margin, the lower the generalization error of the classifier.[4] The drawing: H1 does not separate the classes. H2 does, but only with a small margin. H3 separates them with the maximal margin. Additional info: In addition to performing linear classification, SVMs can efficiently perform a non-linear classification using what is called the kernel trick, implicitly mapping their inputs into high-dimensional feature spaces.
- #13Â Naturally, a realistic Euclidean distance nearest-neighbor system would operate on feature vectors rather than directly on the pixels
- #14Â an unlabeled image of a "9" must be classified by finding the closest prototype image out of two images representing respectively a "9" and a "4". According to the Euclidean distance (sum of the squares of the pixel to pixel differences), the "4" is closer even though the "9" is much more similar once it has been rotated and thickened. The result is an incorrect classification. The key idea is to construct a distance measure which is invariant with respect to some chosen transformations such as translation, rotation and others.
- #15Â Explainaition of the picture from paper below: P, E are patterns Sp, Se are manifolds, obtained through small transformations of P such as (rotation, translation, scaling, etc.). he Euclidean distance between two patterns P and E is in general not appropriate because it is sensitive to irrelevant transformations of P and of E. In contrast, the distance D(E, P) defined to be the minimal distance between the two manifolds Sp and SE is truly invariant with respect to the transformation used to generate Sp and SE. Unfortunately, these manifolds have no analytic expression in general, and finding the distance between them is a hard optimization problem with multiple local minima. Besides, t.rue invariance is not necessarily desirable since a rotation ofa "6" into a "9" does not preserve the correct classification. https://pdfs.semanticscholar.org/8314/dda1ec43ce57ff877f8f02ed89acb68ca035.pdf
- #17 Radial basis function (RBF) networks typically have three layers: an input layer, a hidden layer with a non-linear RBF activation function and a linear output layer. The second layer weights were computed using a regularized pseudo-inverse method.
- #20 Convolution: extract features from the input image. By using âfeature mapâA shared filter (therefore small number of parameters) specifically designed for the data-type at hand (here pictures), that, when trained implicitly learns structured features such as edges in the picture. Pooling or Sub Sampling: reduces the dimensionality of each feature map but retains the most important information. (parameter-free) Classification (Fully Connected Layer)
- #21Â It should be intuitively clear to the audience that convolutions + (down-sampling) lead to small number of parameters, and that mixing those with fully connected layers is still more parameter-efficient compared to deep fully connected networks.
- #22Â In previous experiments with ZIP code data, replacing the last layer of LeNet 4 with a Euclidean Nearest Neighbor classifier, and with the âlocal learningâ method of Bottou and Vapnik, in which a local linear classifier is retrained each time a new test pattern is shown. Neither of those improve the raw error rate, although they did improve the rejection
- #24Â Boosting is a technique to combine the results from several/many weak classifiers to get a more accurate results
- #27Â Boosted LeNet 4 is clearly the best, achieving score of 0.7%, closely followed by LeNet 5 at 0.9%. This can be compared to our estimate of human performance , 0.2%
- #28Â In many applications, rejection performance is more significant than raw error rate. Again boosted LeNet 4 has the best score. The enhanced LeNet 4 did better than original LeNet 4.
- #29 Expectedly, memory-based method are much slower than neural networks. Single-board hardware designed with LeNet in mind performs recognition at 1000 characters/sec (Säckinger & Graf 94). Cost-effective hardware implementations of memory-based techniques are more elusive, due to their enormous memory requirements. Training time was also measured. However, while the training time is marginally relevant to the designer, it is totally irrelevant to the customer.
- #31Â training time