This document describes a machine learning project that uses support vector machines (SVM) and k-nearest neighbors (k-NN) algorithms to segment gesture phases based on radial basis function (RBF) kernels and k-nearest neighbors. The project aims to classify frames of movement data into five gesture phases (rest, preparation, stroke, hold, retraction) using two classifiers. The SVM approach achieved 53.27% accuracy on test data while the k-NN approach achieved significantly higher accuracy of 92.53%. The document provides details on the dataset, feature extraction methods, model selection process and results of applying each classifier to the test data.
We consider the problem of finding anomalies in high-dimensional data using popular PCA based anomaly scores. The naive algorithms for computing these scores explicitly compute the PCA of the covariance matrix which uses space quadratic in the dimensionality of the data. We give the first streaming algorithms
that use space that is linear or sublinear in the dimension. We prove general results showing that any sketch of a matrix that satisfies a certain operator norm guarantee can be used to approximate these scores. We instantiate these results with powerful matrix sketching techniques such as Frequent Directions and random projections to derive efficient and practical algorithms for these problems, which we validate over real-world data sets. Our main technical contribution is to prove matrix perturbation
inequalities for operators arising in the computation of these measures.
-Proceedings: https://arxiv.org/abs/1804.03065
-Origin: https://arxiv.org/abs/1804.03065
Time Series Forecasting Using Recurrent Neural Network and Vector Autoregress...Databricks
Given the resurgence of neural network-based techniques in recent years, it is important for data science practitioner to understand how to apply these techniques and the tradeoffs between neural network-based and traditional statistical methods.
This lecture discusses two specific techniques: Vector Autoregressive (VAR) Models and Recurrent Neural Network (RNN). The former is one of the most important class of multivariate time series statistical models applied in finance while the latter is a neural network architecture that is suitable for time series forecasting. I’ll demonstrate how they are implemented in practice and compares their advantages and disadvantages. Real-world applications, demonstrated using python and Spark, are used to illustrate these techniques. While not the focus in this lecture, exploratory time series data analysis using time-series plot, plots of autocorrelation (i.e. correlogram), plots of partial autocorrelation, plots of cross-correlations, histogram, and kernel density plot, will also be included in the demo.
The attendees will learn – the formulation of a time series forecasting problem statement in context of VAR and RNN – the application of Recurrent Neural Network-based techniques in time series forecasting – the application of Vector Autoregressive Models in multivariate time series forecasting – the pros and cons of using VAR and RNN-based techniques in the context of financial time series forecasting – When to use VAR and when to use RNN-based techniques
We consider the problem of finding anomalies in high-dimensional data using popular PCA based anomaly scores. The naive algorithms for computing these scores explicitly compute the PCA of the covariance matrix which uses space quadratic in the dimensionality of the data. We give the first streaming algorithms
that use space that is linear or sublinear in the dimension. We prove general results showing that any sketch of a matrix that satisfies a certain operator norm guarantee can be used to approximate these scores. We instantiate these results with powerful matrix sketching techniques such as Frequent Directions and random projections to derive efficient and practical algorithms for these problems, which we validate over real-world data sets. Our main technical contribution is to prove matrix perturbation
inequalities for operators arising in the computation of these measures.
-Proceedings: https://arxiv.org/abs/1804.03065
-Origin: https://arxiv.org/abs/1804.03065
Time Series Forecasting Using Recurrent Neural Network and Vector Autoregress...Databricks
Given the resurgence of neural network-based techniques in recent years, it is important for data science practitioner to understand how to apply these techniques and the tradeoffs between neural network-based and traditional statistical methods.
This lecture discusses two specific techniques: Vector Autoregressive (VAR) Models and Recurrent Neural Network (RNN). The former is one of the most important class of multivariate time series statistical models applied in finance while the latter is a neural network architecture that is suitable for time series forecasting. I’ll demonstrate how they are implemented in practice and compares their advantages and disadvantages. Real-world applications, demonstrated using python and Spark, are used to illustrate these techniques. While not the focus in this lecture, exploratory time series data analysis using time-series plot, plots of autocorrelation (i.e. correlogram), plots of partial autocorrelation, plots of cross-correlations, histogram, and kernel density plot, will also be included in the demo.
The attendees will learn – the formulation of a time series forecasting problem statement in context of VAR and RNN – the application of Recurrent Neural Network-based techniques in time series forecasting – the application of Vector Autoregressive Models in multivariate time series forecasting – the pros and cons of using VAR and RNN-based techniques in the context of financial time series forecasting – When to use VAR and when to use RNN-based techniques
Machine learning in science and industry — day 4arogozhnikov
- tabular data approach to machine learning and when it didn't work
- convolutional neural networks and their application
- deep learning: history and today
- generative adversarial networks
- finding optimal hyperparameters
- joint embeddings
Welcome to the Supervised Machine Learning and Data Sciences.
Algorithms for building models. Support Vector Machines.
Classification algorithm explanation and code in Python ( SVM ) .
DESIGN OF DELAY COMPUTATION METHOD FOR CYCLOTOMIC FAST FOURIER TRANSFORMsipij
In this paper the Delay Computation method for Common Sub expression Elimination algorithm is being implemented on Cyclotomic Fast Fourier Transform. The Common Sub Expression Elimination algorithm is combined with the delay computing method and is known as Gate Level Delay Computation with Common Sub expression Elimination Algorithm. Common sub expression elimination is effective
optimization method used to reduce adders in cyclotomic Fourier transform. The delay computing method is based on delay matrix and suitable for implementation with computers. The Gate level delay computation method is used to find critical path delay and it is analyzed on various finite field elements. The presented algorithm is established through a case study in Cyclotomic Fast Fourier Transform over finite field. If Cyclotomic Fast Fourier Transform is implemented directly then the system will have high additive complexities. So by using GLDC-CSE algorithm on cyclotomic fast Fourier transform, the additive
complexities will be reduced and also the area and area delay product will be reduced.
F. Petroni, L. Querzoni, R. Beraldi, M. Paolucci:
"LCBM: Statistics-Based Parallel Collaborative Filtering."
In: Proceedings of the 17th International Conference on Business Information Systems (BIS), 2014.
Abstract: "In the last ten years, recommendation systems evolved from novelties to powerful business tools, deeply changing the internet industry. Collaborative Filtering (CF) represents today’s a widely adopted strategy to build recommendation engines. The most advanced CF techniques (i.e. those based on matrix factorization) provide high quality results, but may incur prohibitive computational costs when applied to very large data sets. In this paper we present Linear Classifier of Beta distributions Means (LCBM), a novel collaborative filtering algorithm for binary ratings that is (i) inherently parallelizable and (ii) provides results whose quality is on-par with state-of-the-art solutions (iii) at a fraction of the computational cost."
Time Series Forecasting Using Recurrent Neural Network and Vector Autoregress...Databricks
Given the resurgence of neural network-based techniques in recent years, it is important for data science practitioner to understand how to apply these techniques and the tradeoffs between neural network-based and traditional statistical methods.
This lecture discusses two specific techniques: Vector Autoregressive (VAR) Models and Recurrent Neural Network (RNN). The former is one of the most important class of multivariate time series statistical models applied in finance while the latter is a neural network architecture that is suitable for time series forecasting. I’ll demonstrate how they are implemented in practice and compares their advantages and disadvantages. Real-world applications, demonstrated using python and Spark, are used to illustrate these techniques. While not the focus in this lecture, exploratory time series data analysis using time-series plot, plots of autocorrelation (i.e. correlogram), plots of partial autocorrelation, plots of cross-correlations, histogram, and kernel density plot, will also be included in the demo.
The attendees will learn – the formulation of a time series forecasting problem statement in context of VAR and RNN – the application of Recurrent Neural Network-based techniques in time series forecasting – the application of Vector Autoregressive Models in multivariate time series forecasting – the pros and cons of using VAR and RNN-based techniques in the context of financial time series forecasting – When to use VAR and when to use RNN-based techniques
Isobar Human Media - Social Media Marketing 2009Isobar
http://www.isobar.nl/humanmedia/
Isobar Human Media is de strategische advies partner bij uitstek voor het ontwikkelen en uitvoeren van Social Media strategiën.
Isobar Human Media is hét centrale punt waar alle specialistisch kennis op het gebied van Social Media uit het Isobar netwerk samenkomt.
Machine learning in science and industry — day 4arogozhnikov
- tabular data approach to machine learning and when it didn't work
- convolutional neural networks and their application
- deep learning: history and today
- generative adversarial networks
- finding optimal hyperparameters
- joint embeddings
Welcome to the Supervised Machine Learning and Data Sciences.
Algorithms for building models. Support Vector Machines.
Classification algorithm explanation and code in Python ( SVM ) .
DESIGN OF DELAY COMPUTATION METHOD FOR CYCLOTOMIC FAST FOURIER TRANSFORMsipij
In this paper the Delay Computation method for Common Sub expression Elimination algorithm is being implemented on Cyclotomic Fast Fourier Transform. The Common Sub Expression Elimination algorithm is combined with the delay computing method and is known as Gate Level Delay Computation with Common Sub expression Elimination Algorithm. Common sub expression elimination is effective
optimization method used to reduce adders in cyclotomic Fourier transform. The delay computing method is based on delay matrix and suitable for implementation with computers. The Gate level delay computation method is used to find critical path delay and it is analyzed on various finite field elements. The presented algorithm is established through a case study in Cyclotomic Fast Fourier Transform over finite field. If Cyclotomic Fast Fourier Transform is implemented directly then the system will have high additive complexities. So by using GLDC-CSE algorithm on cyclotomic fast Fourier transform, the additive
complexities will be reduced and also the area and area delay product will be reduced.
F. Petroni, L. Querzoni, R. Beraldi, M. Paolucci:
"LCBM: Statistics-Based Parallel Collaborative Filtering."
In: Proceedings of the 17th International Conference on Business Information Systems (BIS), 2014.
Abstract: "In the last ten years, recommendation systems evolved from novelties to powerful business tools, deeply changing the internet industry. Collaborative Filtering (CF) represents today’s a widely adopted strategy to build recommendation engines. The most advanced CF techniques (i.e. those based on matrix factorization) provide high quality results, but may incur prohibitive computational costs when applied to very large data sets. In this paper we present Linear Classifier of Beta distributions Means (LCBM), a novel collaborative filtering algorithm for binary ratings that is (i) inherently parallelizable and (ii) provides results whose quality is on-par with state-of-the-art solutions (iii) at a fraction of the computational cost."
Time Series Forecasting Using Recurrent Neural Network and Vector Autoregress...Databricks
Given the resurgence of neural network-based techniques in recent years, it is important for data science practitioner to understand how to apply these techniques and the tradeoffs between neural network-based and traditional statistical methods.
This lecture discusses two specific techniques: Vector Autoregressive (VAR) Models and Recurrent Neural Network (RNN). The former is one of the most important class of multivariate time series statistical models applied in finance while the latter is a neural network architecture that is suitable for time series forecasting. I’ll demonstrate how they are implemented in practice and compares their advantages and disadvantages. Real-world applications, demonstrated using python and Spark, are used to illustrate these techniques. While not the focus in this lecture, exploratory time series data analysis using time-series plot, plots of autocorrelation (i.e. correlogram), plots of partial autocorrelation, plots of cross-correlations, histogram, and kernel density plot, will also be included in the demo.
The attendees will learn – the formulation of a time series forecasting problem statement in context of VAR and RNN – the application of Recurrent Neural Network-based techniques in time series forecasting – the application of Vector Autoregressive Models in multivariate time series forecasting – the pros and cons of using VAR and RNN-based techniques in the context of financial time series forecasting – When to use VAR and when to use RNN-based techniques
Isobar Human Media - Social Media Marketing 2009Isobar
http://www.isobar.nl/humanmedia/
Isobar Human Media is de strategische advies partner bij uitstek voor het ontwikkelen en uitvoeren van Social Media strategiën.
Isobar Human Media is hét centrale punt waar alle specialistisch kennis op het gebied van Social Media uit het Isobar netwerk samenkomt.
Uma estrela brilhava bem acima dela, nas mais elevadas alturas. Esta estrela era de espécie espiritual e prometia um acontecimento grandioso, como os seres humanos ainda não haviam vivenciado.
Презентация Евгения Демура, Директора департамента бизнес-приложений DAN, на конференции РИФ 2016 Иннополис. Тема доклада- "Product Demand Management в международном рекламном холдинге"
Kas yra Etsy platforma. Kaip susikurti gerą parduotuvę. Dažniausiai daromos pradedančiųjų klaidos. Koks geriausias būdas “prasimušti” Etsy platformoje.
Aistės Kesminaitės Jankauskienės pranešimas elektroninės komercijos konferencijoje "E-komercija '16". Kovo 15 diena, 2016 metai.
Evaluation of a hybrid method for constructing multiple SVM kernelsinfopapers
Dana Simian, Florin Stoica, Evaluation of a hybrid method for constructing multiple SVM kernels, Recent Advances in Computers, Proceedings of the 13th WSEAS International Conference on Computers, Recent Advances in Computer Engineering Series, WSEAS Press, Rodos, Greece, July 23-25, 2009, ISSN: 1790-5109, ISBN: 978-960-474-099-4, pp. 619-623
The International Journal of Engineering and Science (The IJES)theijes
The International Journal of Engineering & Science is aimed at providing a platform for researchers, engineers, scientists, or educators to publish their original research results, to exchange new ideas, to disseminate information in innovative designs, engineering experiences and technological skills. It is also the Journal's objective to promote engineering and technology education. All papers submitted to the Journal will be blind peer-reviewed. Only original articles will be published.
Machine learning is the hacker art of describing the features of instances that we want to make predictions about, then fitting the data that describes those instances to a model form. Applied machine learning has come a long way from it's beginnings in academia, and with tools like Scikit-Learn, it's easier than ever to generate operational models for a wide variety of applications. Thanks to the ease and variety of the tools in Scikit-Learn, the primary job of the data scientist is model selection. Model selection involves performing feature engineering, hyperparameter tuning, and algorithm selection. These dimensions of machine learning often lead computer scientists towards automatic model selection via optimization (maximization) of a model's evaluation metric. However, the search space is large, and grid search approaches to machine learning can easily lead to failure and frustration. Human intuition is still essential to machine learning, and visual analysis in concert with automatic methods can allow data scientists to steer model selection towards better fitted models, faster. In this talk, we will discuss interactive visual methods for better understanding, steering, and tuning machine learning models.
Gait Based Person Recognition Using Partial Least Squares Selection Scheme ijcisjournal
The variations of viewing angle and intra-class of human beings have great impact on gait recognition
systems. This work represents an Arbitrary View Transformation Model (AVTM) for recognizing the gait.
Gait energy image (GEI) based gait authentication is effective approach to address the above problem, the
method establishes an AVTM based on principle component analysis (PCA). Feature selection (FS) is
performed using Partial least squares (PLS) method. The comparison of the AVTM PLS method with the
existing methods shows significant advantages in terms of observing angle variation, carrying and attire
changes. Experiments evaluated over CASIA gait database, shows that the proposed method improves the
accuracy of recognition compared to the other existing methods.
Comparative Analysis of Hand Gesture Recognition TechniquesIJERA Editor
During past few years, human hand gesture for interaction with computing devices has continues to be active area of research. In this paper survey of hand gesture recognition is provided. Hand Gesture Recognition is contained three stages: Pre-processing, Feature Extraction or matching and Classification or recognition. Each stage contains different methods and techniques. In this paper define small description of different methods used for hand gesture recognition in existing system with comparative analysis of all method with its benefits and drawbacks are provided.
PERFORMANCE EVALUATION OF FUZZY LOGIC AND BACK PROPAGATION NEURAL NETWORK FOR...ijesajournal
ABSTRACT
Fuzzy c-mean is one of the efficient tools used in character recognition. Back propagation neural network is another powerful that may be used in such field. A comparison between fuzzy c-mean and BP neural network classifiers are presented in this research to obtain the performance of both classifiers. The comparison was based on recognition efficiency; this efficiency was evaluated as the ratio of the number of assigned characters with unknown one to the number of character set related to that character. The fuzzy C-mean and BP neural network algorithms were tested on a set of hand written and machine printed dataset named Chars74K dataset using Matlab (2016 b) programming language and the result was that neural network classifier gave 82% recognition efficiency while fuzzy c –mean gave 78%. Neural network classifier is more superior than fuzzy C-mean in recognition due to the limitations of processing time of fuzzy C-mean that requires smaller image size and eventually this will cause less efficiency.
Performance Evaluation of Object Tracking Technique Based on Position VectorsCSCJournals
In this paper, a novel algorithm for moving object tracking based on position vectors has proposed. The position vector of an object in first frame of a video has been extracted based on selection of region of interest. Based on position vector in first frame object direction has shown in nine different directions. We extract nine position vectors for nine different directions. With these position vectors next frame is cropped into nine blocks. We exploit block matching of the first frame with nine blocks of the next frame in a simple feature space by Descrete wavelet transform and dual tree complex wavelet transform. The matched block is considered as tracked object and its position vector is a reference location for the next successive frame. We describe performance evaluation and algorithm in detail to perform simulation experiments of object tracking using different feature vectors which verifies the tracking algorithm efficiency.
Performance Evaluation of Object Tracking Technique Based on Position Vectors
EE660_Report_YaxinLiu_8448347171
1. EE-660 Machine Learning Final Project
Gesture Phase Segmentation based on Radial Basis Function(RBF) Kernel Support Vec-
tor Machine and K-Nearest Neighbor Algorithms
Yaxin Liu, yaxinliu@usc.edu
December 6, 2014
1. Abstract
People commonly perform one or several movements or “excursion” with their hands, arms or even their bodies
during a natural conversation or when giving a speech. An excursion, particularly regarding the hands, refers to a
movement from a rest position to some region in space, and then bringing them back to the same or another rest
position. While the hands are away from the rest position, the movement is called a gesture unit. In addition, a
gesture unit can be divided into four phases: preparation, stroke, hold and retraction.
This project aims at evaluating the performance of different classifiers, the one based on Support Vector
Machine(SVM) with RBF Kernel and the other based on K-Nearest Neighbor algorithm, that works on segmenting
the 5 gesture phases stated above. The result based on SVM shows an accuracy of 53.27% on the test set and
the KNN algorithm 92.53%.
2. Problem Statement
As it’s stated above, this problem can be treated as a multi-class classification problem which contains five class-
es that corresponds to the five gesture phases: 1-Rest, 2-Preparation, 3-Stroke, 4-Hold and 5-Retraction. Given a
dataset with the information of movements of three individuals and the labels to each captured frame of move-
ment, our goal is to correctly classify these five phases using the best performed model. To make the result as
accuracy as possible, there’s a lot of work to do with the feature selection and preprocessing procedure.
3. Project Formulation
3.1 Support Vector Machine
SVM is based on performing a nonlinear mapping on input vectors from their original feature space into a high-
dimensional feature space, and optimizing a hyperplane capable of separating data in this high-dimensional fea-
ture space[1]. We use toolbox LibSVM in MatLab to implement this algorithm. The following shows the functions I
used in this toolbox:
- svmtrain(labels, samples, parameter): used on the training set to get the cross validation or the model. The
third attribute in the function contains the parameters on building the model. Here we choose ‘-t 2’ for RBF
Kernel;
- svmpredict(labels, samples, model): apply the selected model on the testing set to get the accuracy of the
prediction.
3.2 K-Nearest Neighbor learning algorithm
In pattern recognition, the k-Nearest Neighbors algorithm (or k-NN for short) is a non-parametric method used for
classification and regression.[2] In both cases, the input consists of the k closest training examples in the feature
space. In k-NN classification, the output is a class membership. An object is classified by a majority vote of its
neighbors, with the object being assigned to the class most common among its k nearest neighbors (k is a posi-
tive integer, typically small). If k = 1, then the object is simply assigned to the class of that single nearest neighbor.
k-NN algorithm can be implemented by using the function fitcknn() for the model selection and predict() to get the
accuracy on the testing set.
1
2. EE-660 Machine Learning Final Project
4. Methodology
First of all we need to separate the whole dataset into two parts, the training set and the testing set. Training set is
the set where we can get our classification model and the cross validation accuracy, an important criterion for
model selection.
In this project we need to work on the raw data to do the classification. Regardless of the limitation on the raw
data, we first implement SVM on the original dataset to generate a baseline for the classification precision.
After that, we notice that the only feature we get in the raw dataset the the position of the hands, wrists, head,
spine and the timestamp on each frame of the videos. So it’s necessary to do feature extraction, which consists
of: a pre-processing phase that aims at making the representation invariant to the lateral displacement and dis-
tance of the user in relation to the camera; and a velocity and acceleration extraction phase.
To select features from the expanded feature vector, we implement PCA by using the toolbox PRTools in Mat-
Lab, by calling the function pcam(). Then plot the cross validation accuracy as a function of the number of fea-
tures that is selected, under the circumstances of both algorithms, SVM and k-NN. Choose the optimal models
from both system and test on the testing set to get the final result.
5. Implementation
5.1 Feature Space and Baseline test
There’re 7 subsets of the dataset provided on the website (http://archive.ics.uci.edu/ml/datasets/Gesture+Phase
+Segmentation#). The dataset is organized in 7 raw files, one for each video which compose the dataset. The
name of the file refers to each video: the letter corresponds to the user (A, B, C) and the number corresponds to
the story (1, 2, 3). Number of instances: A1-1747 frames, A2-1264 frames, A3-1834 frames, B1-1073 frames, B3-
1423 frames, C1-1111 frames, C3-1448 frames. Attributes information in raw files:
- 18 numeric attributes (double), a timestamp (integer) and a class attribute (nominal):
• 1~3: Position of left hand (x, y, z coordinates);
• 4~6: Position of right hand (x, y, z coordinates);
• 7~9: Position of head (x, y, z coordinates);
• 10~12: Position of spine (x, y, z coordinates);
• 13~15: Position of left wrist (x, y, z coordinates);
• 16~18: Position of right wrist (x, y, z coordinates);
• 19: Timestamp;
• 20: Phases: 1- Rest, 2- Preparation, 3- Stroke, 4- Hold, 5- Retraction (the labels are changed to be
integers);
Now we combine these 7 files to form one dataset and then randomly pick 1500 samples as the testing set Dtest
and the rest for the training set Dtrain.
We directly use Dtest which has the original feature space for the SVM training procedure, and the cross validation
accuracy results to be 29.84%.
5.2 Preprocessing and Feature Extraction
Due to the fact that each person may stand at a different position in front of the camera, we need to do some pre-
processing on the feature. This includes subtracting the hands and wrists coordinate from the spine coordinates in
each frame and getting it divided by the distance between the points of the head and spine.
2
3. EE-660 Machine Learning Final Project
p’hand = (phand - pspine)/dis(phead - pspine)
p’wrist = (pwrist - pspine)/dis(phead - pspine)
Then, normalize each 3-D position coordinates.
The second thing we did on the feature space is adding features of velocity and the acceleration of the hands and
wrists. For the velocity, the estimation is given by:
where t is the timestamp of frame i, d is the displacement in frames, and is the Euclidean distance between
normalized 3D position of the interest point at frame i and at frame i-d . Here we use a window with size 3, that is
d=2 in the equation above. The acceleration is estimated by:
The proposed approach considers a windowed strategy [3], using information from past and/or future frames to
represent each frame of interest due to the intrinsic temporal aspects of gesture phases segmentation problem. At
last, we added two more features which corresponds to the scalar of the velocity and the acceleration. Therefore,
we have 51 features in total:
• 19~21: Vectorial velocity of left hand (x, y, z coordinates);
• 22~24: Vectorial velocity of right hand (x, y, z coordinates);
• 25~27: Vectorial velocity of left wrist (x, y, z coordinates);
• 28~30: Vectorial velocity of right wrist (x, y, z coordinates);
• 31~33: Vectorial acceleration of left hand (x, y, z coordinates);
• 34~36: Vectorial acceleration of right hand (x, y, z coordinates);
• 37~39: Vectorial acceleration of left wrist (x, y, z coordinates);
• 40~42: Vectorial acceleration of right wrist (x, y, z coordinates);
• 43: Scalar velocity of left hand
• 44: Scalar velocity of right hand
• 45: Scalar velocity of left wrist
• 46: Scalar velocity of right wrist
• 47: Scalar acceleration of left hand
• 48: Scalar acceleration of right hand
• 49: Scalar acceleration of left wrist
• 50: Scalar acceleration of right wrist
• 51: Normalized timestamp
5.2 Training Process and Model Selection
After the preprocessing procedure, we have 8380 samples for training and 1500 samples for testing. To select the
appropriate features in the model, we implement PCA on both SVM and k-NN algorithm and generate the rela-
tionship between the cross validation accuracy and the dimensionality of the feature vector. When the optimal di-
mensionality is determined, we use the chosen feature vector on selecting the best parameters for the models in
each classifier.
- For SVM, the key parameter for the function svmtrain() is the cost value for the model. So we plot a diagram
of the changes in the cross validation accuracy with different cost.
3
vi,i−d =
Δri,i−d
ti − ti−d
Δri,i−d
ai,i−1 =
vi − vi−1
ti − ti−1
4. EE-660 Machine Learning Final Project
- For k-NN, the number of neighbored samples and the types of the distances between samples are key pa-
rameters in the model. So we plot a diagram of the performance of the model with different number of neigh-
bors.
We choose the model that has the highest cross validation accuracy in part 5.2 for each algorithms and use this
model on the testing set to get the final classification results.
5.3 Training Results
The graphs showed above shows that the optimal dimension of the SVM model happens at d=5, with an accuracy
of 53.77% and it goes up as the cost value increases. However, the cost value cannot be too large or it might
cause under fitting with too much regularization. The optimal SVM model should has a 5-dimensional feature
space and the cost of 101.
The two graphs above corresponds to the perform of the parameters of the model generated by k-NN algorithm.
The graph on the left shows that the model has the highest precision of 92.17%, with 1 neighbor, and the one on
the right shows that the optimal dimension appears at d=45 with an accuracy of 92.91%. However, it shows that
different types of the computation between samples makes no big difference in the performance of the classifier.
So we just use the Euclidian distance here in this model.
It turns out that k-NN algorithm won over the RBF Kernel SVM in this instance.
6. Final Results and Interpretation
4
5. EE-660 Machine Learning Final Project
Finally, we apply the two selected models from each classifier on the testing set. Etest with the RBF Kernel SVM
classifier is 46.73%, and that with the one generated by k-NN algorithm is 8.73%. Therefore, the classifier gener-
ated by the k-NN algorithm works better on this gesture phase segmentation problem.
7. Summary and Conclusions
From all the work that has been done so far, it seems that k-NN algorithm wins over the RBF Kernel SVM on this
specific problem. However, the toolbox, LibSVM, we used could has its limitations on selecting models in limited
parameters to choose. Also, the window size implemented while computing the velocity and the acceleration of
the hands and wrists can make difference in the result. There’re researches on the same problem that has a pre-
cision far better than 53.77% with an SVM model. According to the reference paper [3], the authors achieved a
precision of 84.3% in their approach working with SVM.
8. Reference
[1] V. Vapnik. The nature of statistical learning theory. Springer-Verlag, 1995.
[2] Wiki: http://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm
[3] Madeo, R. C. B. ; Lima, C. A. M. ; PERES, S. M. . Gesture Unit Segmentation using Support Vector Ma-
chines: Segmenting Gestures from Rest Positions. In: Symposium on Applied Computing (SAC), 2013,
Coimbra. Proceedings of the 28th Annual ACM Symposium on Applied Computing (SAC), 2013a. p. 46-52.
[4] Wagner, P. K. ; PERES, S. M. ; Madeo, R. C. B. ; Lima, C. A. M. ; Freitas, F. A. . Gesture Unit Segmentation
Using Spatial-Temporal Information and Machine Learning. In: 27th Florida Artificial Intelligence Research
Society Conference (FLAIRS), 2014, Pensacola Beach. Proceedings of the 27th Florida Artificial Intelligence
Research Society Conference (FLAIRS). Palo Alto : The AAAI Press, 2014. p. 101-106.
5
6. EE-660 Machine Learning Final Project
Appendix.
MatLab Code
% —————————————————————————————————————————————————
% - generate_dataset.m
% - separate the dataset into training set and testing set
% - both preprocessed and raw dataset are respectively created
% ————————————————————————————————————————————————-
clear all;
close all;
load dataset;
%----------- pre-processing on trainingset ------------------
d = 2;
k = 1;
[xa1_norm, ya1_pre] = prepocess( xa1, ya1, d, k );
[xa2_norm, ya2_pre] = prepocess( xa2, ya2, d, k );
[xb1_norm, yb1_pre] = prepocess( xb1, yb1, d, k );
[xb3_norm, yb3_pre] = prepocess( xb3, yb3, d, k );
[xa3_norm, ya3_pre] = prepocess( xa3, ya3, d, k );
[xc1_norm, yc1_pre] = prepocess( xc1, yc1, d, k );
[xc3_norm, yc3_pre] = prepocess( xc3, yc3, d, k );
x_data = [xa1_norm;xa2_norm;xa3_norm;xb1_norm;xb3_norm;xc1_norm;xc3_norm;];
x_raw = [xa1(d+k+1:length(xa1(:,1)),:);
xa2(d+k+1:length(xa2(:,1)),:);
xa3(d+k+1:length(xa3(:,1)),:);
xb1(d+k+1:length(xb1(:,1)),:);
xb3(d+k+1:length(xb3(:,1)),:);
xc1(d+k+1:length(xc1(:,1)),:);
xc3(d+k+1:length(xc3(:,1)),:)];
y_data = [ya1_pre;ya2_pre;ya3_pre;yb1_pre;yb3_pre;yc1_pre;yc3_pre];
y_raw = [ya1(d+k+1:length(xa1(:,1)),:);
ya2(d+k+1:length(xa2(:,1)),:);
ya3(d+k+1:length(xa3(:,1)),:);
yb1(d+k+1:length(xb1(:,1)),:);
yb3(d+k+1:length(xb3(:,1)),:);
yc1(d+k+1:length(xc1(:,1)),:);
yc3(d+k+1:length(xc3(:,1)),:)];
Num_test = 1500;
Num_data = length(x_data(:,1));
rand_idx = randperm(Num_data);
idx_test = sort(rand_idx(1,1:Num_test));
idx_train = sort(rand_idx(1,Num_test+1:Num_data));
x_train = x_data(idx_train',:);
y_train = y_data(idx_train',:);
x_test = x_data(idx_test',:);
y_test = y_data(idx_test',:);
xb_train = x_raw(idx_train',:);
yb_train = y_raw(idx_train',:);
save('rawdata.mat','xb_train','yb_train');
save(‘/Users/yaxinliu/Documents/MATLAB/EE660/project/
dataset_p.mat','x_train','y_train','x_test','y_test');
% —————————————————————————————————-
% - baseline.m
% - training on the raw data
% —————————————————————————————————-
clear all;
close all;
load rawdata;
addpath('libsvm-3.17/matlab');
6
7. EE-660 Machine Learning Final Project
%-------------- baseline test ----------------
accuracy = svmtrain(yb_train, xb_train, '-t 2 -v 5 -h 0 -q’);
% ——————————————————————————————————————————————————————————-
% - RBFKernelSVM.m
% - SVM classifier selection and test with the optimal model
% ——————————————————————————————————————————————————————————-
clear all;
close all;
load dataset_p;
addpath('libsvm-3.17/matlab');
addpath('prtools/prtools');
Fnum = length(x_train(1,:));
%------ select optimal parameters for the model -------------
ACC_PCA = zeros(1,Fnum);
for D = 1:2:Fnum
[W, FRAC] = pcam(x_train, D);
ACC_PCA(D) = svmtrain(y_train, x_train*W, '-t 2 -v 5 -h 0 -q -c 100');
end
[MaxVal_PCA, MaxIdx_PCA] = max(ACC_PCA(1:2:Fnum));
MaxIdx_PCA = (MaxIdx_PCA-1)*2 + 1;
%------ Plot -----------
figure;
plot([1:2:Fnum],ACC_PCA(1:2:Fnum),'.-');
title('RBF Kernel(SVM) Performance vs. Dimension');
ylabel('Accuracy');
xlabel('Dimension');
grid on;
ACC_C = zeros(1,6);
C = 1:20:101;
[W, FRAC] = pcam(x_train, MaxIdx_PCA);
for i = 1:6
parameter = sprintf('-t 2 -v 5 -h 0 -q -c %d', C(i));
ACC_C(i) = svmtrain(y_train, x_train*W,parameter);
end
[MaxVal_C, MaxIdx_C] = max(ACC_C);
MaxIdx_C = (MaxIdx_C-1)*20+1;
figure;
plot(C,ACC_C,'.-');
title('RBF Kernel(SVM) Performance vs. Cost (with Opt. Dim)');
ylabel('Accuracy');
xlabel('Cost');
grid on;
parameter = sprintf('-t 2 -h 0 -q -c %d ', MaxIdx_C);
SVMoptModel = svmtrain(y_train, x_train*W,parameter);
[predicted_label, accuracy, prob_estimates] = svmpredict(y_test, x_test*W, SVMoptModel, ‘-
q');
% ——————————————————————————————————————————————————————————————————————
% - KNN.m
% - classifier selection with KNN algo. and test with the optimal model
% ——————————————————————————————————————————————————————————————————————
clear all;
close all;
load dataset_p;
7
8. EE-660 Machine Learning Final Project
addpath('prtools/prtools');
%-------------- KNN classifier ------------------------------
KNNmodel=fitcknn(x_train,y_train);
%------ select optimal parameters for the model -------------
Acc = zeros(1,10);
for D = 1:10
KNNmodel.NumNeighbors = D;
cvmdl = crossval(KNNmodel,'kfold',5);
Acc(D) = 1-kfoldLoss(cvmdl);
end
%------ Plot -----------
figure;
plot([1:10],Acc,'.-');
title('k-NN Performance vs. Num of Neighbors');
ylabel('Accuracy');
xlabel('Dimension');
grid on;
[MaxVal_K, MaxIdx_K] = max(Acc);
KNNmodel.NumNeighbors = MaxIdx_K;
Fnum = length(x_train(1,:));
%------ select optimal parameters for the model -------------
ACC_PCA = zeros(1,Fnum);
for D = 1:Fnum
[W, FRAC] = pcam(x_train, D);
KNNmodel=fitcknn(x_train*W,y_train,'NumNeighbors',MaxIdx_K);
cvmdl = crossval(KNNmodel,'kfold',5);
ACC_PCA(D) = 1-kfoldLoss(cvmdl);
end
[MaxVal_PCA, MaxIdx_PCA] = max(ACC_PCA);
%------ Plot -----------
figure;
plot([1:3:Fnum],ACC_PCA(1:3:Fnum),'.-');
title('k-NN Performance vs. Dimension');
ylabel('Accuracy');
xlabel('Dimension');
grid on;
%------------- Compute error ---------------------------------
y_predict = predict(KNNmodel,x_test*W);
Error = y_predict-y_test;
ErrNum = length(find(Error~=0));
Acc_test = (length(Error)-ErrNum)/length(Error);
% —————————————————————————————————————————————————————————————-
% - normalize.m
% - the func. works on normalizing the 3-D position coordinates
% —————————————————————————————————————————————————————————————-
function Matrix = normalize( train )
for i = 1: length(train(:,1))
hs_dist(i) = normest(train(i,7:9)-train(i,10:12));
train(i,1:3) = (train(i,1:3) - train(i,7:9))/hs_dist(i);
train(i,4:6) = (train(i,1:3) - train(i,7:9))/hs_dist(i);
8
9. EE-660 Machine Learning Final Project
train(i,13:15) = (train(i,1:3) - train(i,7:9))/hs_dist(i);
train(i,16:18) = (train(i,1:3) - train(i,7:9))/hs_dist(i);
end
for i = 1:length(train(:,1))
for j = 1:3:length(train(1,:))-1
scale(i,j) = normest(train(i,j:(j+2)));
Matrix(i,j:(j+2)) = train(i,j:(j+2))/scale(i,j);
end
end
end
% —————————————————————————————————————————————————————————————-
% - preprocess.m
% - the func. works on feature extraction
% —————————————————————————————————————————————————————————————-
function [output_x, output_y] = prepocess( sample_x, sample_y, d, k )
norm_x = normalize(sample_x);
height = length(norm_x(:,1));
velocity = zeros(height,12);
accele = zeros(height,12);
scalar_v = zeros(height,4);
scalar_a = zeros(height,4);
for i = d+1:height
velocity(i,:) = (norm_x(i,[1:3,4:6,13:15,16:18])-norm_x(i-d,[1:3,4:6,13:15,16:18]))/(sam-
ple_x(i,19)-sample_x(i-d,19));
scalar_v(i,:) = [normest(velocity(i,1:3)),normest(velocity(i,4:6)),normest(velocity(i,
7:9)),normest(velocity(i,10:12))];
if(i>d+k)
accele(i,:) = (velocity(i,:)-velocity(i-k,:))/(sample_x(i,19)-sample_x(i-1,19));
scalar_a(i,:) = [normest(accele(i,1:3)),normest(accele(i,4:6)),normest(accele(i,
7:9)),normest(accele(i,10:12))];
end
end
time = (sample_x(:,19)-mean(sample_x(:,19)))/std(sample_x(:,19));
output_x = [norm_x,velocity,accele,scalar_v,scalar_a,time];
output_x = output_x(d+k+1:height,:);
% output_x = output_x(:,[1:6,13:51]);
output_y = sample_y(d+k+1:height,:);
end
9