Feature selection: A Data Perspective
•Download as PPTX, PDF•
0 likes•89 views
Παρουσίαση Δημοσίευσης: Li, J., Cheng, K., Wang, S., Morstatter, F., Trevino, R. P., Tang, J., & Liu, H. (2018). Feature Selection: A Data Perspective. ACM Computing Surveys, 50 (6), 1–45. doi:10.1145/3136625
Recommended
More Related Content
More from Gregory Davrazos
More from Gregory Davrazos (12)
Recently uploaded
Recently uploaded (20)
Feature selection: A Data Perspective
- 1. Feature Selection: A Data Perspective Παρουσίαση Δημοσίευσης Li, J., Cheng, K., Wang, S., Morstatter, F., Trevino, R. P., Tang, J., & Liu, H. (2018). Feature Selection: A Data Perspective. ACM Computing Surveys, 50 (6), 1–45. doi:10.1145/3136625 Γρηγόριος Δαβράζος
- 2. Περιεχόμενα Παρουσίασης Εισαγωγή στην Επιλογή Χαρακτηριστικών (Feature Selection) • Curse Of Dimensionality • Παραδοσιακή Κατηγοριοποίηση Αλγορίθμων Feature Selection • Κατηγοριοποίηση Αλγορίθμων Feature Selection από την Οπτική των Δεδομένων Feature Selection οn Conventional Data Feature Selection with Structured Features Feature Selection with Heterogeneous Data Feature Selection with Streaming Data Evaluation, Open Problems and Summary
- 3. Curse Of Dimensionality (1) Συνέπειες: • Αύξηση Αναγκών Αποθήκευσης & Χρόνου Επεξεργασίας • Yπερπροσαρμογή Μοντέλων (Overfitting) • Απαίτηση για Περισσότερα Δεδομένα DNA fMRI Texts https://www.visiondummy.com/2014/04/curse-dimensionality-affect- classification/
- 4. Curse Of Dimensionality (2) Cure to Feature Selection Feature Extraction Dimensionality Reduction Algorithms
- 5. Feature Selection Επιλογή «Βέλτιστου» υποσυνόλου χαρακτηριστικών (features) από το αρχικό σύνολο υψηλής διάστασης χρησιμοποιώντας συγκεκριμένα κριτήρια
- 6. Παραδοσιακή Κατηγοριοποίηση Αλγορίθμων Feature Selection Label Perspective Supervised Unsupervised Semi- Supervised Selection Strategy Perspective Wrapper methods Filter methods Embedded methods
- 7. Κατηγοριοποίηση Αλγορίθμων Feature Selection από την Οπτική των Δεδομένων
- 8. Περιεχόμενα Παρουσίασης Εισαγωγή στην Επιλογή Χαρακτηριστικών (Feature Selection) Feature Selection οn Conventional Data Feature Selection with Structured Features Feature Selection with Heterogeneous Data Feature Selection with Streaming Data Evaluation, Open Problems and Summary
- 9. Feature Selection on Conventional Data Conventional Feature Selection Similarity based methods Laplacian Score SPEC Fisher Score Trace Ratio Criterion ReliefF Information theoretical based methods Mutual Information Maximization (Information Gain) / Mutual Information Feature Selection Minimum Redundancy Maximum Relevance, / Conditional Infomax Feature Extraction Joint Mutual Information, / Conditional Mutual Information Maximization Informative Fragments, / Interaction Capping Double Input Symmetrical Relevance, / Fast Correlation-Based Filter Sparse learning-based methods Feature Selection with Lp-Norm Regularizer Feature Selection with Lp,q -Norm Regularizer Efficient and Robust Feature Selection Multi-Cluster Feature Selection l2,1-Norm Regularized Discriminative Feature Selection Feature Selection Using Nonnegative Spectral Analysis Statistical based methods Low Variance T-Score Chi-Square Score Gini Index CFS Other methods Hybrid Methods Deep Learning Based Methods Reconstruction based Methods
- 10. Similarity based methods Similarity based methods Laplacian Score SPEC Fisher Score Trace Ratio Criterion ReliefF Πλεονεκτήματα: • Εξαιρετική Απόδοση σε προβλήματα μάθησης Supervised & Unsupervised, • Ανεξάρτητες από τους Αλγόριθμους Μάθησης. Μειονεκτήματα: • Δεν μπορούν να διαχειριστούν την Feature Redundancy.
- 11. Information theoretical based methods Information theoretical based methods Mutual Information Maximization Mutual Information Feature Selection Minimum Redundancy Maximum Relevance Conditional Infomax Feature Extraction Joint Mutual Information Conditional Mutual Information Maximization Informative Fragments Interaction Capping Double Input Symmetrical Relevance Fast Correlation-Based Filter Linear Combination of Shannon terms NonLinear Combination of Shannon terms Πλεονεκτήματα: • Ενοποιούν με πιθανοτικό τρόπο και τα δυο στοιχεία “feature relevance” και “feature redundancy.” • Δεν εξαρτώνται από τους Αλγόριθμους Μάθησης Μειονεκτήματα: • Αφορούν μόνο προβλήματα Supervised μάθησης. • Διαχειρίζονται μόνο Διακριτά Δεδομένα
- 12. Sparse learning-based methods Sparse learning- based methods Feature Selection with lp-Norm Regularizer Feature Selection with lp,q -Norm Regularizer Efficient and Robust Feature Selection Multi-Cluster Feature Selection l2,1-Norm Regularized Discriminative Feature Selection Feature Selection Using Nonnegative Spectral Analysis Πλεονεκτήματα: • Πολύ καλή Απόδοση σε Συγκεκριμένους Αλγόριθμους Μάθησης • Καλή Επεξηγηματικότητα του Μοντέλου Μειονεκτήματα: • Εξαρτώνται από τους Αλγόριθμους Μάθησης • Απαιτούν την Λύση Προβλήματος Βελτιστοποίησης Non-smooth με αυξημένο Υπολογιστικό Κόστος
- 13. Statistical based methods Statistical based methods Low Variance T-Score Chi-Square Score Gini Index CFS Low Variance eliminates features whose variance are below a predefined threshold. Αφορά προβλήματα binary classification. Σε αντίθεση με τις προηγούμενες μεθόδους ο συγκεκριμένος δείκτης απαιτείται να είναι πιο μικρός Πλεονεκτήματα: • Απλές Μέθοδοι • Μικρό Υπολογιστικό Κόστος • Χρησιμοποιούνται σε στάδιο Preprocessing πριν από πιο σοφιστικέ αλγορίθμους Μειονεκτήματα: • Δεν μπορούν να διαχειριστούν την Feature Redundancy. • Διαχειρίζονται μόνο Διακριτά Δεδομένα
- 14. Other methods Other methods Hybrid Methods Deep Learning Based Methods Reconstruction based Methods construct a group of feature subsets from different feature selection algorithms and then produce an aggregated result out of the group Using deep neural networks to learn new feature representations feature relevance = capability of features to approximate the original data via a reconstruction function
- 15. Περιεχόμενα Παρουσίασης Εισαγωγή στην Επιλογή Χαρακτηριστικών (Feature Selection) Feature Selection οn Conventional Data Feature Selection with Structured Features Feature Selection with Heterogeneous Data Feature Selection with Streaming Data Evaluation, Open Problems and Summary
- 16. Feature Selection with Structured Features Feature Selection with Structured Features Feature Selection with Group Feature Structures Group Lasso Sparse Group Lasso Overlapping Sparse Group Lasso Feature Selection with Tree Feature Structures Tree-Guided Group Lasso Feature Selection with Graph Feature Structures Graph Lasso GFLasso GOSCAR
- 17. Feature Selection with Group Feature Structures Feature Selection with Group Feature Structures Group Lasso Sparse Group Lasso Overlapping Sparse Group Lasso Διαφορές
- 18. Feature Selection with Tree Feature Structures Feature Selection with Tree Feature Structures Tree-Guided Group Lasso Τhe structure over the features can be represented as a tree with leaf nodes as features.
- 19. Feature Selection with Graph Feature Structures Feature Selection with Graph Feature Structures Graph Lasso GFLasso GOSCAR Μειονεκτήματα: Απαιτούν την λύση πολύπλοκων αλγορίθμων βελτιστοποίησης με υψηλό υπολογιστικό κόστος Η δομή των feature δίνεται και δεν συνεπάγεται από τα διαθέσιμα δεδομένα
- 20. Περιεχόμενα Παρουσίασης Εισαγωγή στην Επιλογή Χαρακτηριστικών (Feature Selection) Feature Selection οn Conventional Data Feature Selection with Structured Features Feature Selection with Heterogeneous Data Feature Selection with Streaming Data Evaluation, Open Problems and Summary
- 21. Feature Selection with Heterogeneous Data Feature Selection with Heterogeneous Data Feature Selection Algorithms with Linked Data Feature Selection on Networks Feature Selection for Social Media Data (LinkedFS) Unsupervised Feature Selection for Linked Data Robust Unsupervised Feature Selection for Networked Data Multi-Source Feature Selection Multi-Source Feature Selection via Geometry- Dependent Covariance Analysis (GDCOV) Feature Selection Algorithms with Multi-View Data Adaptive Multi-View Feature Selection Unsupervised Feature Selection for Multi-View Data Multi-View Clustering and Feature Learning via Structured Sparsity
- 22. Feature Selection Algorithms with Linked Data Feature Selection Algorithms with Linked Data Feature Selection on Networks Feature Selection for Social Media Data (LinkedFS) Unsupervised Feature Selection for Linked Data Robust Unsupervised Feature Selection for Networked Data
- 23. Multi-View Data / Multi-Source Feature Selection
- 24. Feature Selection Algorithms with Multi-Source / Multi- View Data Multi-Source Feature Selection Multi-Source Feature Selection via Geometry-Dependent Covariance Analysis (GDCOV) GPCOVspca Feature Selection Algorithms with Multi-View Data Adaptive Multi-View Feature Selection Unsupervised Feature Selection for Multi-View Data Multi-View Clustering and Feature Learning via Structured Sparsity Πλεονεκτήματα Διαχείριση διαφορετικών τύπων δεδομένων ταυτόχρονα. Μειονεκτήματα: Επίλυση πολύπλοκου υπολογιστικά προβλήματος βελτιστοποίησης Ανοικτό Θέμα Έρευνας η Εύρεση Αποδοτικών Αλγορίθμων
- 25. Περιεχόμενα Παρουσίασης Εισαγωγή στην Επιλογή Χαρακτηριστικών (Feature Selection) Feature Selection οn Conventional Data Feature Selection with Structured Features Feature Selection with Heterogeneous Data Feature Selection with Streaming Data Evaluation, Open Problems and Summary
- 26. Feature Selection with Streaming Data Feature Selection with Streaming Data Feature Selection Algorithms with Feature Streams Grafting Alpha-Investing Algorithm Online Streaming Feature Selection Algorithm Unsupervised Streaming Feature Selection in Social Media Feature Selection Algorithms with Data Streams Online Feature Selection Unsupervised Feature Selection on Data Streams
- 27. Feature Selection Algorithms with Feature Streams Feature Selection Algorithms with Feature Streams Grafting Alpha-Investing Algorithm Online Streaming Feature Selection Algorithm Unsupervised Streaming Feature Selection in Social Media
- 28. Feature Selection Algorithms with Data Streams Feature Selection Algorithms with Data Streams Online Feature Selection Unsupervised Feature Selection on Data Streams
- 29. Περιεχόμενα Παρουσίασης Εισαγωγή στην Επιλογή Χαρακτηριστικών (Feature Selection) Feature Selection οn Conventional Data Feature Selection with Structured Features Feature Selection with Heterogeneous Data Feature Selection with Streaming Data Evaluation, Open Problems and Summary
- 30. Web Site http://featureselection.asu.edu/ Interactive Graphical User Interface for Feature Selection Feature Selection Repository Scikit-feature in Python 40 Popular Feature Selection Algorithms Για κάθε dataset, αξιολογούν όλους τους εφαρμόσιμους αλγόριθμους σε αυτό
- 31. Evaluation Methods & Metrics Supervised Methods Διαχωρισμός dataset σε ΔΥΟ μέρη: • training set • test set Οι Αλγόριθμοι Feature selection εφαρμόζονται στο training set και μετά ελέγχονται στο test set. Χρησιμοποιείται η τεχνική classification accuracy και τα μοντέλα Linear SVM, Decision Tree, and Naïve Bayes. Για μεγαλύτερη αξιοπιστία χρησιμοποιούν 10-fold cross-validation Unsupervised Methods Χρησιμοποιούνται τα εξής clustering performance metrics • Normalized Mutual Information (NMI) • Accuracy (ACC) Εφαρμόζεται ο αλγόριθμος feature selection algorithm και στη συνέχεια η τεχνική k-means clustering η οποία επαναλαμβάνεται 20 φορές
- 32. ΛίσταΑλγορίθμων
- 33. Open Problems Scalability Εφαρμογή των Αλγορίθμων σε μεγάλης κλίμακας δεδομένα Stability Ευαισθησία του Αλγορίθμους σε διαταραχή στα training data Model Selection Επιλογή του Βέλτιστου αριθμού features. Οι λύσεις μέχρι τώρα είναι Ευρετικές Στην Unsupervised περίπτωση είναι ακόμα πιο δύσκολα. Ανάγκη Για Καλύτερους Αλγορίθμους
- 35. Συμπεράσματα Η επιλογή Features είναι: • αποτελεσματική για data preprocessing και για την μείωση της dimensionality των data. • βασική για εφαρμογές data-mining και machine-learning • προάγει την δημιουργία πιο απλών και κατανοητών μοντέλων Δημιουργία Πολλών Αλγορίθμων Επιλογής Features Συγκεκριμένο Survey Paper
- 36. Links & Video • Recent Advances in Feature Selection: A Data Perspective • Recent Advances in Feature Selection: A Data Perspective part 1 (Video) • Recent Advances in Feature Selection: A Data Perspective part 2 • Recent Advances in Feature Selection: A Data Perspective part 3 • https://machinelearningmastery.com/feature- selection-with-real-and-categorical-data/
Editor's Notes
- Dimensionality reduction is one of the most powerful tools to address the previously described issues. It can be mainly categorized into two main components: feature extraction and feature selection. Feature extraction projects the original high-dimensional features to a new feature space with low dimensionality. The newly constructed feature space is usually a linear or nonlinear combination of the original features. Feature selection, on the other hand, directly selects a subset of relevant features for model construction. Πότε επιλέγουμε την μια τεχνική και πότε την άλλη; On one hand, for many applications where the raw input data do not contain any features understandable to a given learning algorithm, feature extraction is preferred. On the other hand, as feature extraction creates a set of new features, further analysis is problematic as we cannot retain the physical meanings of these features. In contrast, by keeping some of the original features, feature selection maintains physical meanings of the original features and gives models better readability and interpretability. Therefore, feature selection is often preferred in many applications such as text mining and genetic analysis. It should be noted that in some cases even though feature dimensionality is often not that high, feature extraction/selection still plays an essential role such as improving learning performance, preventing overfitting, and reducing computational costs.
- Supervised feature selection is generally designed for classification or regression problems. Unsupervised feature selection is generally designed for clustering problems. in many real-world applications, we usually have a limited number of labeled data. Therefore, it is desirable to develop semi-supervised methods by exploiting both labeled and unlabeled data samples
- Τα χαρακτηριστικά των big data όπως velocity, variety απαιτούν την ανάπτυξη νέων αλγορίθμων Feature Selection. Ιδιως σε εφαρμογές που έχουμε streaming or heterogeneous data
- Από την παραπάνω κατηγοριοποίηση εξαιρούνται οι wrapper methods
- Similarity based methods assessing feature importance by their ability to preserve data similarity. We refer to them as similarity-based methods. For supervised feature selection, data similarity can be derived from label information; while for unsupervised feature selection methods, most methods take advantage of different distance metric measures to obtain data similarity.
- information-theoretic criteria are proposed to maximize feature relevance and minimize feature redundancy. Since the relevance of a feature is usually measured by its correlation with class labels, most algorithms in this family are performed in a supervised way. Most information-theoretic concepts can only be applied to discrete variables. Therefore, feature selection algorithms in this family can only work with discrete data. For continuous feature values, some data discretization techniques are required beforehand. Αντίθετα με τους αλγορίθμους της προηγούμενης κατηγορίας που δεν μπορούν να διαχειριστούν την feature redundancy, οι συγκεκριμένοι ενοποιούν με πιθανοτικό τρόπο και τα δυο στοιχεία “feature relevance” and “feature redundancy.” Δεν εξαρτώνται από τους αλγόριθμους μάθησης Αφορούν μόνο προβλήματα Supervised μάθησης. Διαχειρίζονται μόνο Διακριτά Δεδομένα
- The third type of methods is sparse-learning-based methods that aim to minimize the fitting errors along with some sparse regularization terms. Sparse-learning-based methods have received considerable attention in recent years due to their good performance and interpretability
- As they rely on various statistical measures instead of learning algorithms to assess feature relevance, most of them are filter-based methods. In addition, most statistical-based algorithms analyze features individually. Hence, feature redundancy is inevitably ignored during the selection phase. Low Variance eliminates features whose variance are below a predefined threshold. For example, for the features that have the same values for all instances, the variance is 0 and should be removed, since it cannot help discriminate instances from different classes.
- Existing feature selection methods for conventional data are based on a strong assumption that features are independent of each other (flat) while ignoring the inherent feature structures. However, in many real applications features could exhibit various kinds of structures, for example, spatial or temporal smoothness, disjoint groups, overlap groups, trees and graphs. Ιf this is the case, then feature selection algorithms incorporating knowledge about the structure information may help find more relevant features and therefore can improve subsequent learning tasks. One motivating example is from bioinformatics, in the study of array CGH, features have some natural spatial order, incorporating such spatial structure can help select more important features and achieve more accurate classification accuracy. Therefore, in this section, we discuss some representative feature selection algorithms that explicitly consider feature structures. Specifically, we will focus on group structure, tree structure, and graph structure.
- Παραδείγματα: MANOVA, Different frequency bands represented as groups in signal processing and genes with similar functionalities acting as groups in bioinformatics. Lasso completely ignores the group structures among features, and the selected features are from four different groups. On the contrary, Group Lasso tends to select or not select features from different groups as a whole. As shown in the figure, Group Lasso only selects the second and he fourth groups G2 and G4, and features in the other two groups G1 and G3 are not selected. Once Group Lasso selects a group, all the features in the selected group will be kept. However, in many cases, not all features in the selected group could be useful, and it is desirable to consider the intrinsic feature structures and select features from different selected groups simultaneously. Sparse Group Lasso takes advantage of both Lasso and Group Lasso, and it produces a solution with simultaneous intra-group and inter-group sparsity.
- In addition to the group structures, features can also exhibit tree structures. For example, in face recognition, different pixels can be represented as a tree, where the root node indicates the whole face, its child nodes can be different organs, and each specific pixel is considered as a leaf node. Another motivating example is that genes/proteins may form certain hierarchical tree structures
- Ιn many cases, features may have strong pairwise interactions. For example, in natural language processing, if we take each word as a feature, then we have synonyms and antonyms relationships between different words. Moreover, many biological studies show that there exist strong pairwise dependencies between genes. Since features show certain kinds of dependencies in these cases, we can model them by an undirected graph, where nodes represent features and edges among nodes show the pairwise dependencies between features Graph Lasso. Since features exhibit graph structures, when two nodes (features) Ni and Nj are connected by an edge inG(N, E), the features fi and fj are more likely to be selected together, and they should have similar feature coefficients. Graph Lasso encourages features connected together have similar feature coefficients. However, features can also be negatively correlated. In this case, the feature graph G(N, E) is represented by a signed graph, with both positive and negative edges. GFLasso model both positive and negative feature correlations. A major limitation of GFLasso is that it uses pairwise sample correlations to measure feature dependencies, which may lead to additional estimation bias. The feature dependencies cannot be correctly estimated when the sample size is small. To address the limitations of GFLasso, Yang et al. (2012) propose GOSCAR by putting a loo-norm regularization to enforce pairwise feature coefficients to be equivalent if two features are connected in the feature graph.
- heterogeneous data from different sources is becoming more and more prevalent in the era of big data. For example, in the medical domain, genes are often associated with different types of clinical features. Since data of each source can be noisy, partial, or redundant, how to find relevant sources and how to fuse them together for effective feature selection is a challenging problem. Another example is in social media platforms, instances of high dimensionality are often linked together, finding a way to integrate link information to guide feature selection is another difficult problem.
- Linked data are ubiquitous in real-world applications such as Twitter (tweets linked by hyperlinks), Facebook (users connected by friendships), and biological systems (protein interactions). Figure illustrates an example of linked data and its representation. Figure (a) shows eight linked instances, and the feature information is illustrated in the left part of Figure (b). Linked data provides an extra source of information, which can be represented by an adjacency matrix, illustrated in the right part of Figure (b).
- ulti-source and multi-view feature selection are different in two ways: First, multi-source feature selection aims to select features from the original feature space by integrating multiple sources while multi-view feature selection selects features from different feature spaces for all views simultaneously. Second, multi-source feature selection normally ignores the correlations among sources while multi-view feature selection exploits relations among features from different sources. Multi-View data represent different facets of data instances in different feature spaces. These feature spaces are naturally dependent and high-dimensional. Since multi-view feature selection is designed to select features across multiple views by using their relations, they are naturally different from multi-source feature selection.
- For supervised multi-view feature selection, the most common approach is Sparse Group Lasso. Εδώ περιλαμβάνονται representative algorithms for unsupervised multi-view feature selection.
- Previous methods assume that all data instances and features are known in advance. However, it is not the case in many real-world applications that we are more likely faced with data streams and feature streams. In the worst cases, the size of data or features are unknown or even infinite. Thus it is not practical to wait until all data instances or features are available to perform feature selection. For streaming data, one motivating example online spam email detection problem, where new emails are continuously arriving; it is not easy to employ batch-mode feature selection methods to select relevant features in a timely manner. the task is to timely select a subset of relevant features from all features seen so far. Τhis family employ various strategies to speed up the selection process such that it can deal with new data samples or new features on the arrival. Μost of these algorithms require multiple pass of the data and some even need to store all the historically generated data, which jeopardizes the usage of these algorithms when we only have limited memory or disk storage. It requires further efforts to design streaming algorithms that are effective and efficient with limited storage costs.
- Περιλαμβάνει 40 Popular Feature Selection Algorithms. It is built on one widely used Machine-learning Package Scikit-learn and Numpy and Scipy. The source code of this repository is available at Github .
- Σε αυτό τον πίνακα φαίνονται για κάθε αλγόριθμο (1) the type of data: conventional data or other types of data; (2) usage of labels: supervised or unsupervised (3) output: feature weighting or subset selection; (4) feature type: numerical variables or discrete variables (numerical variables can also be divided into continuous variables and discrete variables).
- Scalability Εφαρμογή των Αλγορίθμων σε μεγάλης κλίμακας δεδομένα Most existing feature selection algorithms have a time complexity proportional n^2 /n^3. For many feature selection algorithms, efficiency deteriorates quickly as could be very large. Well-designed feature selection algorithms work in linear or sub-linear time are preferred Stability Ευαισθησία του Αλγορίθμους σε διαταραχή στα training data. Stability of feature selection helps domain experts be more confident with the selected features. Biologists would like to see the same set of genes selected each time when they obtain new data; otherwise they will not trust the algorithm Many feature selection algorithms suffer from low stability with small perturbation! Model Selection We usually need to specify the number of selected features in feature weighting methods.\ Finding the “optimal” number is difficult A large number will increase the risk in including, irrelevant and redundant features, jeopardizing learning performance. A small number will miss some relevant features In unsupervised feature selection, we often need to specify the number of cluster or pseudo class labels. However, we often have limited knowledge about the intrinsic cluster structure of data. Different cluster number may lead to different cluster structures. May merge smaller clusters into a big cluster, May split one big cluster into multiple small clusters. Lead to different feature selection results. Without label information, we cannot perform cross validation
- No free Lunch Theorem -> no universal algorithm for feature selection
- We provide a structured overview of feature selection from a data perspective – Feature selection for conventional data (four main categories) – Feature selection with structured features – Feature selection with heterogeneous data – Feature selection with streaming data Feature selection is effective to tackle the curse of dimensionality and is essential to many data mining and machine learning problems • The objectives of feature selection include – Building simpler and more comprehensive models – Improving learning performance – Preparing clean and understandable data • Feature selection is equally important in the age of deep learning and big data