Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DMTM Lecture 20 Data preparation

734 views

Published on

Slides for the 2016/2017 edition of the Data Mining and Text Mining Course at the Politecnico di Milano. The course is also part of the joint program with the University of Illinois at Chicago.

Published in: Education
  • Be the first to comment

  • Be the first to like this

DMTM Lecture 20 Data preparation

  1. 1. Prof. Pier Luca Lanzi Data Preparation Data Mining andText Mining (UIC 583 @ Politecnico di Milano)
  2. 2. Prof. Pier Luca Lanzi Why Data Preprocessing? • Data in the real world are “dirty” • They are usually incomplete §Missing attribute values §Missing attributes of interest, or containing only aggregate data • They are usually noisy, containing errors or outliers §e.g., Salary=“-10” • They are usually inconsistent and contain discrepancies §e.g., Age=“42” Birthday=“03/07/1997” §e.g., Was rating “1,2,3”, now rating “A, B, C” §e.g., discrepancy between duplicate records 2
  3. 3. Prof. Pier Luca Lanzi Why Is Data Dirty? • Incomplete data may come from §“Not applicable” data value when collected §Different considerations between the time when the data was collected and when it is analyzed. §Human/hardware/software problems • Noisy data (incorrect values) may come from §Faulty data collection instruments §Human or computer error at data entry §Errors in data transmission • Inconsistent data may come from §Different data sources §Functional dependency violation (e.g., modify some linked data) • Duplicate records also need data cleaning 3
  4. 4. Prof. Pier Luca Lanzi Why Is Data Preprocessing Important? • No quality in data, no quality in the mining results! (trash in, trash out) • Quality decisions needs quality in data • E.g., duplicate or missing data may cause incorrect or even misleading statistics. • Data warehouses need consistent integration of quality data • Data extraction, cleaning, and transformation comprises the majority of the work of building a data warehouse 4
  5. 5. Prof. Pier Luca Lanzi Major Tasks in Data Preprocessing • Data cleaning §Fill in missing values, smooth noisy data, identify or remove outliers, and resolve inconsistencies • Data integration §Integration of multiple databases, data cubes, or files • Data reduction §Dimensionality reduction §Numerosity reduction §Data compression • Data transformation and data discretization §Normalization §Concept hierarchy generation 5
  6. 6. Prof. Pier Luca Lanzi Outliers • Outliers are data objects with characteristics that are considerably different than most of the other data objects in the data set 6
  7. 7. Prof. Pier Luca Lanzi Missing Values • Reasons for missing values §Information is not collected (e.g., people decline to give their age and weight) §Attributes may not be applicable to all cases (e.g., annual income is not applicable to children) • Handling missing values §Eliminate Data Objects §Estimate Missing Values §Ignore the Missing Value During Analysis §Replace with all possible values (weighted by their probabilities) 7
  8. 8. Prof. Pier Luca Lanzi Duplicate Data • Data set may include data objects that are duplicates, or almost duplicates of one another • Major issue when merging data from heterogeous sources • Examples: same person with multiple email addresses • Data cleaning: process of dealing with duplicate data issues 8
  9. 9. Prof. Pier Luca Lanzi Data Preprocessing • Aggregation • Sampling • Dimensionality Reduction • Feature subset selection • Feature creation • Discretization and Binarization 9
  10. 10. Prof. Pier Luca Lanzi Aggregation • Combining two or more attributes (or objects) into a single attribute (or object) • Data reduction §Reduce the number of attributes or objects • Change of scale §Cities aggregated into regions §States, countries, etc • More “stable” data §Aggregated data tends to have less variability 10
  11. 11. Prof. Pier Luca Lanzi Sampling • Sampling is the main technique employed for data selection • It is often used for both the preliminary investigation of the data and the final data analysis. • Statisticians sample because obtaining the entire set of data of interest is too expensive or time consuming • Sampling is used in data mining because processing the entire set of data of interest is too expensive or time consuming • Using a sample will work almost as well as using the entire data sets, if the sample is representative • A sample is representative if it has approximately the same property (of interest) as the original set of data 11
  12. 12. Prof. Pier Luca Lanzi Types of Sampling • Simple Random Sampling §There is an equal probability of selecting any particular item • Sampling without replacement §As each item is selected, it is removed from the population • Sampling with replacement §Objects are not removed from the population as they are selected for the sample. §In sampling with replacement, the same object can be picked up more than once • Stratified sampling §Split the data into several partitions §Then draw random samples from each partition 12
  13. 13. Prof. Pier Luca Lanzi 2000 Points 8000 points 500 Points
  14. 14. Prof. Pier Luca Lanzi Sample Size • What sample size is necessary to get at least one object from each of 10 groups. 14
  15. 15. Prof. Pier Luca Lanzi Dimensionality Reduction • Purpose §Avoid curse of dimensionality §Reduce amount of time and memory required by data mining algorithms §Allow data to be more easily visualized §May help to eliminate irrelevant features or reduce noise • Techniques §Principle Component Analysis §Singular Value Decomposition §Others: supervised and non-linear techniques 15
  16. 16. Prof. Pier Luca Lanzi Dimensionality Reduction: Principal Component Analysis (PCA) • Given N data vectors from n-dimensions, find k ≤ n orthogonal vectors (principal components) that can be best used to represent data • Works for numeric data only • Used when the number of dimensions is large 16
  17. 17. Prof. Pier Luca Lanzi Dimensionality Reduction: Principal Component Analysis (PCA) • Normalize input data • Compute k orthonormal (unit) vectors, i.e., principal components • Each input data (vector) is a linear combination of the k principal component vectors • The principal components are sorted in order of decreasing “significance” or strength • Since the components are sorted, the size of the data can be reduced by eliminating the weak components, i.e., those with low variance. (i.e., using the strongest principal components, it is possible to reconstruct a good approximation of the original data 17
  18. 18. Prof. Pier Luca Lanzi Dimensionality Reduction: Principal Component Analysis (PCA) • Goal is to find a projection that captures the largest amount of variation in data 18 Original axes ** * * * * * * * * * * * * * * * * * * * * * * Data points First principal componentSecond principal component
  19. 19. Prof. Pier Luca Lanzi
  20. 20. Prof. Pier Luca Lanzi
  21. 21. Prof. Pier Luca Lanzi Singular Value Decomposition • Theorem (Press 1992). It is always possible to decompose matrix A into A = UΛVT where U, Λ, V are unique • U, V are column orthonormal, i.e., columns are unit vectors, orthogonal to each other, so that UTU = I; VTV = I • Li,i are the singular values, they are positive, and sorted in decreasing order 21
  22. 22. Prof. Pier Luca Lanzi Singular Value Decomposition • Data are viewed as a matrix A with n examples described by m numerical attributes A[n x m] = U[n x r] Λ [ r x r] (V[m x r])T • A stores the original data • U represents the n examples using r new concepts/attributes • Λ represents the strength of each ‘concept’ (r is the rank of A) • V contain m terms, r concepts • Λi,i are called the singular values of A, if A is singular, some of the wi will be 0; in general rank(A) = number of nonzero wi • SVD is mostly unique (up to permutation of singular values, or if some Li are equal) 22
  23. 23. Prof. Pier Luca Lanzi Singular Value Decomposition for Dimensionality Reduction • Suppose you want to find best rank-k approximation to A • Set all but the largest k singular values to zero • Can form compact representation by eliminating columns of U and V corresponding to zeroed Li 23 1 1 1 0 0 2 2 2 0 0 1 1 1 0 0 5 5 5 0 0 0 0 0 2 2 0 0 0 3 3 0 0 0 1 1 0.18 0 0.36 0 0.18 0 0.90 0 0 0.53 0 0.80 0 0.27 = 9.64 0 0 5.29 x 0.58 0.58 0.58 0 0 0 0 0 0.71 0.71 x
  24. 24. Prof. Pier Luca Lanzi Singular Value Decomposition for Dimensionality Reduction 1 1 1 0 0 2 2 2 0 0 1 1 1 0 0 5 5 5 0 0 0 0 0 2 2 0 0 0 3 3 0 0 0 1 1 0.18 0.36 0.18 0.90 0 0 0 ~ 9.64 x 0.58 0.58 0.58 0 0 x 24
  25. 25. Prof. Pier Luca Lanzi Singular Value Decomposition for Dimensionality Reduction 1 1 1 0 0 2 2 2 0 0 1 1 1 0 0 5 5 5 0 0 0 0 0 2 2 0 0 0 3 3 0 0 0 1 1 ~ 1 1 1 0 0 2 2 2 0 0 1 1 1 0 0 5 5 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 25
  26. 26. Prof. Pier Luca Lanzi Feature Subset Selection • Another way to reduce dimensionality of data • Redundant features §duplicate much or all of the information contained in one or more other attributes §Example: purchase price of a product and the amount of sales tax paid • Irrelevant features §contain no information that is useful for the data mining task at hand §Example: students' ID is often irrelevant to the task of predicting students' GPA 26
  27. 27. Prof. Pier Luca Lanzi Feature Subset Selection • Brute-force approach §Try all possible feature subsets as input to data mining algorithm • Embedded approaches §Feature selection occurs naturally as part of the data mining algorithm • Filter approaches §Features are selected using a procedure that is independent from a specific data mining algorithm §E.g., attributes are selected based on correlation measures • Wrapper approaches §Use a data mining algorithm as a black box to find best subset of attributes §E.g., apply a genetic algorithm and an algorithm for decision tree to find the best set of features for a decision tree 27
  28. 28. Prof. Pier Luca Lanzi Example of using the information gain to rank the importance of attributes.
  29. 29. Prof. Pier Luca Lanzi Decision trees applied to all the attributes.
  30. 30. Prof. Pier Luca Lanzi Decision trees applied to all the breast dataset without the four least ranked attributes.
  31. 31. Prof. Pier Luca Lanzi Wrapper approach using exhaustive search and decision trees.
  32. 32. Prof. Pier Luca Lanzi Feature Creation • Create new attributes that can capture the important information in a data set much more efficiently than the original attributes • E.g., given the birthday, create the attribute age • Three general methodologies: §Feature Extraction: domain-specific §Mapping Data to New Space §Feature Construction: combining features 32
  33. 33. Prof. Pier Luca Lanzi Discretization • Three types of attributes §Nominal: values from an unordered set, e.g., color §Ordinal: values from an ordered set, e.g., military or academic rank §Continuous: real numbers, e.g., integer or real numbers • Discretization §Divide the range of a continuous attribute into intervals §Some classification algorithms only accept categorical attributes §Reduce data size by discretization §Prepare for further analysis 33
  34. 34. Prof. Pier Luca Lanzi Discretization Approaches • Supervised §Attributes are discretized using the class information §Generates intervals that tries to minimize the loss of information about the class • Unsupervised §Attributes are discretized solely based on their values 34
  35. 35. Prof. Pier Luca Lanzi Supervised Discretization • Entropy based approach 35 3 categories for both x and y 5 categories for both x and y
  36. 36. Prof. Pier Luca Lanzi Discretization Without Using Class Labels 36 Data Equal interval width Equal frequency K-means
  37. 37. Prof. Pier Luca Lanzi Segmentation by Natural Partitioning • A simply 3-4-5 rule can be used to segment numeric data into relatively uniform, “natural” intervals. • If an interval covers 3, 6, 7 or 9 distinct values at the most significant digit, partition the range into 3 equi-width intervals (3 equal-width intervals for 3, 6, and 9; and 3 intervals in the grouping of 2-3-2 for 7) • If it covers 2, 4, or 8 distinct values at the most significant digit, partition the range into 4 intervals • If it covers 5 or 10 distinct values at the most significant digit, partition the range into 5 intervals 37
  38. 38. Prof. Pier Luca Lanzi Summary • Data exploration and preparation, or preprocessing, is a big issue for both data warehousing and data mining • Descriptive data summarization is need for quality data preprocessing • Data preparation includes §Data cleaning and data integration §Data reduction and feature selection §Discretization • A lot a methods have been developed but data preprocessing still an active area of research 38
  39. 39. Prof. Pier Luca Lanzi Run the Python notebooks for this lecture

×