Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ι/Ο Data Εngineering


Published on

Slides I use for my data mining lectures wrt input/output data engineering: feature selection, discretization, dimensionality reduction and data projects, etc.

Published in: Science
  • Be the first to comment

  • Be the first to like this

Ι/Ο Data Εngineering

  1. 1. I/O Data Engineering “Garbage in, garbage out” November 2016 Kyriakos C. Chatzidimitriou - 1
  2. 2. Preamble • This work is a fusion of ideas and work (slides, text, images etc.) I found on the internet or had and wrote on my own regarding the area of input/output data engineering in data mining and machine learning. • Attribution: • The slides are based on the PowerPoint accompanying slides of the Data Mining, Practical Machine Learning Tools, Witten et al., 4th ed., 2017 and in particular Chapter 8, available at: • Slides from the Machine Learning MOOC by Prof. Andrew Ng: (PCA parts) • Slides from Learning from Data MOOC by Prof. Yaser S. Abu-Mustafa its support site: (The digits dataset and the non-linear transformation) • Slides from the Pattern Recognition class by Prof. Andreas L. Symeonidis, ECE department, Aristotle University of Thessaloniki • A tutorial on Principal Component Analysis by Lindsay I. Smith, February 2002 ( • Introduction to Data Mining, Tan et al., 2006: November 2016 Kyriakos C. Chatzidimitriou - 2
  3. 3. Successful data mining: Just apply a learner! No? • Select the learning algorithm • Scheme/parameter selection/tuning • Treat selection process as part of the learning process to avoid optimistic performance estimates • Estimate the expected true performance of a learning scheme • Split • Cross-validation • Data Engineering • Engineering the input data into a form suitable for the learning scheme chosen • Data engineering to make learning possible or easier • Engineering the output to make it more effective • Converting multi-class problems into two-class ones • Re-calibrating probability estimates November 2016 Kyriakos C. Chatzidimitriou - 3
  4. 4. Data Transformations – Outline Topics covered: 1. Attribute Selection 2. Discretizing Numeric Attributes 3. Data Projection 4. Data Cleansing 5. Transforming Multiple Classes to Binary Ones November 2016 Kyriakos C. Chatzidimitriou - 4
  5. 5. It is a jungle out there! Data Transformations Feature Selection Feature Engineering Data Engineering Dimensionality Reduction Principal Components Analysis Pre- and post-processing Data Cleansing ETL Feature Learning Wrapper methods Filter methods Independent Component Analysis Outlier Detection November 2016 Kyriakos C. Chatzidimitriou - 5
  6. 6. Attribute/Feature Selection Removing attributes that are not useful to the task at hand November 2016 Kyriakos C. Chatzidimitriou - 6
  7. 7. Motivation • Experiments have showed that adding useless attributes causes the performance of learning schemes (decision tree and rules, linear regression, instance-based learners) to deteriorate • Adding a random binary variable effects: • Divide-and-conquer tree learners and separate-and-conquer rule learners • If you reach depths at which only a small amount of data is available for picking a split, the random attribute will look good by chance • C4.5 deterioration in performance 5-10% for 1 random variable • Instance-based learners • Susceptible as well, reason: work in local neighborhoods • The number of training instances needed to produce a predetermined level of performance for instance-based learning increases exponentially with the number of irrelevant attributes present • Naive Bayes • Not susceptible • It assumes by design that all attributes are independent of one another, an assumption that is just right for random “distracter” attributes • On the other hand: pays a heavy price in other ways because its operation is damaged by adding redundant attributes • Independence “thrown out of the window” • Conclusion: Relevant attributes can also be harmful if they mislead the learning algorithm November 2016 Kyriakos C. Chatzidimitriou - 7
  8. 8. Advantages of Attribute Selection • Improves performance of learning algorithms • Speeds them up • Outweighed by the computation involved in attribute selection • Yields a more compact, more easily interpretable representation November 2016 Kyriakos C. Chatzidimitriou - 8
  9. 9. Attribute Selection Types • Manually • The best way • Requires deep understanding of the learning problem and what the attributes actually mean • Filter-method – Scheme-Independent Attribute Selection • Make an independent assessment based on general characteristics of the data • Wrapper method – Scheme-Dependent Attribute Selection • Evaluate the subset using the machine learning algorithm that will ultimately be employed for learning November 2016 Kyriakos C. Chatzidimitriou - 9
  10. 10. Scheme-Independent Attribute Selection • aka Filter approach to attribute selection: assess attributes based on general characteristics of the data • Attributes are selected in a manner that is independent of the target machine learning scheme • One method: find smallest subset of attributes that separates data • Another method: use a fast learning scheme that is different from the target learning scheme to find relevant attributes • E.g., use attributes selected by C4.5, or coefficients of linear model, possibly applied recursively (recursive feature elimination) November 2016 Kyriakos C. Chatzidimitriou - 10 By Lucien Mousin - Own work, GFDL,
  11. 11. Recursive Feature Elimination 1 2 … F1 F2 F3 Learning Algorithm Ranking: F2 F1 F3 1 2 … F1 F2 Learning Algorithm Ranking: F1 F2 Final Ranking: F1 F2 F3 Learning algorithm should produce a ranking, i.e. a linear SVM, where ranks are based on the size of the coefficients November 2016 Kyriakos C. Chatzidimitriou - 11
  12. 12. Correlation-based Feature Selection (CFS) • Correlation between attributes measured by symmetric uncertainty: where H is the entropy function: • Goodness of subset of attributes measured by where C is the class attribute, breaking ties in favour of smaller subsets. ]1,0[ )()( ),()()( 2),(     BHAH BAHBHAH BAU   j i j jij AAUCAU ),(),( 𝐻 𝑋, 𝑌 = − 𝑆 𝑋 𝑆 𝑌 𝑝 𝑥, 𝑦 log(𝑝 𝑥, 𝑦 )𝐻 𝑋 = − 𝑆 𝑋 𝑝 𝑥 log(𝑝 𝑥 ) November 2016 Kyriakos C. Chatzidimitriou - 12
  13. 13. The Weather Data Outlook Temperature Humidity Windy Play Sunny Hot High False No Sunny Hot High True No Overcast Hot High False Yes Rainy Mild High False Yes Rainy Cool Normal False Yes Rainy Cool Normal True No Overcast Cool Normal True Yes Sunny Mild High False No Sunny Cool Normal False Yes Rainy Mild Normal False Yes Sunny Mild Normal True Yes Overcast Mild High True Yes Overcast Hot Normal False Yes Rainy Mild High True No November 2016 Kyriakos C. Chatzidimitriou - 13
  14. 14. Symmetric Uncertainty Example Calculation • H(Outlook) = - 5/14log(5/14) – 4/14log(4/14) – 5/14 log(5/14) = 1.577 • H(Temperature) = - 4/14log(4/14) – 6/14log(6/14) – 4/14 log(4/14) = 1.556 • H(Outlook, Temperature) = - p(s,h)logp(s,h) - p(s,m)logp(s,m) - p(s,c)logp(s,c) - p(o,h)logp(o,h) - p(o,m)logp(o,m) - p(o,c)logp(o,c) - p(r,h)logp(r,h) - p(r,m)logp(r,m) - p(r,c)logp(r,c) = - 2/14log(2/14) – 2/14log(2/14) – 1/14 log(1/14) - 2/14log(2/14) – 1/14log(1/14) – 1/14 log(1/14) - 0/14log(0/14) – 3/14log(3/14) – 2/14 log(2/14) = 2.896 • U(Outlook, Temperature) = 2*(1.577 + 1.566 – 2.896)/(1.577 + 1.566) = 0.1512927 November 2016 Kyriakos C. Chatzidimitriou - 14
  15. 15. Attribute subsets for weather data The number of possible attribute subsets increases exponentially with the number of attributes, making exhaustive search impractical on all but the simplest problems. November 2016 Kyriakos C. Chatzidimitriou - 15
  16. 16. Scheme-specific selection • Wrapper approach to attribute selection: attributes are selected with target scheme in the loop • Implement “wrapper” around learning scheme • Evaluation criterion: cross-validation performance • Time consuming in general • greedy approach, k attributes, evaluation time multiplied by a factor of k2, worst case • prior ranking of attributes, complexity linear in k • Can use significance test (paired t-test) to stop cross-validation for a subset early if it is unlikely to “win” (race search) • Can be used with forward, backward selection, prior ranking, or special-purpose schemata search • Efficient for decision tables and Naïve Bayes (Selective Naïve Bayes) November 2016 Kyriakos C. Chatzidimitriou - 16 By Lastdreamer7591 - Own work, CC BY-SA 4.0,
  17. 17. Selective Naïve Bayes • Use the forward selection algorithm • Better able to detect a redundant attribute than backward elimination • Use as metric the quality of an attribute to be simply the performance on the training set • We know that: Training set performance not a reliable indicator of test set performance • But Naïve Bayes is less likely to overfit • Plus, as discussed, robust to random variables November 2016 Kyriakos C. Chatzidimitriou - 17
  18. 18. Complexity example • If I do 10-fold CV I must train the algorithm 10 times = 10 • I should do also the 10-fold CV 10 times to obtain a more reliable estimate = 10*10 • If I have 10 features the total search space is 2^10 = 1024 different subsets = 10*10*1024 = 102,400 • Then I should also tune the parameters of the learning algorithm… • Or should I do that before… November 2016 Kyriakos C. Chatzidimitriou - 18
  19. 19. Searching the attribute space • Number of attribute subsets is exponential in the number of attributes • Common greedy approaches: • forward selection • backward elimination • More sophisticated strategies: • Bidirectional search • Best-first search: • can find optimum solution, • does not just terminate when the performance starts to drop keeps a list of all attribute subsets evaluated so far, sorted in order of the performance measure, so that it can revisit an earlier configuration • Beam search: approximation to best-first search, keeps a truncated list • Genetic algorithms November 2016 Kyriakos C. Chatzidimitriou - 19
  20. 20. Discretization Transforming numeric attributes into discrete ones November 2016 Kyriakos C. Chatzidimitriou - 20
  21. 21. Motivation • Essential if the task involves numeric attributes but the chosen learning scheme can only handle categorical ones • Schemes that can handle numeric attributes often produce better results, or work faster, if the attributes are pre-discretized. • The converse situation, in which categorical attributes must be represented numerically, also occurs (although less often) November 2016 Kyriakos C. Chatzidimitriou - 21
  22. 22. Attribute discretization • Discretization can be useful even if a learning algorithm can be run on numeric attributes directly • Avoids normality assumption in Naïve Bayes and clustering • Examples of discretization we have already encountered: • Decision trees perform local discretization • Global discretization can be advantageous because it is based on more data • Apply learner to • k-valued discretized attribute or to • k – 1 binary attributes that code the cut points • The latter approach often works better when learning decision trees or rule sets November 2016 Kyriakos C. Chatzidimitriou - 22
  23. 23. Discretization: unsupervised • Unsupervised discretization: determine intervals without knowing class labels • When clustering, the only possible way! • Two well-known strategies: • Equal-interval binning • Equal-frequency binning (also called histogram equalization) • Unsupervised discretization is normally inferior to supervised schemes when applied in classification tasks • But equal-frequency binning works well with Naïve Bayes if the number of intervals is set to the square root of the size of dataset (proportional k-interval discretization) Data Equal interval width Equal frequency K-means November 2016 Kyriakos C. Chatzidimitriou - 23
  24. 24. Discretization: supervised • Classic approach to supervised discretization is entropy-based • This method builds a decision tree with pre-pruning on the attribute being discretized • Uses entropy as splitting criterion • Uses the minimum description length principle as the stopping criterion for pre-pruning • Works well: still the state of the art • To apply the minimum description length principle, the “theory” is • the splitting point (can be coded in log2[N – 1] bits) • plus class distribution in each subset (a more involved expression) • Description length is the number of bits needed for coding both the splitting point and the class distributions • Compare description lengths before/after adding split November 2016 Kyriakos C. Chatzidimitriou - 24
  25. 25. Example: temperature attribute Play Temperature Yes No Yes Yes Yes No No Yes Yes Yes No Yes Yes No 64 65 68 69 70 71 72 72 75 75 80 81 83 85 November 2016 Kyriakos C. Chatzidimitriou - 25
  26. 26. Final It can be shown theoretically that a cut point that minimizes the information value will never occur between two instances of the same class November 2016 Kyriakos C. Chatzidimitriou - 26
  27. 27. Formula for MDL stopping criterion • Can be formulated in terms of the information gain • Assume we have N instances • Original set: k classes, entropy E • First subset: k1 classes, entropy E1 • Second subset: k2 classes, entropy E2 • If the information gain is greater than the expression on the right, we continue splitting • Results in no discretization intervals for the temperature attribute in the weather data • Fail to play a role in the final decision structure N EkEkkE N N gain k 221122 )23(log)1(log     November 2016 Kyriakos C. Chatzidimitriou - 27
  28. 28. Supervised discretization: other methods • Can replace top-down procedure by bottom-up method • This bottom-up method has been applied in conjunction with the chi- squared test • Continue to merge intervals until they become significantly different • Can use dynamic programming to find optimum k-way split for given additive criterion • Requires time quadratic in the number of instances • But can be done in linear time if error rate is used instead of entropy • Error rate: count the number of errors that a discretization makes when predicting each training instance’s class, assuming that each interval receives the majority class. • However, using error rate is generally not a good idea when discretizing an attribute as we will see November 2016 Kyriakos C. Chatzidimitriou - 28
  29. 29. Error-based vs. entropy-based • Question: could the best discretization ever have two adjacent intervals with the same class? • Wrong answer: No. For if so, • Collapse the two • Free up an interval • Use it somewhere else • (This is what error-based discretization will do) • Right answer: Surprisingly, yes. • (and entropy-based discretization can do it) November 2016 Kyriakos C. Chatzidimitriou - 29
  30. 30. Error-based vs. entropy-based A 2-class, 2-attribute problem Entropy-based discretization can detect change of class distribution (from 100% to 50%) Class 1: a1 < 0.3 or if a1 < 0.7 and a2 < 0.5 Class 2: otherwise Best discretization a2: no problem a1: middle will have whatever label happens to occur most November 2016 Kyriakos C. Chatzidimitriou - 30
  31. 31. Data Projections and Dimensionality Reduction Projecting data into a more suitable space November 2016 Kyriakos C. Chatzidimitriou - 31
  32. 32. Motivation • Curse of Dimensionality • Visualization • Add new, synthetic attributes whose purpose is to present existing information in a form that is suitable for the machine learning scheme to pick up on. November 2016 Kyriakos C. Chatzidimitriou - 32
  33. 33. Curse of Dimensionality • When dimensions increase, data become increasingly sparse • Density and distance between points which are important criteria for clustering and outlier detection loose their importance •Create 500 points •Calculate the max and min distance between any pair of points November 2016 Kyriakos C. Chatzidimitriou - 33
  34. 34. Projections • Definition: a projection is a kind of function or mapping that transforms data in some way • Simple transformations can often make a large difference in performance • Example transformations (not necessarily for performance improvement): • Difference of two date attributes  age • Ratio of two numeric (ratio-scale) attributes • Useful for algorithms doing axis parallel splits • Concatenating the values of nominal attributes • Encoding cluster membership • Adding noise to data • Removing data randomly or selectively • Obfuscating the data • Anonymising November 2016 Kyriakos C. Chatzidimitriou - 34
  35. 35. Digits dataset November 2016 Kyriakos C. Chatzidimitriou - 35 From: Learning from data MOOC,
  36. 36. Input representation • ‘raw’ input x = (x0, x1, x2, …, x256) • Linear model: (w0, w1, w2, …, w256) • Features: extract useful information, e.g., • Intensity and symmetry x = (x0, x1, x2) • Linear model: (w0, w1, w2) From: Learning from data MOOC, November 2016 Kyriakos C. Chatzidimitriou - 36
  37. 37. Illustration of features November 2016 Kyriakos C. Chatzidimitriou - 37 From: Learning from data MOOC,
  38. 38. Another one November 2016 Kyriakos C. Chatzidimitriou - 38 From: Learning from data MOOC,
  39. 39. Methods • Unsupervised • Principal Components Analysis (PCA) • Independent Component Analysis (ICA) • Random Projections • Supervised • Partial Least Squares (PLS) • Linear Discriminant Analysis (LDA) November 2016 Kyriakos C. Chatzidimitriou - 39
  40. 40. Principal Components Analysis aka PCA November 2016 Kyriakos C. Chatzidimitriou - 40
  41. 41. Principal component analysis in a glance • Unsupervised method for identifying the important directions in a dataset • We can then rotate the data into the new coordinate system that is given by those directions • Finally we can keep the new dimension that are of more importance • PCA is a method for dimensionality reduction • Algorithm: 1. Find direction (axis) of greatest variance 2. Find direction of greatest variance that is perpendicular to previous direction and repeat • Implementation: find eigenvectors of the covariance matrix of the data • Eigenvectors (sorted by eigenvalues) are the directions November 2016 Kyriakos C. Chatzidimitriou - 41
  42. 42. PCA problem formulation Reduce from 2-dimension to 1-dimension: Find a direction (a vector ) onto which to project the data so as to minimize the projection error. Reduce from n-dimension to k-dimension: Find vectors onto which to project the data, so as to minimize the projection error. November 2016 Kyriakos C. Chatzidimitriou - 42
  43. 43. Data in a matrix form • Let n instances with d attributes. Every instance is described by d numerical values. • We represent our data as a nd matrix A with real numbers. • We can use linear algebra to process the matrix • Our goal is to produce a new nk matrix B such as: • It contains as much information as the original matrix A • To reveal something about the structure of data of A November 2016 Kyriakos C. Chatzidimitriou - 43
  44. 44. Principal Components • The first principal component is the direction of the axis with the largest variance in the data • The second principal component is the next orthogonal direction with the largest variance in the data • And so on.. • The 1st PC contains the largest variance • The kth PC contains the kth fraction of variance • For n original dimensions, the covariance matrix is nxn and has up to n eigenvectors. Thus, n PC. November 2016 Kyriakos C. Chatzidimitriou - 44
  45. 45. Example: 10-dimensional data • Data is normally standardized or mean-centered for PCA • Can also apply this recursively in a tree learnerNovember 2016 Kyriakos C. Chatzidimitriou - 45
  46. 46. PCA example • Dataset with 2 attributes x1 and x2. November 2016 Kyriakos C. Chatzidimitriou - 46
  47. 47. PCA example – Step 1: Get some data November 2016 Kyriakos C. Chatzidimitriou - 47 X - Data: x1 x2 2.5 2.4 0.5 0.7 2.2 2.9 1.9 2.2 3.1 3.0 2.3 2.7 2 1.6 1 1.1 1.5 1.6 1.1 0.9
  48. 48. Data Pre-processing Preprocessing (feature scaling/mean normalization): Replace each with . If different features on different scales (e.g., size of house, number of bedrooms), scale features to have comparable range of values. November 2016 Kyriakos C. Chatzidimitriou - 48
  49. 49. PCA example – Step 2: Subtract the mean November 2016 Kyriakos C. Chatzidimitriou - 49 X’- Mean normalization: x1 x2 .69 .49 -1.31 -1.21 .39 .99 .09 .29 1.29 1.09 .49 .79 .19 -.31 -.81 -.81 -.31 -.31 -.71 -1.01
  50. 50. Principal Component Analysis (PCA) algorithm Reduce data from -dimensions to -dimensions Compute “covariance matrix”: Compute “eigenvectors” of matrix : [U,S,V] = svd(Sigma); November 2016 Kyriakos C. Chatzidimitriou - 50
  51. 51. PCA Example – Step 3: Calculate the covariance matrix • Given that non-diagonal values are positive, we expect that x1 and x2 will increase together (+ sign of cov(x1, x2)) November 2016 Kyriakos C. Chatzidimitriou - 51 S = 0.6165556 0.6154444 0.6154444 0.7165556
  52. 52. Principal Component Analysis (PCA) algorithm From , we get:[U,S,V] = svd(Sigma) November 2016 Kyriakos C. Chatzidimitriou - 52
  53. 53. PCA Example – Step 4: Compute the eigenvectors of S • [U, D, V] = SVD(S) • The 1st eigenvector has an eigenvalue of 1.2840277, while the 2nd an eigenvalue of 0.0490834 • Eigenvectors are perpendicular to each other: orthogonal November 2016 Kyriakos C. Chatzidimitriou - 53 U = −0.6778734 −0.7351787 −0.7351787 0.6778734
  54. 54. Choosing k number of principal components Typically, choose to be smallest value so that “99% of variance is retained” (1%) November 2016 Kyriakos C. Chatzidimitriou - 54
  55. 55. Choosing k number of principal components [U,S,V] = svd(Sigma) Pick smallest value of for which (99% of variance retained) November 2016 Kyriakos C. Chatzidimitriou - 55
  56. 56. PCA Example – Step 5: Choosing components • Choosing the 1st component will retain more than 95% of the variance November 2016 Kyriakos C. Chatzidimitriou - 56
  57. 57. Principal Component Analysis (PCA) algorithm After mean normalization (ensure every feature has zero mean) and optionally feature scaling: Sigma = [U,S,V] = svd(Sigma); Ureduce = U(:,1:k); z = Ureduce’*x; November 2016 Kyriakos C. Chatzidimitriou - 57
  58. 58. PCA Example – Step 6: Deriving the new data • FinalData = RowFeatureVector x RowDataAdjust • RowDataAdjust = X’T • Normalized with zero mean with every row being a dimension and every column a point (inverted form) • RowFeatureVector = U’T • Eigenvectors are in rows with the most important in the first row November 2016 Kyriakos C. Chatzidimitriou - 58
  59. 59. Supervised learning speedup Extract inputs: Unlabeled dataset: New training set: Note: Mapping should be defined by running PCA only on the training set. This mapping can be applied as well to the examples and in the cross validation and test sets. November 2016 Kyriakos C. Chatzidimitriou - 59
  60. 60. Applications Application of PCA - Compression - Reduce memory/disk needed to store data - Speed up learning algorithm - Visualization November 2016 Kyriakos C. Chatzidimitriou - 60
  61. 61. Bad use of PCA: To prevent overfitting Use instead of to reduce the number of features to Thus, fewer features, less likely to overfit. This might work OK, but isn’t a good way to address overfitting. Use regularization instead. November 2016 Kyriakos C. Chatzidimitriou - 61
  62. 62. PCA is sometimes used where it shouldn’t be Design of ML system: - Get training set - Run PCA to reduce in dimension to get - Train logistic regression on - Test on test set: Map to . Run on How about doing the whole thing without using PCA? Before implementing PCA, first try running whatever you want to do with the original/raw data . Only if that doesn’t do what you want, then implement PCA and consider using . November 2016 Kyriakos C. Chatzidimitriou - 62
  63. 63. Data Cleansing Data Cleaning, Data Scrubbing, or Data Reconciliation November 2016 Kyriakos C. Chatzidimitriou - 63
  64. 64. What is data cleansing? • "Detect and remove errors and inconsistencies from data in order to improve the quality of data" [Rahm] • "The process of detecting and correcting (or removing) corrupt or inaccurate records from a record set, table, or database" [Wikipedia] • Integral part of data processing and maintenance • Usually semi-automatic process, highly application specific November 2016 Kyriakos C. Chatzidimitriou - 64
  65. 65. How to • Necessity of getting to know your data: understanding the meaning of all the different attributes, the conventions used in coding them, the significance of missing values and duplicate data, measurement noise, typographical errors, and the presence of systematic errors—even deliberate ones. • There are also automatic methods of cleansing data, of detecting outliers, and of spotting anomalies, which we describe—including a class of techniques referred to as “one-class learning” in which only a single class of instances is available at training time. November 2016 Kyriakos C. Chatzidimitriou - 65
  66. 66. Data Cleansing vs. Data Validation • Data validation almost invariably means data is rejected from the system at entry and is performed at entry time, rather than on batches of data. • Example: Data validation in web forms November 2016 Kyriakos C. Chatzidimitriou - 66
  67. 67. Anomalies Classification • Syntactical Anomalies • Lexical Errors (Gender: {M, M, F, 5' 8}) • Domain format errors (Smith, John vs. Smith John) • Irregularities: non-uniform use of values, units, abbreviations (examples: different currencies in the salaries, different use of abbreviations) • Semantic Anomalies • Integrity constraint violations (AGE >= 0) • Contradictions (AGE vs. CURRENT_DATE - DATE_OF_BIRTH) • Duplicates • Invalid tuples • Coverage Anomalies • Missing values • Missing tuples November 2016 Kyriakos C. Chatzidimitriou - 67
  68. 68. Data Quality Criteria: Accuracy + Uniqueness • Accuracy = Integrity + Consistency + Density • Integrity = Completeness + Validity • Completeness: M in D / M i.e. I should have 1000 tuples and I have 500 => 50% (Missing values) • Validity: M in D / D, i.e. From the 500 tuples I have in D the 400 are valid => 80% (Illegal values) • Consistency = Schema conformance + Uniformity • Schema conformance: tuples conforming to syntactical structure / overall number of tuples (if in the database then it conforms) • Uniformity: attributes with no irregularities (non-uniform use of values) / total number of attributes • Density: missing values in the tuples in D / total values in D • Uniqueness: tuples of the same entity / total number of tuples November 2016 Kyriakos C. Chatzidimitriou - 68
  69. 69. Data Cleansing Operations 1. Format adaptation for tuples and values 2. Integrity constraint enforcement 3. Derivation of missing values from existing ones 4. Removing contradictions within or between tuples 5. Merging and eliminating duplicates 6. Detection of outliers, i.e. tuples and values having a high potential of being invalid November 2016 Kyriakos C. Chatzidimitriou - 69
  70. 70. Data Cleansing Process 1. Data auditing 2. Workflow specification, i.e. choose appropriate methods to automatically detect and remove them 3. Workflow execution, apply the methods to the tuples in the data collection 4. Post-processing / Control November 2016 Kyriakos C. Chatzidimitriou - 70
  71. 71. Data Auditing • Data Profiling: Instance analysis • Data Mining: Whole data collection analysis • Examples • Minimal, Maximal values • Value range • Variance • Uniquness • Null value occurences • Typical string patterns (through RegExps for example) • Search for characteristics that could be used for the correction of anomalies November 2016 Kyriakos C. Chatzidimitriou - 71
  72. 72. Methods for Data Cleansing • Parsing (Syntax Errors) • Data Transformation (source to target format) • Integrity Constraint Enforcement (checking & maintenance) • Duplicate Elimination • Statistical Methods • Outliers • Detection: mean, std, range, clustering, association rules • Remedy: set to average or other statistical value, censored, truncated (dropped) • Missing • Detection: It's missing :) • Remedy: Filling-in (imputing) by a number of ways (mean, median, regression, propensity score, Markov-Chain-Monte-Carlo method) November 2016 Kyriakos C. Chatzidimitriou - 72
  73. 73. Outlier (Error) Detection in Datasets • Statistical: mean (μ), std (σ), range (Chebyshev's theorem) • Accept (μ - ε * σ) < f < (μ + ε σ), where ε=5, else reject • Needs training/testing data for finding the best ε {3,4,5,6, etc.} • Boxplots (univariate data) • Clustering, high-computational burden • Pattern-based, i.e. find a pattern where 90% of data exhibit the same characteristics • Association rules: pattern = association rule with high confidence and support November 2016 Kyriakos C. Chatzidimitriou - 73
  74. 74. Detecting anomalies • Visualization can help to detect anomalies • Automatic approach: apply committee of different learning schemes, e.g., • decision tree • nearest-neighbor learner • linear discriminant function • Conservative consensus approach: delete instances incorrectly classified by all of them • Problem: might sacrifice instances of small classes November 2016 Kyriakos C. Chatzidimitriou - 74
  75. 75. One-Class Learning • Usually training data is available for all classes • Some problems exhibit only a single class at training time • Test instances may belong to this class or a new class not present at training time • This the problem of one-class classification • Predict either target or unknown • Note that, in practice, some one-class problems can be re- formulated into two-class ones by collecting negative data • Other applications truly do not have negative data, e.g., password hardening, nuclear plant operational status November 2016 Kyriakos C. Chatzidimitriou - 75
  76. 76. Outlier detection • One-class classification is often used for outlier/anomaly/novelty detection • First, a one-class models is built from the dataset • Then, outliers are defined as instances that are classified as unknown • Another method: identify outliers as instances that lie beyond distance d from percentage p of training data • Density estimation is a very useful approach for one-class classification and outlier detection • Estimate density of the target class and mark low probability test instances as outliers • Threshold can be adjusted to calibrate sensitivity November 2016 Kyriakos C. Chatzidimitriou - 76
  77. 77. Transforming multiple classes to binary ones Output processing November 2016 Kyriakos C. Chatzidimitriou - 77
  78. 78. Transforming multiple classes to binary ones • Some learning algorithms only work with two class problems • e.g., standard support vector machines—only work with two-class problems. • Sophisticated multi-class variants exist in many cases but can be very slow or difficult to implement • A common alternative is to transform multi-class problems into multiple two-class ones • Simple methods: • Discriminate each class against the union of the others – one-vs.-rest • Build a classifier for every pair of classes – pairwise classification • We will discuss error-correcting output codes, which can often improve on these November 2016 Kyriakos C. Chatzidimitriou - 78
  79. 79. Error-correcting output codes • Multiclass problem  multiple binary problems • Simple one-vs.-rest scheme: One-per-class coding • 1010?? • Idea: use error-correcting codes instead • base classifiers predict 1011111, true class = ?? • Use bit vectors (codes) so that we have large Hamming distance between any pair of bit vectors: • Can correct up to (d – 1)/2 single-bit errors 0001d 0010c 0100b 1000a class vectorclass 0101010d 0011001c 0000111b 1111111a class vectorclass November 2016 Kyriakos C. Chatzidimitriou - 79
  80. 80. The End Kyriakos C. Chatzidimitriou November 2016 Kyriakos C. Chatzidimitriou - 80