Chapter: Introduction to Data Warehousing
and Data Mining




                             Introduction to
               ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                          Data Mining Introd...
Chapter: Introduction to Data Warehousing
and Data Mining




                                    Advantages of Data Minin...
Chapter: Introduction to Data Warehousing
and Data Mining




                             Data Mining using Separate Data...
Chapter: Introduction to Data Warehousing
and Data Mining




                                           Application Areas...
Chapter: Introduction to Data Warehousing
and Data Mining




                                                Lift Chart
 ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                                  Cross sell...
Chapter: Introduction to Data Warehousing
and Data Mining




                                                    Detectio...
Chapter: Introduction to Data Warehousing
and Data Mining




                                        Training Data vs. Te...
Chapter: Introduction to Data Warehousing
and Data Mining




                                                 Decision Tr...
Chapter: Introduction to Data Warehousing
and Data Mining




                                 Pros and Cons of Decision T...
Chapter: Introduction to Data Warehousing
and Data Mining




                                              How a Node Wor...
Chapter: Introduction to Data Warehousing
and Data Mining




                                           Neural Networks
 ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                             K-Nearest Neigh...
Chapter: Introduction to Data Warehousing
and Data Mining




                                                 Products
  ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                        Overview
           ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                   Overview of the process
 ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                     Suitability of the Prob...
Chapter: Introduction to Data Warehousing
and Data Mining




                                          Evaluate Results
 ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                    Examples of Classificati...
Chapter: Introduction to Data Warehousing
and Data Mining




                         Overlap of classification and estim...
Chapter: Introduction to Data Warehousing
and Data Mining




                                          Input and Output
 ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                      How to Collect Data
  ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                            Preparing Data
 ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                      Handling Missing Data
...
Chapter: Introduction to Data Warehousing
and Data Mining




                                         Transforming Data
 ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                Details on
                 ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                     Need for Preprocessing
...
Chapter: Introduction to Data Warehousing
and Data Mining




                                             Thresholding
  ...
Chapter: Introduction to Data Warehousing
and Data Mining




                             More on reducing the input spac...
Chapter: Introduction to Data Warehousing
and Data Mining




                                    Min-max normalization
  ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                                    Zscore
 ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                   Sigmoidal Normalization
 ...
Chapter: Introduction to Data Warehousing
and Data Mining




                           Sigmoidal Normalization Example 2...
Chapter: Introduction to Data Warehousing
and Data Mining




                                            Summary
        ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                                 Overview
  ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                    Deriving the Algorithm
 ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                Solving for Multiple Variabl...
Chapter: Introduction to Data Warehousing
and Data Mining




                                     Solving the Equation
  ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                                  Algorithm
...
Chapter: Introduction to Data Warehousing
and Data Mining




                                              Example (compu...
Chapter: Introduction to Data Warehousing
and Data Mining




                                      Predicted vs Actual
  ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                                 Decision Tr...
Chapter: Introduction to Data Warehousing
and Data Mining




                                         Computing a good Sp...
Chapter: Introduction to Data Warehousing
and Data Mining




                                   Computing Goodness of Spl...
Chapter: Introduction to Data Warehousing
and Data Mining




                                                     Summary...
Chapter: Introduction to Data Warehousing
and Data Mining




                          Bootstrapped Optimistic Algorithm ...
Chapter: Introduction to Data Warehousing
and Data Mining




                                             BOAT (continued...
Chapter: Introduction to Data Warehousing
and Data Mining




                                                    Problem
...
Introduction to Data Mining
Introduction to Data Mining
Introduction to Data Mining
Introduction to Data Mining
Introduction to Data Mining
Introduction to Data Mining
Introduction to Data Mining
Introduction to Data Mining
Upcoming SlideShare
Loading in...5
×

Introduction to Data Mining

1,144

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,144
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
64
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Introduction to Data Mining"

  1. 1. Chapter: Introduction to Data Warehousing and Data Mining Introduction to Data Mining (many slides in this section Are used courtesy of Carrig E m e r g i n g T e c h n o l o g i e s Ph: 410- 553- 6760 www.c a r r i g e t. c o m ) 1 Introduction to Data Mining Introduction to Data Mining • What is data mining • Need a data warehouse • Need to clean up data for mininig • Lots of algorithms 2 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 1
  2. 2. Chapter: Introduction to Data Warehousing and Data Mining Data Mining Introduction Data Mining Introduction • Data Mining examines a database and looks for patterns in the data • A data warehouse by itself will respond to queries from users – It will not tell users about patterns in data that users may not have thought about – To find patterns in data, data mining is used to try and mine key information from a data warehouse 3 Typical Data Architecture Typical Data Architecture Inventory OLTP Finance Inventory Data OLTP Sales OLTP OLTP Application Application Application Finance Enterprise S a lle s Sa es OLTP Information OLTP OLTP Data Data System Data Data Warehouse Finance Inventory Sales Subject Subject Subject California New York Area Area Area Data Mart Data Mart 4 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 2
  3. 3. Chapter: Introduction to Data Warehousing and Data Mining Advantages of Data Mining Advantages of Data Mining • Data mining allows companies to exploit information and use this to obtain competitive advantage. • Data mining helps identify trends such as: – why customers buy certain products – ideas for very direct marketing – ideas for shelf placement – training of employees vs. employee retention – employee benefits vs. employee retention 5 Implementing Data Mining Implementing Data Mining • Apply data mining tools to run data mining algorithms against data • There are two approaches: – Copy data from the Data Warehouse and mine it – Mine the data within the Data Warehouse • Popular tools use a variety of different data mining algorithms: – regression – association rules – genetic algorithms – decision trees – neural networks – clustering – classification 6 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 3
  4. 4. Chapter: Introduction to Data Warehousing and Data Mining Data Mining using Separate Data Data Mining using Separate Data • You can move data from the data warehouse to data mining tools – Advantages – Data mining tools may organize data so they can run faster – Disadvantages – Could be very expensive to move large amounts of data Data Warehouse Data Mining Tool Copy of data made by the Data Mining Tool 7 Data Mining Against the Data Warehouse Data Mining Against the Data Warehouse • Data mining tools can access data directly in the Data Warehouse – Advantages – No copy of data is needed for data mining – Disadvantages – Data may not be organized in a way that is efficient for the tool Data Warehouse Data Mining Tool 8 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 4
  5. 5. Chapter: Introduction to Data Warehousing and Data Mining Application Areas Application Areas • Database Marketing • Time-series prediction • Detection • Probability Estimation • Information compression • Sensitivity Analysis 9 Database Marketing Database Marketing • Response Modeling – Improve customer response by targeting prospects that are predicted as most likiely to respond to a particular advertisement or promotion (customer relationship management) • Benefits – Improved Return on Investment (ROI) – Improved inventory control and supply-side management – Improved customer relationships and retention • Lift Chart – A lift chart is often used to visualize and measure response modeling performance. – Shows how careful selection of customers ensures good response 10 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 5
  6. 6. Chapter: Introduction to Data Warehousing and Data Mining Lift Chart Lift Chart 100 90 80 70 60 random 50 40 careful 30 20 10 0 0 10 20 30 40 50 60 70 80 90 11 Cross Selling Cross Selling • Leverage existing customer base by selling them additional products or services • Similar to response modeling but only target existing customers • Coming histrorical purchase data with demographic data – Lifestyle – Credit reports – Purchases of other products • Targeted marketing – Only contact those customers that look promising – Create specialized mailing lists 12 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 6
  7. 7. Chapter: Introduction to Data Warehousing and Data Mining Cross selling Cross selling • Input – Purchase history – Purchase date – Amount of purchase – Number of purchases – Products purchased – Demographics – Age – Gender – Household size – Other – Did customer request information – Number of solicitations sent – How customer heard of the company • Output – Measure of how “approachable” the customer would be for a given product 13 Time-Series Prediction Time-Series Prediction • Time series – Sequence of numerical vectors collected over time – Trajectory of a baseball – Stock prices • Estimate one or more values appearing later in the sequence • Important aspect is you have to discard a lot of useless data. 14 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 7
  8. 8. Chapter: Introduction to Data Warehousing and Data Mining Detection Detection • Identify the existence or the occurrence of a condition • Fraud detection – May have only a few cases of fraud, have to find out what is different about these transactions • Intrusion Detection – Find patterns that indicate when an attack is made on an network • Need to compute the cost of missing a fraud and the cost of falsely accusing someone of fraud. • Can define classes of a fault. For baking cookies you might have: – Good-no defect – Broken – Underbaked – Overbaked – Too small – Too few chips 15 Others Others • Probability Estimation – Approximate the likelihood of an event given an observation. • Information Compression – Can be viewed as a special type of estimation problem. For a given set of data, estimate the key components that be can be used to construct the data. • Sensitivity Analysis – Understand how changes in one variable affect others. – Identify sensitivity of one variable on another (find out if dependencies existg). 16 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 8
  9. 9. Chapter: Introduction to Data Warehousing and Data Mining Training Data vs. Test Data Training Data vs. Test Data • Training data is known results that are used to make predictions on test data (unknown results). • Steps: – Step 1: Split training data into two halves (A & B) – Step 2: Build model against A – Step 3: Run model on B – Step 4: See how accurate model is – Step 5: If accurate, use it, if not return to Step 1 AGE SALARY BUYS MAGAZINE 20 30,000 Y 25 35,000 Y 30 40,000 N 35 45,000 Y 40 50,000 N 45 55,000 N 17 Typical Use of Training Data Typical Use of Training Data Steps: AGE SALARY BUYS MAGAZINE 1 20 30,000 Y A 40 50,000 N 35 45,000 Y 45 55,000 N B 25 35,000 Y 30 40,000 N MAKE MODEL FROM “A” ? 2 If AGE >= 20 AND AGE <= 40 3 APPLY IT TO “B” AND SALARY >= 30,000 AND It works 2 out of 3 SALARY >= 45,000 times!! 33 percent error THEN BUYS MAGAZINE 4 5 BACK TO STEP 1! IF 18 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 9
  10. 10. Chapter: Introduction to Data Warehousing and Data Mining Decision Trees Decision Trees • Used to represent a sequence of decisions to arrive at a predicted result • More general than matrices or flat tables ROOT SALARY > 50,000 Y N LEAVES AGE < 40 AGE > 20 Y N Y N AGE >30 AGE<30 AGE<20 AGE >50 Y Y Y N BUYS BUYS BUYS AGE>25 MAGAZINE MAGAZINE MAGAZINE 19 Prune the Tree Prune the Tree • Pruning the Tree means to analyze a portion of the Decision Tree. AGE > 20 Y N AGE<30 AGE<20 Y N BUYS MAGAZINE AGE>25 20 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 10
  11. 11. Chapter: Introduction to Data Warehousing and Data Mining Pros and Cons of Decision Trees Pros and Cons of Decision Trees • Pro – Easy to understand why they make decisions • Con – Can spend your life tuning the Decision Trees – Can take a long time to build 21 Neural Networks Neural Networks • Neural Networks Hidden Layers Input NODE Layer Output Layer 22 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 11
  12. 12. Chapter: Introduction to Data Warehousing and Data Mining How a Node Works How a Node Works Weight (a0)(w 0) + (a 1)(w1) (a2)(w 2) a0 w0 Outgoing activation NODES a1 w1 WT w2 Threshold If W > 0 Output=1 a2 OTHERWISE Output=0 23 How a Node Works (cont.) How a Node Works (cont.) Legend Weight Threshold Brown = 1 (a0)(w 0) + (a 1)(w1) If W > 0 Output=1 Blue = 2 (a2)(w 2) OTHERWISE Output=0 a0 Age = 24 a0 W 0 = -0.6 -15.7 a1 W 1 = .1 Hair = Brown a1 WT 0 a2 W 2 = -0.7 Eyes = Blue a2 a0 16 Age = 30 a0 W 0 = .5 a1 W 1 = .2 a1 WT 1 Hair = Brown a2 W 2 = .4 Eyes = Blue a2 24 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 12
  13. 13. Chapter: Introduction to Data Warehousing and Data Mining Neural Networks Neural Networks • Back Propagation Input Hidden Output Layer Layers Layer 25 Neural Networks Neural Networks • Pros – There are lots of Neural Network algorithms and some are very accurate – OCR (Optical Character Recognition) is a big success • Cons – Does not explain why it makes a decision – Back propagation is a very computationally expensive process 26 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 13
  14. 14. Chapter: Introduction to Data Warehousing and Data Mining K-Nearest Neighbor K-Nearest Neighbor • Idea is that related items are very similar in values. 40 SALARY 30 20 10 10 15 20 25 AGE 27 Research in Data Mining Research in Data Mining • Research is on-going – Knowledge and Data Discovery – VLDB (conference on very large databases) – SIGMOD (special interest group on management of data) – CIKM (conference on information and knowledge management) – Conference on Data and Knowledge Engineering 28 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 14
  15. 15. Chapter: Introduction to Data Warehousing and Data Mining Products Products • Large Scale – A few big ones, very expensive – Oracle (Darwin) – IBM (Intelligent Miner) – SAS (Enterprise Miner) – SPSS (Clementine) – Silicon Graphics (Mindset), more on visualization • Small Scale – Lots of these (desktop data miniing) – Pattern Recognition Workbench – CART, etc. 29 Data Mining: Summary Data Mining: Summary • Data mining attempts to find patterns in data that we did not know about • Often data mining is just a new buzzword for statistics • Data mining differs from statistics in that large volumes of data are used • Many different data mining algorithms exist and we will discuss them in the course • Examples – identify users who are most likely to commit credit card fraud – identify what attributes about a person most results in them buying product x. 30 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 15
  16. 16. Chapter: Introduction to Data Warehousing and Data Mining Overview Overview Data preparation for Data Mining Data preparation for Data Mining 31 Overview Overview • Development Process • Defining the Problem • Collection data • Preparing data 32 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 16
  17. 17. Chapter: Introduction to Data Warehousing and Data Mining Overview of the process Overview of the process Define the Problem Define the Problem Develop Choose Algorithm Model with Training data Define the Problem Evaluate Define the The Model Problem Repeat Whole Process 33 Define the problem Define the problem • What problems are suitable • Evaluating the results • What are the inputs and the outputs 34 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 17
  18. 18. Chapter: Introduction to Data Warehousing and Data Mining Suitability of the Problem Suitability of the Problem • Don’t Use – If there is a nice formula to answer the question (how long will it take light to travel from Mars to Pluto?) • Use – No good existing solution (nothing to lose) – Lots of historical data is lying around – Problem is not easily understood – Problem can be characterized as input-to-output relationship – Existing models have lots of problems 35 Modeling the Error Modeling the Error • Sometimes a known technique results in a certain amount of error. • If data mining can be used to estimate the error, this can be useful. • You can then use data mining in conjunction with a known technique. 36 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 18
  19. 19. Chapter: Introduction to Data Warehousing and Data Mining Evaluate Results Evaluate Results • Level of accuracy needed • What level of accuracy defines success • How will we benchmark performance • What other techniques will be tested • What kind of data will be used to evaluate the models. 37 Classification or Estimation Classification or Estimation • Classification says we need to categorize data (good customer, bad customer, etc.) • Estimation we want to define an estimate of how good a customer will be. • Very fine line. 38 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 19
  20. 20. Chapter: Introduction to Data Warehousing and Data Mining Examples of Classification Examples of Classification • Classification – Predicting likelihood of a prospect responding to a marketing offer – Automatically classifying bitmaps of handwritten characters (A-Z). – Identifying cookies as “tasty”, “burnt”, “hopeless” – Classifying bank mortgage as “good risk” or “bad risk” 39 Examples of Estimation Examples of Estimation • Predict the amount of money a customer will spend in next six months. • Estimate the market price for an option based on time to expiration. • Estimate the highest temperature for Chicago in the winter of 2008. • Estimate who will win the super bowl. 40 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 20
  21. 21. Chapter: Introduction to Data Warehousing and Data Mining Overlap of classification and estimation Overlap of classification and estimation • We could either classify cookies as “excellent”, “bad”, “tasty”, “hopeless” or we could estimate their “quality value” on a scale of 1-10 with 1 being “hopeless” and 10 being “excellent”. • In general classification is better if there is a natural grouping of classes. 41 Rules to choose between classification and Rules to choose between classification and estimation estimation • Are there a finite number of classes • Classes should be independent such that a “class 1” is not any closer to a “class 2”. The difference in “young”, “teenage”, “middleaged” is really not a good classification problem (too easy to miss and land in the wrong class). • Good classes might be to predict someones occuptation (e.g; farmer, baseball player, janitor, educator). • Discrete or continuous output. If output is continuous estimation is probably better. 42 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 21
  22. 22. Chapter: Introduction to Data Warehousing and Data Mining Input and Output Input and Output • For modeling, the output is the dependent variable • For estimation, the output is the classification of the input. • Start with small number of inputs and small number of outputs. • Most data mining is time consuming because key data is often missing or inaccurate. • Inputs should be casually related to output. – E.g. not like the “super bowl effect” where if the NFC wins, stock prices rise. 43 Collecting Data Collecting Data • What data to collect – Collect example patterns – Use recent data if system changes – Start with a small set of data 44 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 22
  23. 23. Chapter: Introduction to Data Warehousing and Data Mining How to Collect Data How to Collect Data • Choose an appropriate sample rate for time series • Make sure data measurements are consistent (can’t use some in meters and some in yards) • Leave out non-essential input variables • Make sure variables that vary have different values in your training set • Make sure no major physical changes have occurred since you last collected data (e.g. new baseball stadium would affect baseball predictions). 45 How much data in enough? How much data in enough? • Test by training on a small subset and compare with a large subset. • If the model does not change you might be done. • No good answer, it turns out to be very hard to compute how much data is needed for most data mining algorithms. • Note that sampling is often an effective technique, but if a large dataset is available, data mining algorithms are becoming scalable enough so that sampling is not a necessity. 46 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 23
  24. 24. Chapter: Introduction to Data Warehousing and Data Mining Preparing Data Preparing Data • Very few organizations have “clean” data. • Data entry systems often lack integrity rules or many different systems are used to populate a data mining input set. ; 47 Preparing Data Preparing Data • Handling missing data • Choosing numerical encodings (transforming categorical data) • Discarding erroneous data 48 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 24
  25. 25. Chapter: Introduction to Data Warehousing and Data Mining Handling Missing Data Handling Missing Data • Manually examine missing data and enter reasonable values • Use automated methods to fill in the missing values. – Average – Mode • Encode the fact the value is missing, instead of just A, B, C you have A, B, C with A-missing, B- missing, and C-missing. 49 Example Example Initial Input Preprocessed Input A BC A’ AM B’ BM C’ CM 100 0 120 100 0 0 0 120 0 ? 1 140 0 1 1 0 140 0 97 ? 200 97 0 0 1 200 0 96 2 ? 96 0 2 0 0 1 98 1 160 98 0 1 0 160 0 AM indicates whether or not A is a missing value. 50 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 25
  26. 26. Chapter: Introduction to Data Warehousing and Data Mining Transforming Data Transforming Data • Possible to transform data in categories input input values – Manual – PCA (principle component analysis) Reduction – Uses eigenvectors to “represent” a vector space 51 Inconsistent Data Inconsistent Data • Display input data in a graph to visually check for outliers • Thresholding to ensure that data falls within certain bounds • Calculate mean and standard deviation to look for outliers • Winzorize data – Set all values below a threshold to the min value – Set all values above the threshold to a max value 52 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 26
  27. 27. Chapter: Introduction to Data Warehousing and Data Mining Details on Preprocessing 53 Overview Overview • Need for preprocessing • Averaging • Thresholding • Reducing the input space • Normalizing • Modifying prior probabilities 54 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 27
  28. 28. Chapter: Introduction to Data Warehousing and Data Mining Need for Preprocessing Need for Preprocessing • Input space is large 10 – If you have 10 variables with only 10 possible values you have 10 = 10,000,000,000 possible different values. Need lots of training data to find meaningful patterns. • Often data mining can benefit from feature extraction – Instead of just have words, perhaps identify key features such as Person, Organization, etc. • Data often has missing values, need to fill them in. • Normalize data (many sources might use different units of measure) 55 Averaging Averaging • If the data is primarily time series data, a good method is to average several data points at a time. • For every 5 points simply use as input the average of each 5. Time Value T1 100 T2 200 T3 300 T4 400 T5 500 • Preprocessed: T1-T5: 300 56 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 28
  29. 29. Chapter: Introduction to Data Warehousing and Data Mining Thresholding Thresholding • For an image you might change all values of “dark gray” to black and all values of “very light gray” to bare. 57 Reducing the Input Space Reducing the Input Space • Principle Component Analysis – Factor analysis that redduces the number of input variables – Attempts to identify an m-dimensional subspace of the n- dimensional input space that seems most “significant” – PCA works by computing n vectors known as the prinicpal components which provide a basis for the input space. The vecotrs are ordered in sequence from most to least significant. – PCA requires all input variables to be normalized within the same range of data. – The highest prinicpal compoents will exhibit the highest variance. – Input space can be reduced by throwing out the ones with the lowest variance. 58 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 29
  30. 30. Chapter: Introduction to Data Warehousing and Data Mining More on reducing the input space More on reducing the input space • Sensitivity Analysis – Studies how the changes of an input variable affects the output variable. – If you do a regression and find one of the coefficients is zero, that variable may be viewed as having no real bearing in the production of the output. – Different from other data reduction in that we are looking at the output. – The idea is that if a variable changes and the output changes it’s a useful variable. Otherwise, it may not be so important. Here, we can see A, and B are more important C. C changes when output Y doesn’t change and vice versa. Y A B C 100 10 20 -50 100 12 14 300 150 35 40 -50 59 200 20 30 -50 Normalizing Data Normalizing Data • Many algorithms require normalized data – all data lies on a common scale (e.g. between zero and one) • Three flavors – Min-max – Zscore – Sigmoidal 60 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 30
  31. 31. Chapter: Introduction to Data Warehousing and Data Mining Min-max normalization Min-max normalization • Linear transformation of the original input range into a newly specified data range (typically 0-1). • Old min value is mapped to new min • Old max is mapped to new max • Let y be the original value, y’ be the new value. • Min1, max1 are the original min, max • Min2, max2 are the new min max  y − min 1  y'=  (max 2 − min 2) + min 2  max 1 − min 1  61 Min-max examples Min-max examples  y − min 1  y' =  (max 2 − min 2) + min 2  max 1 − min 1  • Consider old data that ranged from 0-100, we now obtain an equation to migrate it to 5-10.  y  y' =   + 5  20  • Old = 0, New = 5 • Old = 10, New = 5.5 • Old = 90, New = 9.5 62 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 31
  32. 32. Chapter: Introduction to Data Warehousing and Data Mining Zscore Zscore • Translate the input variable so that the mean is zero and the variance is one. • The attempts to center the data around the origin. The goal is that most of the data will lie within the origin to one standard deviation. • Works well when min and max are unknown • If majority of data falls within 50 and 100, but you have a few data points outside of that range, zscore will compress most of the data into a small range. 63 Zscore Continued Zscore Continued y − mean Ex: 2 y' = Y Y' std 45 35 0.24 Average 0.10 27.90 70 0.58 Std-dev 72.08 Y Y' 69 0.57 0.23 -0.56 Average 0.80 0.79 -0.01 22 -0.08 0.52 -0.28 Std-dev 1.02 10 -0.25 Ex: 1 0.25 -0.54 5 -0.32 0.80 0.00 15 -0.18 0.65 -0.15 30 0.03 0.92 0.11 300 3.78 0.21 -0.58 21 -0.10 0.64 -0.15 0.17 -0.62 11 -0.23 0.63 -0.17 14 -0.19 0.65 -0.15 27 -0.01 0.67 -0.13 -50 -1.08 0.71 -0.09 -75 -1.43 0.98 0.17 2 -0.36 0.84 0.04 0.10 -0.69 2 -0.36 0.77 -0.03 2 -0.36 0.50 -0.30 3 -0.35 64 5.00 4.11 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 32
  33. 33. Chapter: Introduction to Data Warehousing and Data Mining Sigmoidal Normalization Sigmoidal Normalization • Transforms input data non-linearly into the range –1 to 1. • Uses mean and standard deviation • Outliers are compressed along the means of the sigmoidal function • Good for data where it is desirable to keep the outliers 65 Sigmoidal Normalization Example 1 Sigmoidal Normalization Example 1 Alpha Y Sig Y' -0.56 0.23 -0.27 Average 0.80 y − mean -0.01 0.79 -0.01 α= -0.28 0.52 -0.14 Std-dev 1.02 std -0.54 0.25 -0.26 0.00 0.80 0.00 -0.15 0.65 -0.08 0.11 0.92 0.06 1 − e −α -0.58 0.21 -0.28 y' = -0.15 0.64 -0.08 -0.62 0.17 -0.30 1 + e −α -0.17 -0.15 0.63 0.65 -0.08 -0.07 -0.13 0.67 -0.06 -0.09 0.71 -0.04 0.17 0.98 0.09 0.04 0.84 0.02 -0.69 0.10 -0.33 -0.03 0.77 -0.02 -0.29 0.50 -0.15 4.11 5.00 0.97 66 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 33
  34. 34. Chapter: Introduction to Data Warehousing and Data Mining Sigmoidal Normalization Example 2 Sigmoidal Normalization Example 2 Alpha Y Sig Y' 0.24 45 0.12 Average 27.90 0.10 35 0.05 0.58 70 0.28 Std-dev 72.08 0.57 69 0.28 -0.08 22 -0.04 -0.25 10 -0.12 -0.32 5 -0.16 -0.18 15 -0.09 0.03 30 0.01 3.78 300 0.96 -0.10 21 -0.05 -0.23 11 -0.12 -0.19 14 -0.10 -0.01 27 -0.01 -1.08 -50 -0.49 -1.43 -75 -0.61 -0.36 2 -0.18 -0.36 2 -0.18 -0.36 2 -0.18 -0.35 3 -0.17 67 Sigmoidal and Zscore (Example 1) Sigmoidal and Zscore (Example 1) Y Zscore Y' Sig Y' Y Zcore Y' Sig Y' 0.23 -0.56 -0.27 45.00 0.24 0.12 0.79 -0.01 -0.01 35.00 0.10 0.05 0.52 -0.28 -0.14 70.00 0.58 0.28 0.25 -0.54 -0.26 69.00 0.57 0.28 0.80 0.00 0.00 22.00 -0.08 -0.04 0.65 -0.15 -0.08 10.00 -0.25 -0.12 0.92 0.11 0.06 5.00 -0.32 -0.16 Ex: 1 0.21 -0.58 -0.28 15.00 -0.18 -0.09 Ex: 2 0.64 -0.15 -0.08 30.00 0.03 0.01 0.17 -0.62 -0.30 300.00 3.78 0.96 0.63 -0.17 -0.08 21.00 -0.10 -0.05 0.65 -0.15 -0.07 11.00 -0.23 -0.12 0.67 -0.13 -0.06 14.00 -0.19 -0.10 0.71 -0.09 -0.04 27.00 -0.01 -0.01 0.98 0.17 0.09 -50.00 -1.08 -0.49 0.84 0.04 0.02 -75.00 -1.43 -0.61 0.10 -0.69 -0.33 2.00 -0.36 -0.18 0.77 -0.03 -0.02 2.00 -0.36 -0.18 0.50 -0.30 -0.15 2.00 -0.36 -0.18 5.00 4.11 0.97 3.00 -0.35 -0.17 68 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 34
  35. 35. Chapter: Introduction to Data Warehousing and Data Mining Summary Summary • Many large data mining projects spend most of their time on data cleanup and preparation (often cited as 80 percent of their work) • Various normalization techniques – Min-max – Zscore – Sigmoidal 69 Linear Regression Linear Regression 70 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 35
  36. 36. Chapter: Introduction to Data Warehousing and Data Mining Overview Overview • Goal is to fit points to a line • Once equation for a line is obtained, new values can be estimated (predicted) from inputs. 71 Example Example • Consider the following points for y and x. Given a value of x we would like a model that allows us to predict y. Sample Data Y X 15 5 60 50 22 10 40 35 20 30 Y 60 25 20 10 70 30 0 80 40 0 10 20 30 40 50 60 70 80 90 100 90 50 X 72 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 36
  37. 37. Chapter: Introduction to Data Warehousing and Data Mining Deriving the Algorithm Deriving the Algorithm • Equation for a line (one variable) is y = mx + b • where x is the input variable and b is the slope. Goal is to compute the slope m and the intercept b. • Equation for a plane in multidimensional space (we need for this multiple input variables). Here we use w y = w0 + w1x1 + w2 x2 + ... + wN x N 73 Input Input • We might have several records of training data, where each record indicates the values of each input variable and the output variable. • Input file would look like: Record 1: y1 x11 x12 Record 2: y2 x21 x22 Record 3: y2 x31 x32 74 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 37
  38. 38. Chapter: Introduction to Data Warehousing and Data Mining Solving for Multiple Variables Solving for Multiple Variables Record 1: y1 x11 x12 Record 2: y2 x21 x22 Record 3: y2 x31 x32 • We now need one set of slopes that will come close to solving the following equations: y1 = w0 + w1 x11 + w2 x12 y2 = w0 + w1 x 21 + w2 x22 y3 = w0 + w1 x31 + w2 x32 75 Using Matrices for Regression Using Matrices for Regression • Start with: y1 = w0 + w1 x11 + w2 x12 y2 = w0 + w1 x 21 + w2 x22 y3 = w0 + w1 x31 + w2 x32 X W D • Cast this 1 x11 x12   w0   y1 = w0 + w1 x11 + w2 x12  as a matrix      multiplicat 1 x21 x 22   w1  =  y2 = w0 + w1 x 21 + w2 x 22  1 x x   w   y = w + w x + w x  ion:  31 32   2   3 0 1 31 2 32  Now we just need to solve XW = D 76 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 38
  39. 39. Chapter: Introduction to Data Warehousing and Data Mining Solving the Equation Solving the Equation • Start with: XW = D • We want to solve for W. Pre-multiply each side by XT (the transpose of X). We now obtain: (XT X)W = (X TD) • Now pre-multiply each side by the inverse of (X TX)-1 (X T X) −1 (X T X)W = (X T X) −1(X T D) T − • We obtain: W= (X X)1(X D) T • Since: (XT X) −1(XT X) = I • I is the Identity 77 Matrix Important Points Important Points • W is just an estimate – not always possible to find an exact matrix inverse • Matrix inversion can be a computationally expensive process 78 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 39
  40. 40. Chapter: Introduction to Data Warehousing and Data Mining Algorithm Algorithm • Read input variables from a file • Compute X-transpose • Matrix Multiple X-transpose with X, store in temp • Take the matrix inverse of temp and store the result in left • Multiply X-transpose with D and store in right • Multiply left and right 79 Example (compute left) Example (compute left) T • Start with our two dimensional example and compute X 1 1 1 1 1 1 1 1  X T =    1 5  1 10 20 25 25 30 40 50     1 10   1 20    X = 1 25  7 180      ( X X) =  T   1 30  180 6150  1 40     1 50   0.577 - 0.017  ( XT X) −1 =   - 0.017 0  80 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 40
  41. 41. Chapter: Introduction to Data Warehousing and Data Mining Example (compute right) Example (compute right) • Start with the matrix and compute XTD 1 1 1 1 1 1 1 1  X T =    15  1 10 20 25 25 30 40 50    22    372    XT D =  35  =   12295 D  60   70     0.577 - 0.017    ( XT X) −1 =  80    - 0.017 0   90  7.014  W = (XT X)−1 (XT D) =   1.794  81 Graphing Solution Graphing Solution • Using the estimate for W we obtain XW  1 5  15.984    7.014   24.953  W=  1 10   1 20     X =  1 25  1.794    1 30    42.892    1 40       1 50   XW = 51.862  60.831    78.77  96.709   82  © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 41
  42. 42. Chapter: Introduction to Data Warehousing and Data Mining Predicted vs Actual Predicted vs Actual 15.984  15   24.953   22      Sample Data  42.892  35  120     100 XW = 51.862  D = 60  80 60 Y 60.831  70  40     20 78.77  80  0 96.709  90  0 10 20 30 40 50 60     X 83 Implementation Issues Implementation Issues • Lots of good matrix class libraries • Many matrix multiplication and matrix inversion algorithms • Many libraries lack support for cases where the matrix does not fit into main memory 84 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 42
  43. 43. Chapter: Introduction to Data Warehousing and Data Mining Decision Trees Decision Trees 85 Algorithm Algorithm • Assume we have a data set D and a list of variables to use to build the tree. • A split criteria is an object that consists of a variable and a value (e.g.; x > 10) Algorithm BuildTree(D, vars) currSplit := bestSplit(D,vars) vars := vars – currSplit.variable leftD = Partition(D, currSplit, left) rightD = Partition(D, currSplit, right)’ BuildTree(leftD, vars) BuildTree(rightD, vars) End 86 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 43
  44. 44. Chapter: Introduction to Data Warehousing and Data Mining Computing a good Split Computing a good Split Φ ( s ' , t ) = Max( Φ i ( si ' , t )) numClasses Φ(s,t) = 2P PR L ∑| P( j, t ) − P( j,t ) | L R j=1 tL = left split criteria (e.g.; x < 10) tR = right split criteria (e.gl; x >=10) vL = number of values for this variable (e.g.; x) that match criteria tL vR = number of values for this variable that match criteria tR cL = number of class j patterns for tL cR = number of class j patterns for tR n = size of the training set p = number of patterns at t (size of training set at this node) vL v c c P= L P = R P( j,tL ) = L P( j,tR) = R R n n p p 87 Details of Splitting Details of Splitting Φ( s' , t ) = Max(Φ i ( si ' , t )) Takes the maximum value of “goodness” of splits for each possible split criteria. For a variable with a range of 1 to 5, we must take the maximum of: X < 1, X >= 1 X < 2, X >= 2 X < 3, X >= 3 X < 4, X >= 4 X < 5, X >= 5 88 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 44
  45. 45. Chapter: Introduction to Data Warehousing and Data Mining Computing Goodness of Split Computing Goodness of Split numClasses Φ (s , t ) = 2PL PR ∑ | P( j , t j =1 L ) − P( j , t R ) | For a single split value, we compute the “goodness” by testing how well it splits all three classes. Consider a split of X < 5 for the following X, Y combination where we are using X to predict Y. X Y P L = 4/8 = 0.5 P R = 4/8 = 0.5 1 0 2 0 2 P L P R = (2)(.5)(.5) = .5 3 0 4 0 For Class 0: 5 1 P(j, tL) = 4/4 = 1 P(j, tR ) = 0 / 4 = 0 6 1 7 1 8 1 For Class 1: P(j, tL) = 0/4 = 1 P(j, tR ) = 4/4 = 1 Total for X < 5: (.5) (2) = 1.0 89 Split Example Split Example 1 0 NumRows 8 2 0 3 0 4 0 5 1 6 1 7 1 8 1 Split X<2 X<3 X<4 X<5 X<6 X<7 X<8 patl 1 2 3 4 5 6 7 patr 7 6 5 4 3 2 1 Pl 0.125 0.25 0.375 0.5 0.625 0.75 0.875 Pr 0.875 0.75 0.625 0.5 0.375 0.25 0.125 Pjl0 1.00 1.00 1.00 1.00 0.80 0.67 0.57 Pjr0 0.43 0.33 0.20 0.00 0.00 0.00 0.00 Pjl1 0.00 0.00 0.00 0.00 0.20 0.33 0.43 Pjr1 0.57 0.67 0.80 1.00 1.00 1.00 1.00 2PlPr 0.22 0.38 0.47 0.50 0.47 0.38 0.22 Pj0 0.57 0.67 0.80 1.00 0.80 0.67 0.57 Pj1 0.57 0.67 0.80 1.00 0.80 0.67 0.57 Sum 0.25 0.50 0.75 1.00 0.75 0.50 0.25 90 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 45
  46. 46. Chapter: Introduction to Data Warehousing and Data Mining Summary Summary • For each variable, obtain all distinct values – For each value I – For each possible output class – Compute the split as applied to the class – Sum all the classes • Take the max of all “goodness of split” values. 91 Scalability Scalability • Decision Tree is usually built such that it fits entirely in memory • Can do a memory-management scheme in which each split is computed incrementally. • Also, can build several decision trees and attempt to merge them. 92 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 46
  47. 47. Chapter: Introduction to Data Warehousing and Data Mining Bootstrapped Optimistic Algorithm for Tree Bootstrapped Optimistic Algorithm for Tree Construction (BOAT) Construction (BOAT) • BOAT – Goal is to improve scalability of decision tree algorithms – Most algorithms require training data to fit into main memory 93 Details of BOAT Details of BOAT • Basic Steps of BOAT – Build an initial estimate of a good tree by taking several samples of the training data (each sample fits into main memory) – Build a decision tree for each sample – Using all the samples, build an estimated tree. This is done by scanning in a top down fashion each sample. If the splitting variable matches at a given node there are two cases: – Categorical variable » (e.g.; gender = male). If the variable and value match perfectly in all the estimated trees, use this criteria as the estimated criteria. » Continuous variable. If the variable matches, then build a wide estimated split range that includes all of the ranges. For example: If one tree chooses X < 10 and another uses X < 15 – use X < 15 in our estimated tree. – Refine the estimates 94 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 47
  48. 48. Chapter: Introduction to Data Warehousing and Data Mining BOAT (continued) BOAT (continued) • Refining the estimates – For a wide confidence range, obtain all training data that fits into that range (this is a much smaller set than the whole training data). This either fits into main memory or can be periodically written to temp files. – Now if it’s a categorical variable there is no range, so there is no refinement to do. – If it s a continuous variable (10 < x < 30), try all splits within the range and test the purity of each split. Use the split with the best purity. – Now with the refined split, we can move on down the tree • More details are in the paper – J. Gehrke, V. Ganti, R. Ramakrishnan, W. Loh, “BOAT-Optimistic Decision Tree Construction”, In Proceedings of the 1999 SIGMOD Conference, Philadelphia, Pennsylvania, 1999. – (paper is at: http://www.cs.cornell.edu/johannes/) 95 Overview Overview • Statement of the Problem • Simple Examples • Algorithm • Multilevel associations 96 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 48
  49. 49. Chapter: Introduction to Data Warehousing and Data Mining Problem Problem • Basic idea is that we are given a set of transactions such as person bought bread, butter, milk, eggs and we would like to identify rules such that: – BREAD, BUTTER à Eggs • The idea is that we need to count various combinations of items that co-occur in a transaction and then filter them for the ones that are most common. 97 Queries Queries • Some common queries that association rules might provide some answers: – Find all rules that have Apple Juice in the consequent. Might give some idea of how to boost Juice sales. – Find all rules that have “shoe polish” in the antecedent. Might give some idea of what would happen to sales of other products if “shoe polish” was discontinued. – Find all rules that have “jelly” in the antecedent and “peanut butter” in the consequent. Tell us what else might be sold with the peanut butter and jelly combination. – Find all rules relating items located on a shelf A. Determine if co- locating on a shelf boosts sales. – Note that some large installations have said that they are not t hat inclined to re-arrange their shelves on a frequent basis. – Which parts of a shelf result in the most sales. 98 © 1999 Carrig Emerging Technology, Inc. All rights reserved. v1.0 49

×