This document provides an overview of data mining concepts and techniques. It discusses topics such as predictive analytics, machine learning, pattern recognition, and artificial intelligence as they relate to data mining. It also covers specific data mining algorithms like decision trees, neural networks, and association rules. The document discusses supervised and unsupervised learning approaches and explains model evaluation techniques like accuracy, ROC curves, gains/lift curves, and cross-entropy. It emphasizes the importance of evaluating models on test data and monitoring performance over time as patterns change.

1. Dan Steinberg
N Scott Cardell
Mykhaylo Golovnya
November, 2011
Salford Systems

3. Data Mining Data Mining Cont.
• Predictive Analytics • Statistics • OLAP
• Machine Learning • Computer science • CART
• Pattern Recognition • Database • SVM
• Artificial Management • NN
Intelligence • Insurance • CRISP-DM
• Business • Finance • CRM
Intelligence • Marketing • KDD
• Data Warehousing • Electrical • Etc.
Engineering
• Robotics
• Biotech and more

4.  Data mining is the search for patterns in data
using modern highly automated, computer
intensive methods
◦ Data mining may be best defined as the use of a specific
class of tools (data mining methods) in the analysis of
data
◦ The term search is key to this definition, as is
“automated”
 The literature often refers to finding hidden
information in data

5. • Study the phenomenon
• Understand its nature
Science • Try to discover a law
• The laws usually hold for a long time
•Collect some data
•Guess the model (perhaps, using science)
Statistics •Use the data to clarify and/or validate the model
•If looks “fishy”, pick another model and do it
again
• Access to lots of data
• No clue what the model might be
Data Mining • No long term law is even possible
• Let the machine build a model
• And let‟s use this model while we can

6.  Quest for the Holy Grail- build an algorithm that will
always find 100% accurate models
 Absolute Powers- data mining will finally find and
explain everything
 Gold Rush- with the right tool one can rip the stock-
market and become obscenely rich
 Magic Wand- getting a complete solution from start
to finish with a single button push
 Doomsday Scenario- all conventional analysts will
eventually be replaced by smart computer chips

7.  This is known as “supervised learning”
◦ We will focus on patterns that allow us to accomplish two tasks
 Classification
 Regression
 This is known as “unsupervised learning”
◦ We will briefly touch on a third common task
 Finding groups in data (clustering, density estimation)
 There are other patterns we will not discuss today
including
◦ Patterns in sequences
◦ Connections in networks (the web, social networks, link analysis)

8.  CART® (Decision Trees, C4.5, CHAID among others)
 MARS® (Multivariate Adaptive Regression Splines)
 Artificial Neural Networks (ANNs, many commercial)
 Association Rules (Clustering, market basket analysis)
 TreeNet® (Stochastic Gradient Tree Boosting)
 RandomForests® (Ensembles of trees w/ random splits)
 Genetic Algorithms (evolutionary model development)
 Self Organizing Maps (SOM, like k-means clustering)
 Support Vector Machine (SVM wrapped in many patents)
 Nearest Neighbor Classifiers

9.  (Insert chart)
 In a nutshell: Use historical data to gain
insights and/or make predictions on the new
data

10.  Given enough learning iterations, most data mining methods are
capable of explaining everything they see in the input
data, including noise
 Thus one cannot rely on conventional (whole sample) statistical
measures of model quality
 A common technique is to partition historical data into several
mutually exclusive parts
◦ LEARN set is used to build a sequence of models varying in size and level
of explained details
◦ TEST set is used to evaluate each candidate model and suggest the
optimal one
◦ VALIDATE set is sometimes used to independently confirm the optimal
model performance on yet another sample

11. Historical Data Build a
Sequence of
Learn Models
Monitor
Test
Performance
Confirm
Validate
Findings

12.  Analyst needs to indicate where the TEST data is to be
found
◦ Stored in a separate file
◦ Selected at random from the available data
◦ Pre-selected from available data and marked by a special indicator
 Other things to consider
◦ Population: LEARN and TEST sets come from different populations
(within-sample versus out-of-sample)
◦ Time: LEARN and TEST sets come from different time periods
(within-time versus out-of-time)
◦ Aggregation: logically grouped records must be all included or all
excluded within each set (self-correlation)

13.  Any model is built on past data!
 Fortunately, many models trace stable patterns of behavior
 However, any model will eventually have to be rebuilt:
◦ Banks like to refresh risk models about every 12 months
◦ Targeted marketing models are typically refreshed every 3 months
◦ Ad web-server models may be refreshed every 24 hours
 Credit risk score card expert Professor David
Hand, University of London maintains:
◦ A predictive model is obsolete the day it is first deployed

15.  Model evaluation is at the core of the learning process (choosing
the optimal model from a list of candidates)
 Model evaluation is also a key part in comparing performance of
different algorithms
 Finally, model evaluation is needed to continuously monitor
model performance over time
 In predictive modeling (classification and regression) all we need
is a sample of data with known outcome; different evaluation
criteria can then be applied
 There will never be “the best for all” model; the optimality is
contingent upon current evaluation criterion and thus depends
on the context in which the model is applied

16.  (insert graph)
 One usually computes some measure of average
discrepancy between the continuous model predictions f
and the actual outcome y
◦ Least Squared Deviation: R= Σ(y-f)^2
◦ Least Absolute Deviation: R= Σ Iy-fI
 Fancier definitions also exist
◦ Huber-M Loss: is defined as a hybrid between the LS and LAD
losses
◦ SVM Loss: ignores very small discrepancies and then switches to
LAD-style
 The raw loss value is often re-expressed in relative terms
as R-squared

17.  There are three progressively more demanding approaches to
solving binary classification problems
 Division: a model makes the final class assignment for each
observation internally
◦ Observations with identical class assignment are no longer discriminated
◦ A model needs to be rebuilt to change decision rules
 Rank: a model assigns a continuous score to each observation
◦ The score on its own bears no direct interpretation
◦ But, higher class score means higher likelihood of class presence in
general (without precise quantitative statements)
◦ Any monotone transformation of scores is admissible
◦ A spectrum of decision rules can be constructed strictly based on varying
score threshold without model rebuilding
 Probability: a model assigns a probability score to each observation
◦ Same as above, but the output is interpreted directly in the exact probabilistic
terms

18.  Depending on the prediction emphasis, various performance evaluation
criteria can be constructed for binary classification models
 The following list, far from being exhausting, presents some of the
frequently used evaluation criteria
◦ Accuracy (more generally- Expected Cost)
 Applicable to all models
◦ ROC Curve and Area Under Curve
 Not Applicable to Division Models
◦ Gains and Lift
 Not Applicable to Division Models
◦ Log-likelihood (a.k.a Cross-Entropy, Deviate)
 Not Applicable to Division and Rank Models
 The criteria above are listed in the order from the least specific to the
most
 It is not guaranteed that all criteria will suggest the same model as the
optimal from a list of candidate models

19.  Most intuitive and also the weakest evaluation method that can be
applied to any classification model
 Each record must be assigned to a specific class
 One first constructs a Prediction Success Table- a 2 by 2 matrix showing
how many true 0s and 1s (rows) were classified by the model correctly or
incorrectly (columns)
 The classification accuracy is then the number of correct class
assignments divided by the sample size
 More general approaches will also include user supplied prior
probabilities and cost matrix to compute the Expected Cost
 The example below reports prediction success tables for two separate
models along with the accuracy calculations
 The method is not sensitive enough to emphasize larger class unbalance
in model 1
 (insert table)

20.  The classification accuracy approach assumes that each record
has already been classified which is not always convenient
◦ Those algorithms producing a continuous score (Rank or Probability) will
require a user-specified threshold to make final class assignments
◦ Different thresholds will result to different class assignments and likely
different classification accuracies
 The accuracy approach focuses on the separating boundary and
ignores fine probability structure outside the boundary
 Ideally, need an evaluator working directly with the score itself
and not dependent on any external considerations like costs and
thresholds
 Also, for Rank models the evaluator needs to be invariant with
respect to monotone transformation of the scores so that the
“spirit” of such models is not violated

21.  The following approach will take full advantage of the set of
continuous scores produced by Rank or Probability models
 Pick one of the two target classes as the class in focus
 Sort a database by predicted score in descending order
 Choose a set of different score values
◦ Could be ALL of the unique scores produced by the model
◦ More often a set of scores obtained by binning sorted records into equal
size bins
 For any fixed value of the score we can now compute:
◦ Sensitivity (a.k.a True Positive): Percent of the class in focus with the
predicted scores above the threshold
◦ Specificity (a.k.a False Positive): Percent of the opposite class with the
predicted scores below the threshold
 We then display the results as a plot of [sensitivity] versus [1-specificity]
 The resulting curve is known as the ROC Curve

22.  (insert graph)
 ROC Curves for three different rank models are shown
 No model can be considered as the absolute best in all times
 The optimal model selection will rest with the user
 Average overall performance can be measured as Area Under
ROC Curve (AUC)
◦ ROC Curve (up to orientation) and AUC are invariant with respect to the
focus class selection
◦ The best attainable AUS is always 1.0
◦ AUC of a model with randomly assigned scores is 0.5
 AUC can be interpreted
◦ Suppose we randomly and repeatedly pick one observation at random
from the focus class and another observation from the opposite class
◦ Then AUC is the fraction of trials resulting to the focus class observation
having greater predicted score than the opposite class observation
◦ AUC below 0.5 means that something is fundamentally wrong

23.  The following example justifies another slightly different approach to model
evaluation
 Suppose we want to mail a certain offer to P fraction of the population
 Mailing to a randomly chosen sample will capture about P fraction of the
responders (random sampling procedure)
 Now suppose that we have access to a response model which ranks each potential
responder by a score
 Now if we sample the P fraction of the population targeting members with the
highest predicted scores first (model guided sampling), we could now get T
fraction of the responders which we expect to be higher than P
 The lift in P(th) percentile is defined as the ratio T/P
 Obviously, meaningful models will always produce lift greater than 1
 The process can be repeated for all possible percentiles and the results can be
summarized graphically as Gains and Cumulative Lift curves
 In practice, one usually first sorts observations by scores and then partitions
sorted data into a fixed number of bins to save on calculations just like it is
usually done for ROC curves

24.  (insert graphs and tables)

25.  (insert graphs)
 Lift in the given percentile provides a point measure of
performance for the given population cutoff
◦ Can be viewed as the relative length of the vertical line segment
connecting the gains curve at the given population cutoff
 Area Under the Gains curve (AUG): Provides an integral measure
of performance across all bins
◦ Unlike AUC, the largest attainable value of AUG is (1-p/2), P being the
fraction of responders in the population
 Just like ROC-curves, gains and lift curves for different models
can intersect, so that performance-wise one model is better for
one range of cutoffs while another model is better for a different
range
 Unlike ROC-curve, gains and lift curves do depend on the class
in focus
◦ For the dominant class, gains and lift curves degenerate to the trivial 45-
degree line random case

26.  ROC, Gains, and lift curves together with AUC and AUG are invariant
with respect to monotone transformation of the model scores
◦ Scores are only used to sort records in the evaluation set, the actual score
values are of no consequence
 All these measures address the same conceptual phenomenon
emphasizing different sides and thus can be easily derived from
each other
◦ Any point (P,G) on a gains curve corresponds to the point (P,G/P) on the
lift curve
◦ Suppose that the focus class occupies fraction F of the population; then
any point (P,G) on a gains curve corresponds to the point {(P-FG)/(1-F),G}
on the ROC curve
 It follows that the ROC graph “pushes” the gains graph “away” from the 45
degree line
 Dominant focus class (large F) is “pushed” harder so that the degeneracy of
its gain curve disappears
 In contrast, rare focus class (small F) has ROC curve naturally “close” to the
gains curve
 All of these measures are widely used as robust performance
evaluations in various practical applications

27.  When the output score can be interpreted as probability, a more specific
evaluation criterion can be constructed to access probabilistic accuracy
of the model
 We assume that the model generates P(X)-the conditional probability of
1 given X
 We also assume that the binary target Y is coded as -1 and +1 (only for
notational convenience)
 The Cross-Entropy (CXE) criterion is then computed as (insert equation)
◦ The inner Log computes the log-odds of Y=1
◦ The value itself is the negative log-likelihood assuming independence of
responses
◦ Alternative notation assumes 0/1 target coding and uses the following
formula (insert equation)
◦ The values produced by either of the formula will be identical to each
other
 Model with the smallest CXE means the largest likelihood and thus
considered to be the best in terms of capturing the right probability
structure

28.  The example shows true non-monotonic conditional probability
(dark blue curve)
 We generated 5,000 LEARN and TEST observations based on this
probability model
 We report predicted responses generated by different modeling
approaches
◦ Red- best accuracy MART model
◦ Yellow- best CXE MART model
◦ Cyan- univariate LOGIT model
 Performance-wise
◦ All models have identical accuracy but the best accuracy model is
substantially worse in terms of CXE
◦ LOGIT can‟t capture departure from monotonicity as reported by CXE

30.  MARS is a highly-automated tool for regression
 Developed by Jerome H. Friedman of Stanford University
◦ Annals of statistics, 1991 dense 65 page article
◦ Takes some inspiration from its ancestor CART®
◦ Produces smooth curves and surfaces, not the step-functions of CART
 Appropriate target variables are continuous
 End result of a MARS run is a regression model
◦ MARS automatically chooses which variables to use
◦ Variables are optimally transformed
◦ Interactions are detected
◦ Model is self-tested to protect against over-fitting
 Can also perform well on binary dependent variables
◦ Censored survival model (waiting time models as in churn)

31.  Harrison, D. and D. Rubinfeld.
Hedonic Housing Prices and Demand for Clean Air. Journal of
Environmental Economics and Management v5, 81-102, 1978
 506 census tracts in city of Boston for the year 1970
 Goal: study relationship between quality of life variables and property
values
◦ MV- median value of owner-occupied homes in tract („000s)
◦ CRIM- per capita crime rates
◦ NOX- concentration of nitrogen oxides (pphm)
◦ AGE- percent built before 1940
◦ DIS- weighted distance to centers of employment
◦ RM- average number of rooms per house
◦ LSTAT- percent neighborhood „lower socio-economic status‟
◦ RAD- accessibility to radial highways
◦ CHAS- borders Charles River (0/1)
◦ INDUS- percent non-retail business
◦ TAX- tax rate
◦ PT- pupil teacher ratio

32.  (insert graph)
 The dataset poses significant challenges to
conventional regression modeling
◦ Clearly departure from normality, non-linear
relationships, and skewed distributions
◦ Multicollinearity, mutual dependency, and outlying
observations

33.  (insert graph)
 A typical MARS solution (univariate for simplicity)
is shown above
◦ Essentially a piece-wise linear regression model with the
continuity requirement at the transition points called
knots
◦ The locations and number of knots were determined
automatically to ensure the best possible model fit
◦ The solution can be analytically expressed as
conventional regression equations

34.  Finding the one best knot in a simple regression is a straightforward
search problem
◦ Try a large number of potential knots and choose one with the best R-
squared
◦ Computation can be implemented efficiently using update algorithms;
entire regression does not have to be rerun for every possible knot (just
update X‟X matrices)
 Finding k knots simultaneously would require n^k order of
computations assuming N observations
 To preserve linear problem complexity, multiple knot
replacement is implemented in a step-wise manner:
◦ Need a forward/backward procedure
◦ The forward procedure adds knots sequentially one at a time
 The resulting model will have many knots and overfit the training data
◦ The backward procedure removes least contributing knots one at a time
 This produces a list of models of varying complexity
◦ Using appropriate evaluation criterion, identify the optimal model
 Resulting model will have approximately correct knot locations

35.  (insert graphs)
 True conditional mean has two knots at X=30
and X=60, observed data includes additional
random error
 Best single knot will be at X=45, subsequent best
locations are true knots around 30 and 60
 The backward elimination step is needed to
remove the redundant node at X=45

36.  Thinking in terms of knot selection works very well to
illustrate splines in one dimension but unwieldy for
working with a large number of variables simultaneously
◦ Need a concise notation easy to program and extend in multiple
dimensions
◦ Need to support interactions, categorical variables, and missing
values
 Basis functions (BF) provide analytical machinery to
express the knot placement strategy
 Basis function is a continuous univariate transform that
reduces predictor influence to a smaller range of values
controlled by a parameter c (20 in the example below)
◦ Direct BF: max(X-c, 0)- the original range is cut below c
◦ Mirror BF: max (c-X, 0)- the original range is cut above c
◦ (insert graphs)

37.  The following model represents a 3-knot
univariate solution for the Boston Housing
Dataset using two direct and one mirror basis
functions
 (insert equations)
 All three line segments have negative slope
even though two coefficients are above zero
 (insert graph)

38.  MARS core technology:
◦ Forward step: add basis function pairs one at a time in conventional step-
wise forward manner until the largest model size (specified by the user) is
reached
 Possible collinearity due to redundancy in pairs must be detected and
eliminated
 For categorical predictors define basis functions as indicator variables for all
possible subsets of levels
 To support interactions, allow cross products between a new candidate pair
and basis functions already present in the model
◦ Backward step: remove basis functions one at a time in conventional step-
wise backward manner to obtain a sequence of candidate models
◦ Use test sample or cross-validation to identify the optimal model size
 Missing values are treated by constructing missing value
indicator (MVI) variables and nesting the basis functions within
the corresponding MVIs
 Fast update formulae and smart computational shortcuts exist to
make the MARS process as fast and efficient as possible

39.  OLS and MARS regression (insert graphs)
 We compare the results of classical linear regression
and MARS
◦ Top three significant predictors are shown for each model
◦ Linear regression provides global insights
◦ MARS regression provides local insights and has superior
accuracy
 All cut points were automatically discovered by MARS
 MARS model can be presented as a linear regression model in
the BF space

41.  One of the oldest Data Mining tools for classification
 The method was originally developed by Fix and Hodges (1951) in an
unpublished technical report
 Later on it was reproduced by Agrawala (1977), Silverman and Jones
(1989)
 A review book with many references on the topic is Dasarathy (1991)
 Other books that treat the issue:
◦ Ripley B.D. 1996. Pattern Recognition and Neural Networks (chapter 6)
◦ Hastie T, Tibshirani R and Friedman J. 2001. The Elements of Statistical Learning Data
Mining, Inference and Prediction (chapter 13)
 The underlying idea is quite simple: make the predictions by proximity
or similarity
 Example: we are interested in predicting if a customer will respond to an
offer. A NN classifier will do the following:
◦ Identify a set of people most similar to the customer- the nearest neighbor
◦ Observe what they have done in the past on a similar offer
◦ Classify by majority voting: if most of them are responders, predict a
responder, otherwise, predict a non-responder

42.  (insert graphs)
 Consider binary classification problem
 Want to classify the new case highlighted in yellow
 The circle contains the nearest neighbors (the most similar
cases)
◦ Number of neighbors= 16
◦ Votes for blue class= 13
◦ Votes for red class= 3
 Classify the new case in the blue class. The estimated probability
of belonging to the blue class is 13/16=0.8125
 Similarly in this example:
◦ Classify the yellow instance in the blue class
◦ Classify the green instance in the red class
◦ The black point receives three votes from the blue class and another three
from the red one- the resulting classification is indeterminate

43.  There are two decisions that should be made in advance before
applying the NN classifier
◦ The shape of the neighborhood
 Answers the question “Who are our nearest neighbors?”
◦ The number of neighbors (neighborhood size)
 Answers the question “How many neighbors do we want to consider?”
 Neighborhood shape amounts to choosing the
proximity/distance measure
◦ Manhattan distance
◦ Euclidean distance
◦ Infinity distance
◦ Adaptive distances
 Neighborhood size K can vary between 1 and N (the dataset size)
◦ K=1-classification is based on the closest case in the dataset
◦ K=N-classification is always to the majority class
◦ Thus K acts as a smoothing parameter and can be determined by using a
test sample or cross-validation

44.  NN advantages
◦ Simple to understand and easy to implement
◦ The underlying idea is appealing and makes logical sense
◦ Available for both classification and regression problems
 Predictions determined by averaging the values of nearest neighbors
◦ Can produce surprisingly accurate results in a number of
applications
 NN have been proved to perform equal or better than LDA, CART, Neural
Networks and other approaches when applied to remote sensed data
 NN disadvantages
◦ Unlike decision trees, LDA, or logistic regression, their decision
boundaries are not easy to describe and interpret
◦ No variable selection of any kind- vulnerable to noisy inputs
 All the variables have the same weight when computing the distance, so
two cases could be considered similar (or dissimilar) due to the role of
irrelevant features (masking effects)
◦ Subject to the curse of dimensionality in high dimension datasets
◦ The technique is quite time consuming. However, Friedman et. Al.
(1975 and 1977) have proposed fast algorithms

46.  Classification and Regression Trees (CART®)- original approach
based on the “let the data decide local regions” concept
developed by Breiman, Friedman, Olshen, and Stone in 1984
 The algorithm can be summarized as:
◦ For each current data region, consider all possible orthogonal splits (based
on one variable) into 2 sub-regions
◦ The best split is defined as the one having the smallest MSE after fitting a
constant in each sub-region (regression) or the smallest resulting class
impurity (classification)
◦ Proceed recursively until all structure in the training set has been
completely exhausted- largest tree is produced
◦ Create a sequence of nested sub-trees with different amount of
localization (tree pruning)
◦ Pick the best tree based on the performance on a test set or cross-
validated
 One can view CART tree as a set of dynamically constructed
orthogonal nearest neighbor boxes of varying sizes guided by
the response variable (homogeneity of response within each box)

47.  CART is best illustrated with a famous example- the UCSD Heart
Disease study
◦ Given the diagnosis of a heart attack based on
 Chest pain, Indicative EKGs, Elevation of enzymes typically released by
damaged heart muscle, etc.
◦ Predict who is at risk of a 2nd heart attack and early death within 30 days
◦ Prediction will determine treatment program (intensive care or not)
 For each patient about 100 variables were available, including:
◦ Demographics, medical history, lab results
◦ 19 noninvasive variables were used in the analysis
 Age, gender, blood pressure, heart rate, etc.
 CART discovered a very useful model utilizing only 3 final
variables

48.  (insert classification tree)
 Example of a CLASSIFICATION tree
 Dependent variable is categorical (SURVIVE, DIE)
 The model structure is inherently hierarchical and cannot be represented
by an equivalent logistic regression equation
 Each terminal node describes a segment in the population
 All internal splits are binary
 Rules can be extracted to describe each terminal node
 Terminal node class assignment is determined by the distribution of the
target in the node itself
 The tree effectively compresses the decision logic

49.  CART advantages:
◦ One of the fastest data mining algorithms available
◦ Requires minimal supervision and produces easy to understand
models
◦ Focuses on finding interactions and signal discontinuities
◦ Important variables are automatically identified
◦ Handles missing values via surrogate splits
 A surrogate split is an alternative decision rule supporting the main rule
by exploiting local rank-correlation in a node
◦ Invariant to monotone transformations of predictors
 CART disadvantages:
◦ Model structure is fundamentally different from conventional
modeling paradigms- may confuse reviewers and classical
modelers
◦ Has limited number of positions to accommodate available
predictors- ineffective at presenting global linear structure (but
great for interactions)
◦ Produces coarse-grained piece-wise constant response surfaces

50.  (insert charts)
 10-node CART tree was built on the cell phone dataset
introduced earlier
 The root Node 1 displays details of TARGET variable in the
training data
◦ 15.2% of the 830 households accepted the marketing offer
 CART tried all variable predictors one at a time and found out
that partitioning the set of subjects based on the Handset Price
variable is most effective at separating responders from non-
responders at this point
◦ Those offered the phone with a price>130 contain only 9.9% responders
◦ Those offered a lower price<130 respond at 21.9%
 The process of splitting continues recursively until the largest
tree is grown
 Subsequent tree pruning eliminates least important branches
and creates a sequence of nested trees- candidate models

51.  (insert charts)
 The red nodes indicate good responders while the blue nodes
indicate poor responders
 Observations with high values on a split variable always go right
while those with low values go left
 Terminal nodes are numbered left to right and provide the
following useful insights
◦ Node 1: young prospects having very small phone bill, living in specific
cities are likely to respond to an offer with a cheap handset
◦ Node 5: mature prospects having small phone bill, living in specific cities
(opposite Node1) are likely to respond to an offer with a cheap handset
◦ Nodes 6 and 8: prospects with large phone bill are likely to respond as
long as the handset is cheap
◦ Node 10: “high-tech” prospects (having a pager) with large phone bill are
likely to respond to even offers with expensive handset

52.  (insert graph, table and chart)
 A number of variables were identified as
important
◦ Note the presence of surrogates not seen on the main
tree diagram previously
 Prediction Success table reports classification
accuracy on the test sample
 Top decile (10% of the population with the
highest scores) captures 40% of the responders
(lift of 4)

53.  (insert graphs)
 CART has a powerful mechanism of priors built
into the core of the tree building mechanism
 Here we report the results of an experiment with
prior on responders varying from 0.05 to 0.95 in
increments of 0.05
 The resulting CART models “sweep” the modeling
space enforcing different sensitivity-specificity
tradeoff

54.  As prior on the given class decreases
 The class assignment threshold increases
 Node richness goes up
 But class accuracy goes down
 PRIORS EQUAL uses the root node class ratio as the class assignment
threshold- hence, most favorable conditions to build a tree
 PRIORS DATA uses the majority rule as the class assignment threshold-
hence, difficult modeling conditions on unbalanced classes.
 In reality, a proper combination of priors can be found experimentally
 Eventually, when priors are too extreme, CART will refuse to build a tree.
◦ Often the hottest spot is a single node in the tree built with the most
extreme priors with which CART will still build a tree.
◦ Comparing hotspots in successive trees can be informative, particularly in
moderately-sized data sets.

55.  (insert graph)
 We have a mixture of two overlapping classes
 The vertical lines show root node splits for
different sets of priors. (the left child is classified
as red, the right child is classified as blue)
 Varying priors provides effective control over the
tradeoff between class purity and class accuracy

56.  Hot spots are areas of data very rich in the event of interest, even
though they could only cover a small fraction of the targeted
group
◦ A set of prospects rich in responders
◦ A set of transactions with abnormal amount of fraud
 The varying-priors collection of runs introduced above gives
perfect raw material in the search of hot spots
◦ Simply look at all terminal nodes across all trees in the collection and
identify the highest response segments
◦ Also want to have such segments as large as possible
◦ Once identified, the rules leading to such segments (nodes) are easily
available
◦ (insert graph)
◦ The graph on the left reports all nodes according to their target coverage
and lift
◦ The blue curve connects the nodes most likely to be a hot spot

57.  (insert graph)
 Our next experiment (variable shaving) runs as follows:
◦ Build a CART model with the full set of predictors
◦ Check the variable importance, remove the least important
variable and rebuild CART model
◦ Repeat previous step until all variables have been removed
 Six-variable model has the best performance so far
 Alternative shaving techniques include:
◦ Proceed by removing the most important variable- useful in
removal of model “hijackers”- variables looking very strong on the
train data but failing on the test data (e.g. ID variables)
◦ Set up nested looping to remove redundant variables from the
inner positions on the variable importance list

58.  (insert tree)
 Many predictive models benefit from Salford Systems
patent on “Structured Trees”
 Trees constrained in how they are grown to reflect
decision support requirements
◦ Variables allowed/disallowed depending on a level in a tree
◦ Variable allowed/disallowed depending on a node size
 In mobile phone example: want tree to first segment on
customer characteristics and then complete using price
variables
◦ Price variables are under the control of the company
◦ Customer characteristics are beyond company control

59.  Various areas of research were spawned by CART
 We report on some of the most interesting and well developed
approaches
 Hybrid models
◦ Combining CART with linear and Logistic Regression
◦ Combining CART with Neural Nets
 Linear combination splits
 Committees of trees
◦ Bagging
◦ Arcing
◦ Random Forest
 Stochastic Gradient Boosting (MART a.k.a TreeNet)
 Rule Fit and Path Finder

61.  (insert images)
 Grow a tree on training data
 Find a way to grow another tree, different from currently
available (change something in set up)
 Repeat many times, say 500 replications
 Average results or create voting scheme
◦ For example, relate PD to fraction of trees predicting default for a given
 Beauty of the method is that every new tree starts with a
complete set of data
 Any one tree can run out of data, but when that happens we just
start again with a new tree and all the data (before sampling)

62.  Have a training set of size N
 Create a new data set of size N by doing sampling with
replacement from the training set
 The new set (called bootstrap sample) will be different from the
original:
◦ 36.5% of the original records are excluded
◦ 37.5% of the original records are included once
◦ 18% of the original records are included twice
◦ 6% of the original records are included three times
◦ 2% of the original records are included four or more times
 May do this repeatedly to generate numerous bootstrap samples
 Example: distribution of record weights in one realized bootstrap
sample
 (insert table)

63.  To generate predicted response, multiple trees are combined via
voting (classification) or averaging (regression) schemas
 Classification trees “vote”
◦ Recall that classification trees classify
 Assign each case to ONE class only
◦ With 100 trees, 100 separate class assignment (votes) for each record
◦ Winner is the class with the most votes
◦ Fraction of votes can be used as a crude approximation to class
probability
◦ Votes could be weighted- say by accuracy of individual trees or node sizes
◦ Class weights can be introduced to counter the effects of dominant classes
 Regression trees assign a real predicted value for each case
◦ Predictions are combined via averaging
◦ Results will be much smoother than from a single tree

64.  Breiman reports the results of running bootstrap
aggregation (bagger) on four publicly available
datasets from Statlog project
 In all cases the bagger shows substantial
improvement in the classification accuracy
 It all comes at a price of no longer having a
single interpretable model, substantially longer
run time and greater demand on model storage
space
 (insert tables)

65.  Bagging proceeds by independent, identically-distributed
sampling draws
 Adaptive resampling: probability that a case is sampled varies
dynamically
◦ Cases with higher current prediction errors have greater probability of
being sampled in the next round
◦ Idea is to focus on these cases most difficult to predict correctly
 Similar procedure first introduced by Freund & Schapire (1996)
 Breiman variant (ARC-x4) is easier to understand:
◦ Suppose we have already grown K trees: let m= # times case i was
misclassified (0≤m≤k) (insert equations)
◦ Weight=1 for cases with zero occurrences of misclassification
◦ Weight= 1+k^4 for cases with K misclassifications
 Weigh rapidly becomes large is case is difficult to classify

66.  The results of running bagger and ARCer on the Boston
Housing Data are reported below
 Bagger shows substantial improvement over the single-
tree model
 ARCer shows marginal improvement over the bagger
 (insert table)
 Single tree now performs worse than stand alone CART run
(R-squared=72%) because in bagging we always work with
exploratory trees only
 Arcing performance beats MARS additive model but is still
inferior to the MARS interactions model

67.  Boosting (and Bagging) are very slow and consume a lot of
memory, the final models tend to be awkwardly large and
unwieldy
 Boosting in general is vulnerable to overtraining
◦ Much better fit on training than on test data
◦ Tendency to perform poorly on future data
◦ Important to employ additional considerations to reduce overfitting
 Boosting is also highly vulnerable to errors in the data
◦ Technique designed to obsess over errors
◦ Will keep trying to “learn” patterns to predict miscoded data
◦ Ideally would like to be able to identify miscoded and outlying data and
exclude those records from the learning process
◦ Documented in study by Dietterich (1998)
 An Experimental Comparison of Three Methods for Constructing Ensembles
of Decision Trees, Bagging, Boosting, and Randomization

69.  New approach for many data analytical tasks developed by
Leo Breiman of University of California, Berkeley
◦ Co-author of CART® with Friedman, Olshen, and Stone
◦ Author of Bagging and Arcing approaches to combining trees
 Good for classification and regression problems
◦ Also for clustering, density estimation
◦ Outlier and anomaly detection
◦ Explicit missing value imputation
 Builds on the notions of committees of experts but is
substantially different in key implementation details

70.  A random forest is a collection of single trees grown in a
special way
◦ Each tree is grown on a bootstrap sample from the learning set
◦ A number R is specified (square root by defualt) such that is
noticeably smaller than the total number of available predictors
◦ During tree growing phase, at each node only R predictors are
randomly selected and tried
 The overall prediction is determined by voting (in
classification) or averaging (in regression)
 The law of Large Numbers ensures convergence
 The key to accuracy is low correlation and bias
 To keep bias low, trees are grown to maximum depth

71.  Randomness is introduced in two distinct ways
 Each tree is grown on a bootstrap sample from the learning set
◦ Default bootstrap sample size equals original sample size
◦ Smaller bootstrap sample sizes are sometimes useful
 A number R is specified (square root by default) such that it is
noticeably smaller than the total number of available predictors
 During tree growing phase, at each node only R predictors are
randomly selected and tried.
 Randomness also reduces the signal to noise ratio in a single
tree
◦ A low correlation between trees is more important than a high signal when
many trees contribute to forming the model
◦ RandomForests™ trees often have very low signal strength, even when the
signal strength of the forest is high.

72.  (insert graph)
 Gold- Average of 50 Base Learners
 Blue- Average of 100 Base Learners
 Red- Average of 500 Base Learners

73.  (insert graph)
 Averaging many base learners improves the
signal to noise ratio dramatically provided
that the correlation of errors is kept low
 Hundreds of base learners are needed for the
most noticeable effect

74.  All major advantages of a single tree are automatically preserved
 Since each tree is grown on a bootstrap sample, one can
◦ Use out of bag samples to compute an unbiased estimate of the accuracy
◦ Use out of bag samples to determine variable importances
 There is no overfitting as the number of trees increases
 It is possible to compute generalized proximity between any pair
of cases
 Based on proximities one can
◦ Proceed with a target-driven clustering solution
◦ Detect outliers
◦ Generate informative data views/projections using scaling coordinates
◦ Do missing value imputation
 Interesting approaches to expanding the methodology into
survival models and the unsupervised learning domain

75.  RF introduces a novel way to define proximity between two observations:
◦ For a dataset of size N define an NXN matrix of proximities
◦ Initialize all proximities to zeroes
◦ For any given tree, apply the tree to the dataset
◦ If case i and case j both end up in the same node, increase proximity
proxij between i and j by one
◦ Accumulate over all trees in RF and normalize by twice the number of trees
in RF
 The resulting matrix provides intrinsic measure of proximity
◦ Observations that are “alike” will have proximities close to one
◦ The closer the proximity to 0, the more dissimilar cases i and j are
◦ The measure is invariant to monotone transformations
◦ The measure is clearly defined for any type of independent
variables, including categorical

77.  TreeNet (TN) is a new approach to machine learning and function
approximation developed by Jerome H, Friedman at Stanford
University
◦ Co-author of CART® with Breiman, Olshen and Stone
◦ Author of MARS®, PRIM, Projection Pursuit, COSA, RuleFit™ and more
 Also known as Stochastic Gradient Boosting and MART (Multiple
Additive Regression Trees)
 Naturally supports the following classes of predictive models
◦ Regression (continuous target, LS and LAD loss functions)
◦ Binary Classification (binary target, logistic likelihood loss function)
◦ Multinomial classification (multiclass target, multinomial likelihood loss
function)
◦ Poisson regression (counting target, Poisson Likelihood loss function)
◦ Exponential survival (positive target with censoring)
◦ Proportional hazard cox survival model
 TN builds on the notions of committees of experts and boosting
but is substantially different in key implementation details

78.  We focus on TreeNet because:
 It is the method introduced in the original Stochastic Gradient
Boosting article
 It is the method used in many successful real world studies
 We have found it to be more accurate than the other methods
◦ Many decisions that affect many people are made using a TreeNet model
◦ Major new fraud detection engine uses TreeNet
◦ David Cossock of Yahoo recently published a paper on uses of TreeNet in
web search
 TreeNet is a fully developed methodology. New capabilities
include:
◦ Graphical display of the impact of any predictor
◦ New automated ways to test for existence of interactions
◦ New ways to identify and rank interactions
◦ Ability to constrain model: allow some interactions and disallow others.
◦ Method to recast TreeNet model as a logistic regression.

79.  Built on CART trees and thus
◦ Immune to outliers
◦ Selects variables
◦ Results invariant with monotone transformations of variables
◦ Handles missing values automatically
 Resistant to mislabeled target data
◦ In medicine cases are commonly misdiagnosed
◦ In business, occasionally non-responders flagged as “responders”
 Resistant to overtraining- generalizes very well
 Can be remarkably accurate with little effort
 Trains very rapidly; comparable to CART

80.  2007 PAKDD competition: home loans up-sell to credit card owners
2nd place
◦ Model built in half a day using previous year submission as a blueprint
 2006 PAKDD competition: customer type discrimination 3rd place
◦ Model built in one day. 1st place accuracy 81.9% TreeNet accuracy 81.2%
 2005 BI-CUP Sponsored by University of Chile attracted 60 competitors
 2004 KDDCup “Most Accurate”
 2003 “Duke University/NCR Teradata CRN modeling competition
◦ Most Accurate and Best Top Decile Lift on both in and out of time samples
 A major financial services company has tested TreeNet across a
broad range of targeted marketing and risk models for the past two
years
◦ TreeNet consistently outperforms previous best models (around 10%
AUROC)
◦ TreeNet models can be built in a fraction of the time previously devoted
◦ TreeNet reveals previously undetected predictive power in data

81.  Begin with one very small tree as initial model
◦ Could be as small as ONE split generating 2 terminal nodes
◦ Typical model will have 3-5 splits in a tree, generating 4-6 terminal nodes
◦ Output is a continuous response surface regardless of the target type
 Hence, Probability modeling type for classification
◦ Model is intentionally “weak”- shrink all model predictions towards zero
by multiplying all predictions by a small positive learn rate
 Compute “residuals” for this simple model (prediction error) for
every record in data
◦ The actual definition of the residual in this case is driven by the type of the
loss function
 Grow second small tree to predict the residuals from first tree
 Continue adding more and more trees until a reasonable amount
has been added
◦ It is important to monitor accuracy on an independent test sample

82.  (insert chart)

83.  Trees are kept small (2-6 nodes common)
 Updates are small- can be as small as .01,.001,.0001
 Use random subsets of the training data in each cycle
◦ Never train on all the training data in any one cycle
 Highly problematic cases are IGNORED
◦ If model prediction starts to diverge substantially from observed data, that
data will not be used in further updates
 TN allows very flexible control over interactions:
◦ Strictly Additive Models (no interactions allowed)
◦ Low level interactions allowed
◦ High level interactions allowed
◦ Constraints: only specific interactions allowed (TN PRO)

84.  As TN models consist of hundreds or even thousands of trees there is no
useful way to represent the model via a display of one or two trees
 However, the model can be summarized in a variety of ways
◦ Partial Dependency Plots: These exhibit the relationship between the
target and any predictor- as captured by the model
◦ Variable Importance Rankings: These stable rankings give an excellent
assessment of the relative importance of predictors
◦ ROC and Gains Curves: TN Models produce scores that are typically unique
for each scored record
◦ Confusion Matrix: Using an adjustable score threshold this matrix displays
the model false positive and false negative rates
 TreeNet models based on 2-node trees by definition EXCLUDE interactions
◦ Model may be highly nonlinear but is by definition strictly additive
◦ Every term in the model is based on a single variable (single split)
 Build TreeNet on a larger tree (default is 6 nodes)
◦ Permits up to 5-way interaction but in practice is more like 3-way interaction
 Can conduct informal likelihood ratio test TN(2-node) versus TN(6-
node)
 Large differences signal important interactions

85.  (insert graphs)
 The results of running TN on the Boston
Housing Database are shown
 All of the key insights agree with previous
findings by MARS and CART

86.  Slope reverses due to interaction
 Note that the dominant pattern is downward
sloping, but that a key segment defined by
the 3rd variable is upward sloping
 (insert graph)

88.  CART: Model is one optimized Tree
◦ Model is easy to interpret as rules
 Can be useful for data exploration, prior to attempting a more complex
model
◦ Model can be applied quickly with a variety of workers:
 A series of questions for phone bank operators to detect fraudulent
purchases
 Rapid triage in hospital emergency rooms
◦ In some cases may produce the best or the most predictive model, for example
in classification with a barely detectable signal
◦ Missing values handled easily and naturally. Can be deployed effectively even
when new data have a different missingness pattern
 Random Forests: combination of many LARGE trees
◦ Unique nonparametric distance metric that works in high dimensional spaces
◦ Often predicts well when other models work poorly, e.g. data with high level
interactions
◦ In the most difficult data sets can be the best way to identify important
variables
 Tree Net: combination of MANY small trees
◦ Best overall forecast performance in many cases
◦ Constrained models can be used to test the complexity of the data structure
non-parametrically
◦ Exceptionally good with binary targets

89.  Neural Networks, combination of a few sigmoidal activation
functions
◦ Very complex models can be represented in a very compact form
◦ Can accurately forecast both levels and slopes and even higher order
derivatives
◦ Can efficiently use vector dependent variables
 Cross equation constraints can be imposed. (see Symmetry constraints for
feedforward network models of gradient systems, Cardell, Joerding, and
Li, IEEE Transactions on Neural Networks, 1993)
◦ During deployment phase, forecasts can be computed very quickly
 High voltage transmission lines use a neural network to detect whether there
has been a lightning strike and are fast enough to shut down the line before
it can be damaged
 Kernel function estimators, use a local mean or a local regression
◦ Local estimates easy to understand and interpret
◦ Local regression versions can estimate slopes and levels
◦ Initial estimation can be quick

90.  Random Forests:
◦ Models are large, complex and un-interpretable
◦ Limited to moderate sample sizes (usually less than 100,000
observations)
◦ Hard to tell in advance which case Random Forests will work well
on
◦ Deployed models require substantial computation
 Tree Net
◦ Models are large and complex, interpretation requires additional
work
◦ Deployed models either require substantial computation or post-
processing of the original model into a more compact form
 CART
◦ In most cases models are less accurate than TreeNet
◦ Works poorly in cases where effects are approximately linear in
continuous variables or additive over many variables

91.  Neural Networks:
◦ Neural Networks cover such a wide variety of models that no good widely-
applicable modeling software exists or may even be possible
 The most dramatic successes have been with Neural Network models that are
idiosyncratic to the specific case, and ere developed with great effort
 Fully optimized Neural Network parameter estimates can be very difficult to
compute, and sometimes perform substantially worse than initial statistically
inferior estimates. (this is called the “over training” issue)
◦ In almost all cases initial estimation is very compute intensive
◦ Limited to very small numbers of variables (typically between about 6 and
20 depending on the application)
 Kernel Function Estimators:
◦ Deployed models can require substantial computation
◦ Limited to small numbers of variables
◦ Sensitive to distance measures. Even a modest number of variables can
degrade performance substantially, due to the influence of relatively
unimportant variables on the distance metric

92.  Breiman, L., J. Friedman, R. Olshen and C. Stone
(1984), Classification and Regression Trees, Pacific Grove:
Wadsworth
 Breiman, L. (1996). Bagging predictors. Machine
Learning, 24, 123-140.
 Hastie, T., Tibshirani, R., and Friedman, J.H (2000). The
Elements of Statistical Learning. Springer.
 Freund, Y. & Schapire, R. E. (1996). Experiments with a
new boosting algorithm. In L. Saitta, ed., Machine
Learning: Proceedings of the Thirteenth National
Conference, Morgan Kaufmann, pp. 148-156.
 Friedman, J.H. (1999). Stochastic gradient boosting.
Stanford: Statistics Department, Stanford University.
 Friedman, J.H. (1999). Greedy function approximation: a
gradient boosting machine. Stanford: Statistics
Department, Stanford University.