Barga Data Science lecture 4

Deriving Knowledge from Data at Scale

• Opening Discussion 30 minutes
Review Discussion…
• Hands-On with Decision Trees 30 minutes
• Ensembles, Random Forests 60 minutes
• Data Science Modelling 30 minutes
Model performance evaluation…
• Machine Learning Boot Camp ~60 minutes
Clustering, k-Means…

• Optional Reading: Data Science Weekly (2)
• Two Homework Assignments, due next Wednesday
1. One is described in the lecture notes
2. Two is uploaded to the class Catalyst site
• Key Points to Understand, review and discuss
1. Ensembles, the techniques of Bagging and Boosting
2. Random Forests
3. Clustering, specifically K-Means Clustering
What will your data science workflow be? (not having one is a fail…)

gender age smoker eye
color
male 19 yes green
female 44 yes gray
male 49 yes blue
male 12 no brown
female 37 no brown
female 60 no brown
male 44 no blue
female 27 yes brown
female 51 yes green
female 81 yes gray
male 22 yes brown
male 29 no blue
lung
cancer
no
yes
yes
no
no
yes
no
no
yes
no
no
no
male 77 yes gray
male 19 yes green
female 44 no gray
?
?
?

color
male 19 yes green
female 44 yes gray
male 49 yes blue
male 12 no brown
female 37 no brown
female 60 no brown
male 44 no blue
female 27 yes brown
female 51 yes green
female 81 yes gray
male 22 yes brown
male 29 no blue
lung
cancer
no
yes
yes
no
no
yes
no
no
yes
no
no
no
male 77 yes gray
male 19 yes green
female 44 no gray
?
?
?
Train
ML Model

color
male 19 yes green
female 44 yes gray
male 49 yes blue
male 12 no brown
female 37 no brown
female 60 no brown
male 44 no blue
female 27 yes brown
female 51 yes green
female 81 yes gray
male 22 yes brown
male 29 no blue
lung
cancer
no
yes
yes
no
no
yes
no
no
yes
no
no
no
male 77 yes gray
male 19 yes green
female 44 no gray
yes
no
no
Train
ML Model

Define
Objective
Access and
Understand the
Data
Pre-processing
Feature and/or
Target
construction
1. Define the objective and quantify it with a metric – optionally with constraints,
if any. This typically requires domain knowledge.
2. Collect and understand the data, deal with the vagaries and biases in the data
acquisition (missing data, outliers due to errors in the data collection process,
more sophisticated biases due to the data collection procedure etc
3. Frame the problem in terms of a machine learning problem – classification,
regression, ranking, clustering, forecasting, outlier detection etc. – some
combination of domain knowledge and ML knowledge is useful.
4. Transform the raw data into a “modeling dataset”, with features, weights,
targets etc., which can be used for modeling. Feature construction can often
be improved with domain knowledge. Target must be identical (or a very
good proxy) of the quantitative metric identified step 1.

Feature selection
Model training
Model scoring
Evaluation
Train/ Test split
5. Train, test and evaluate, taking care to control
bias/variance and ensure the metrics are
reported with the right confidence intervals
(cross-validation helps here), be vigilant
against target leaks (which typically leads to
unbelievably good test metrics) – this is the
ML heavy step.

Data Science Workflow.pdf
Develop your own for defining
and evaluating project
opportunities…

Example 1: Amazon, big spenders. Target of the competition was
to predict customers who spend a lot of money among customers
using past purchases. The data consisted of transaction data in
different categories. But a winning model identified that ‘Free
shipping = True’ was an excellent predictor.
Leakage: “Free Shipping = True” was simultaneous with the sale,
which is a no-no… We can only use data from beforehand to
predict the future…

Example 2: Cancer patients plotted by Patient ID – what happened?
What could you do to improve this?...

Winning competition on leakage is easier than building good models.
But even if you don’t explicitly understand and game the leakage,
your model will do it for you. Either way, leakage is a huge problem.
• You need a strict temporal cutoff: remove all information just prior to the
event of interest.
• There has to be a timestamp on every entry and you need to keep it
• The best practice is to start from scratch with clean, raw data after careful
consideration
• You need to know how the data was created! I (try to ) work only with data I
pulled and prepared myself…

To avoid overfitting, we cross-validate and we cut down on the complexity of the model to
begin with. Here’s a standard picture (although keep in mind we generally work in high
dimensional space and don’t have a pretty picture to look at)

The art in data science
The science in data science
some
evaluation metric
rigorous
testing and experimentation to either validate or refute

Given
We need to determine
evaluation metrics

0.65
0.67
0.69
0.71
0.73
0.75
0.77
0.79
0 500 1000 1500 2000 2500 3000
A
c
c
u
r
a
c
y
# Training Records
Accuracy on test data stabilizes above 1000 training samples

Review: Decision Tree
1. Automatically selects features
2. Able to handle large number of features
3. Numeric, nominal, missing
4. Easy to ensemble (Random Forrest, Boosted DT)
5. I can romance on DTs for hours …

Blood pressure
Drug A Age
Drug A
Drug B
Drug B
high normal low
≤ 40 > 40
Assignment of drug to a patient

+
+
++
+
+
+
+
+
+
+
+

+
+
++
+
+
+
+
+
+
+
+
pm=5/6
Once regions are
chosen class
probabilities are easy
to calculate

Decision Trees & Weka

When completed, submit this assignment to
the dropbox for homework Lecture 4

Your task for this assignment: Design a simple, low-cost sensor that can distinguish
between red wine and white wine.
Your sensor must correctly distinguish between red and white wine for at least 95% of the
samples in a set of 6497 test samples of red and white wine.
Your technology is capable of sensing the following wine attributes:
- Fixed acidity - Free sulphur dioxide
- Volatile acidity - Total sulphur dioxide
- Citric acid - Sulphates
- Residual sugar - pH
- Chlorides - Alcohol
- Density
To keep your sensor cheap and simple, you need to sense as few of these attributes as
possible to meet the 95% requirement.
Question: Which attributes should your sensor be capable of measuring?

1. Go to our class website, Lecture 4, and download the associated homework files
2. Read WineQuality.pdf.
3. Open the RedWhiteWine.arff file in Weka, and remove the quality attribute, which you
will not need for this assignment.
4. Run J48 with default set-up to see what kind of percent correct classification results
you get using all attributes.
5. Remove attributes to find the minimum
number of attributes needed to meet
the 95% correct classification requirement.
Remove

Use these buttons to
simplify the task of
removing attributes
You can use these buttons
to simplify the task of
removing attributes

(Paste a screenshot showing your
minimum attribute set here)

(Paste a screenshot showing
your results for your minimum
attribute set here)

• Close

bagging
Decision trees

Diversity of Opinion
Independence
Decentralization
Aggregation

Ensemble Classification

Given
Method
Goal

The basic idea:
Randomly draw datasets with replacement from the
training data, each sample the same size as the original training set

Training
• Regression
• Classification

Two examples of random decisions in RFs

Ellipsoid separation 
Two categories,
Two predictors
Single tree decision boundary 100 bagged trees..

Random Forest Classifier
NexamplesTraining Data
M features

Nexamples
Create bootstrap samples
from the training data
....…
M features

Nexamples
Construct a decision tree
....…
M features

Nexamples
....…
M features
At each node in choosing the split feature
choose only among m<M features

Create decision tree
from each bootstrap sample
Nexamples
....…
....…
M features

Nexamples
....…
....…
Take the
majority
vote
M features

Random Forests

Consensus
Independence
Decentralization
Aggregation

Diversity of Opinion
private information
Independence
Decentralization
Aggregation

Decision Trees and Decision Forests
A forest is an ensemble of trees. The trees are all slightly different from one another.
terminal (leaf) node
internal
(split) node
root node0
1 2
3 4 5 6
7 8 9 10 11 12 13 14
A general tree structure
Is top
part blue?
Is bottom
part green?
Is bottom
part blue?
A decision tree

Decision Forest Model: the randomness model
1) Bagging (randomizing the training set)
The full training set
The randomly sampled subset of training data made available for the tree t
Forest training

Decision Forest Model: the randomness model
The full set of all possible node test parameters
For each node the set of randomly sampled features
Randomness control parameter.
For no randomness and maximum tree correlation.
For max randomness and minimum tree correlation.
2) Randomized node optimization (RNO)
Small value of ; little tree correlation. Large value of ; large tree correlation.
The effect of
Node weak learner
Node test params
Node training

Decision Forest Model: training and information gain
Beforesplit
Information gain
Shannon’s entropy
Node training
(for categorical, non-parametric distributions)
Split1Split2

Why we prune…

Classification Forest
Training data in feature space
?
?
?
Entropy of a discrete distribution
with
Classification tree
training
Obj. funct. for node j (information gain)
Training node j
Output is categorical
Input data point
Node weak learner
Predictor model (class posterior)
Model specialization for classification
( is feature response)
(discrete set)

Classification Forest: the weak learner model
Node weak learner
Node test params
Splitting data at node j
Weak learner: axis aligned Weak learner: oriented line Weak learner: conic section
Examples of weak learners
Feature response
for 2D example.
With a generic line in homog. coordinates.
Feature response
for 2D example.
With a matrix representing a conic.
Feature response
for 2D example.
In general may select only a very small subset of features
With or

Classification Forest: the prediction model
What do we do at the leaf?
leaf
leaf
leaf
Prediction model: probabilistic

Classification Forest: the ensemble model
Tree t=1 t=2 t=3
Forest output probability
The ensemble model

Training different trees in the forest
Testing different trees in the forest
(2 videos in this page)
Classification Forest: effect of the weak learner model
Parameters: T=200, D=2, weak learner = aligned, leaf model = probabilistic
• “Accuracy of prediction”
• “Quality of confidence”
• “Generalization”
Three concepts to keep in mind:
Training points

Classification Forest: with >2 classes
Training different trees in the forest
Testing different trees in the forest
Parameters: T=200, D=3, weak learner = conic, leaf model = probabilistic
Training points

Classification Forest: effect of tree depth
max tree depth, D
overfittingunderfitting
T=200, D=3, w. l. = conic T=200, D=6, w. l. = conic T=200, D=15, w. l. = conic
Predictor model = prob.(3 videos in this page)
Training points: 4-class mixed

Classification Forest: analysing generalization
Parameters: T=200, D=13, w. l. = conic, predictor = prob.
Training points: 4-class spiral Training pts: 4-class spiral, large gaps Tr. pts: 4-class spiral, larger gapsTestingposteriors

Q

10 Minute Break…

Data
Acquisition
Data
Exploration
Pre-processing
Feature and
Target
construction
Train/ Test
split
Feature
selection
Model
training
Model
scoring
Model
scoring
Evaluation
Evaluation
Compare
metrics

Model Scoring (subject for today…)

1 2 3 4 (k-1) k
Train Test

• Class
• Score

True
Label
Predicted Label
Confusion
matrix

Performance Metrics
Percent Reduction in Error
• 80% accuracy = 20% error
• Suppose learning increases accuracy from 80% to 90%
error reduced from 20% to 10%
• 50% reduction in error
• 99.90% to 99.99% = 90% reduction in error
• 50% to 75% = 50% reduction in error, can be applied to
many other measures

Performance Metrics
Precision and Recall
• Typically used in document retrieval
• Precision:
– how many of the returned documents are correct
– precision (threshold)
• Recall:
– how many of the positives does the model return
– recall (threshold)

Performance Metrics
Precision and Recall

Next week we will go deeper with ROC
curves, kappa, lift charts, etc…

• Break 5 minutes
• Close

similar
unsupervised learning
data exploration

grouping within a group are
similar and different from (or unrelated to)
the objects in other groups
Inter-cluster
distances are
maximized
Intra-cluster
distances are
minimized

• Outliers objects that do not belong to any cluster
outlier analysis
cluster
outliers

data reduction
natural clusters useful outlier detection

How many clusters?
Four ClustersTwo Clusters
Six Clusters

hierarchical
partitional

Original Points A Partitional Clustering

p4
p1
p3
p2
p4
p1
p3
p2
p4p1 p2 p3
p4p1 p2 p3
Traditional Hierarchical Clustering
Non-traditional Hierarchical Clustering
Non-traditional Dendrogram
Traditional Dendrogram

Single Linkage:
Minimum distance* *
Complete Linkage:
Maximum distance* *
Average Linkage:
Average distance*
*
*
*
Wards method:
Minimization of
within-cluster variance
*
*
*
*
*
¤
*
* *
*
¤
Centroid method:
Distance between
centres
*
*
*
* *
**
*
* *
¤ ¤
Non overlapping Overlapping
Hierarchical Non-hierarchical
1a 1b
1c
1a 1b
1b1
1b22
Agglomerative Divisive

d(x, y) x y metric
• d(i, j)  0 non-negativity
• d(i, i) = 0 isolation
• d(i, j) = d(j, i) symmetry
• d(i, j) ≤ d(i, h)+d(h, j) triangular inequality
real,
boolean, categorical, ordinal

p = 2, L2 Euclidean distance
weighted distance
)||...|
22
||
11
(|),( 222
d
y
d
xyxyxyxd 
)||...|
22
|
2
|
11
|
1
(),( 222
d
y
d
x
d
wxxwxxwyxd 
d
y
d
x
d
wyxwyxwyxd  ...
222111
),(

Q1 Q2 Q3 Q4 Q5 Q6
X 1 0 0 1 1 1
Y 0 1 1 0 1 0
• Jaccard similarity between binary vectors X and Y
• Jaccard distance between binary vectors X and Y
Jdist(X,Y) = 1- JSim(X,Y)
• Example:
• JSim = 1/6
• Jdist = 5/6
YX
YX
YXJSim


),(

• Lp Minkowski
p
p = 1, L1 Manhattan (or city block)
p
d
i i
y
i
x
pp
d
x
d
x
p
yx
p
yxyxpL
/1
1
)(
/1
||...|
22
||
11
|),(
























d
i i
y
i
x
d
y
d
xyxyxyxL
1
||...|
22
|||),(
1 11

centroid

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Sub-optimal Clustering
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Optimal Clustering
Original Points

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 1
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 3
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 4
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 5
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 6

 

K
i Cx
i
i
xmdistSSE
1
2
),(

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 1
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 3
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 4
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
x
y
Iteration 5

• Boolean Values
• Categories

That’s all for tonight….

Barga Data Science lecture 4

More Related Content

What's hot

Viewers also liked

Similar to Barga Data Science lecture 4

Recently uploaded

Barga Data Science lecture 4