Legal Analytics Course - Class 7 - Binary Classification with Decision Tree Learning - Professor Daniel Martin Katz + Professor Michael J Bommarito

Class 7
Binary Classiﬁcation & DecisionTree Learning
Legal Analytics
Professor Daniel Martin Katz
Professor Michael J Bommarito II
legalanalyticscourse.com

< Binary Classiﬁcation >
access more at legalanalyticscourse.com

http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html

Classiﬁcation to Predict Quantity
Classiﬁcation to Predict Category
Regression Methods
Trees, Forests, Knn, etc.

Adapted from Slides By
Victor Lavrenko and Nigel Goddard
@ University of Edinburgh
Take A LookThese 12

72
Female
Human
3
Female
Horse
36
Male
Human
21
Male
Human
67
Male
Human
29
Female
Human
54
Male
Human
44
Male
Human
50
Male
Human
42
Female
Human
6
Male
Dog
7
Female
Human

Task = Determine Whether the Agents
Will Obtain Employment?
Yes
No
f( )
Job?
Binary Classiﬁcation (Supervised Learning)

Classiﬁcation (Supervised Learning)
Yes
No
f( )
Job?

Classiﬁcation (Supervised Learning)
decision boundary
Yes
No
f( )
Job?
decision boundary

Multi Class Classiﬁcation

https://www.youtube.com/watch?v=p5rTio1G4ys

Task = Determine Whether the Agents
Will Obtain a Loan?
Yes
Perhapsf( )
Loan?
Multi Class Classiﬁcation (Supervised Learning)
No

f( )
Loan?
Yes
Perhaps
No

f( )
Loan?
Yes
No
Maybe
Yes
Perhaps
No

Multiclass = Hyperplane

Task = Determine the Age of the
Respective Agents
f( )
Age?
Regression (Supervised Learning)
#

Generative
vs.
Discriminant Models

Follow the video
and take your
own notes

Intro to DecisionTree Learning
Classiﬁcation And RegressionTree (CART)

DecisionTrees in DecisionTheory
DecisionTrees in Machine Learning
≠

Uses a set of binary rules applied to calculate a
target value
Used for classiﬁcation (categorical variables)
or regression (continuous variables)
Different algorithms are used to determine the
“best” split at a node
Introduction to DecisionTrees

“CART Approach”
to Decision Trees
Classiﬁcation And RegressionTree (CART)

https://www.youtube.com/watch?v=WOOTNBxbi8c

http://www.r-bloggers.com/a-brief-tour-of-the-trees-and-forests/

http://www.r-bloggers.com/classiﬁcation-tree-models/

https://www.youtube.com/watch?v=_RxqyvRK0Rw&list=PLD0F06AA0D2E8FFBA

Given Some Data:
(X1, Y1), ... , (Xn, Yn)
Now We Have a New Set of X’s
We Want to Predict the Y

Form a BinaryTree that
Minimizes the Error
in each leaf of the tree
CART
(Classiﬁcation & RegressionTrees)

Observe the Correspondence
Between the Data andTrees

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
Adapted from Example
By Mathematical Monk

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
We want to build an
approach which can
lead to the proper
classiﬁcation (labeling)
of new data points
( ) that are dropped
into this space

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
L e t s B e g i n t o
Partition the Space

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
1 2
1
2
L e t s B e g i n t o
Partition the Space
split 1
(a)

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
1 2
1
2
This Split Will Be
Memorialized in theTree
split 1
(a)

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
1 2
1
2
We Ask the Question is
Xi1 > 1 ? - with a binary
(yes or no) response
split 1
(a)
Xi1 > 1 ?
YesNo

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
1 2
1
2
If No - then we are in zone (a) ...
we tally the number of zeros and ones
Using Majority Rule do we assign a
classiﬁcation to this rule this leaf
split 1
(a)
Xi1 > 1 ?
YesNo
(0,5)
Classify as 1
zone (a)

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
1 2
1
2
Here we Classify as a 1 because
(0,5) which is 0 zero’s and 5 one’s
split 1
(a)
Xi1 > 1 ?
YesNo
(0,5)
Classify as 1
zone (a)

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
1 2
1
2
Using a Similar Approach Lets
Begin to Fill in the Rest of theTree
split 1
(a)
Xi1 > 1 ?
YesNo
(0,5)
Classify as 1
zone (a)

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
1 2
1
2
split 1
(a)
Xi1 > 1 ?
YesNo
(0,5)
Classify as 1
zone (a) Xi2 > 1.45 ?
No Yes
split 2

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0split 1
split 2
split 3
1 2 2.2
1
2
Xi1 > 1 ?
(0,5)
Xi2 > 1.45 ?
(4,1)(2,3)
Classify as 1
Classify as 1 Classify as 0
(a)
zone (a)
1.45
YesNo
No
(b)
(c)
zone (b) zone (c)
YesNo
Yes
Xi1 > 2 ?

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0split 1
split 2
split 3
split 4
1 2 2.2
1
2
Xi1 > 1 ?
(0,5)
Xi2 > 1.45 ?
Xi1 > 2.2 ?
(1,4)(5,0)(4,1)(2,3)
Classify as 1
(a)
zone (a)
1.45
YesNo
No
(b)
(c)
(d)
(e)
zone (b) zone (c)
YesNo YesNo
Yes
zone (d)
zone (e)
Xi1 > 2 ?

Okay Lets Add Back the ( )
which are new items
to be classiﬁed

For simplicity sake there
is one in each zone

We Will Use theTree Because
theTree Is Our Prediction Machine

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0split 1
split 2
split 3
split 4
1 2 2.2
1
2
Xi1 > 1 ?
(0,5)
Xi2 > 1.45 ?
Xi1 > 2.2 ?
(1,4)(5,0)(4,1)(2,3)
Classify as 1
(a)
zone (a)
1.45
YesNo
No
(b)
(c)
(d)
(e)
zone (b) zone (c)
Yes No YesNo
Yes
zone (d)
zone (e)
1
1
1
0 1
0
Xi1 > 2 ?

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
1 2
1
2
3
0
0
0
0
1
1
1
1
1
1 10
0
0
0
1
1 1
1
1 1
0
0
1
1 1
0
A B C
D
E
F
G
How about this one?

In this simple example, we
eyeballed the 2D space, partitioned
it and stopped after 4 Splits

Most Real Problems
are Not So Simple ...

Real problems are
n-dimensional (not 2D)
(1)

For real problems, you
need to select criteria
(or a criterion) for
deciding where to
partition (split) the data
(2)

For real problems you must
develop a stopping condition
or pursue recursive
partitioning of the space
(3)

Solutions to these 3 Problems
are among the core questions in
algorithm selection / development

From an Algorithmic Perspective -
TheTask is to Develop a
Method to Partition theTrees

Must Do So Without Knowing
the Speciﬁc Contours of the
Data / Problem in Question

So How Do We
TraverseThrough
The Data?

Optimal Partitioning of Trees is
NP-Complete

“Although any given solution to an NP-complete problem can
be verified quickly (in polynomial time), there is no known
efficient way to locate a solution in the first place; indeed, the
most notable characteristic of NP-complete problems is that no
fast solution to them is known.That is, the time required to
solve the problem using any currently known algorithm
increases very quickly as the size of the problem grows”

key implication is that one
cannot in advance determine
the “optimal tree”

Breiman, et al (1984) uses a
Greedy Optimization Method

Greedy Optimization Method
is used to calculate the MLE
(maximum-likelihood estimation)

Greedy is a Heuristic
“makes the locally optimal choice at each stage
with the hope of ﬁnding a global optimum. In
many problems, a greedy strategy does not in
general produce an optimal solution, but
nonetheless a greedy heuristic may yield locally
optimal solutions that approximate a global optimal
solution in a reasonable time.”

More onTrees (and Forests)
NextTime ...

Legal Analytics Course - Class 7 - Binary Classification with Decision Tree Learning - Professor Daniel Martin Katz + Professor Michael J Bommarito

More Related Content

What's hot

Viewers also liked

Similar to Legal Analytics Course - Class 7 - Binary Classification with Decision Tree Learning - Professor Daniel Martin Katz + Professor Michael J Bommarito

More from Daniel Katz

Recently uploaded

Legal Analytics Course - Class 7 - Binary Classification with Decision Tree Learning - Professor Daniel Martin Katz + Professor Michael J Bommarito