The Three Forms of (Legal) Prediction: Experts, Crowds and Algorith...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

The Three Forms
of (Legal) Prediction
professor daniel martin katz
home | Illinois tech - chicago kent
blog | Computationa...

Three Types of Lawyers
(as described by paul lippe)

play “whack-a-mole”, reacting to
problems by creating fear and
friction within organizations and
the impression that there...

can help clients shape
(perhaps distort)
external perception of risk.
Merely Clever Lawyers

design systems that
balance risk and improve
transparency, helping clients
correctly price risk internally
Great Lawyers

On Background

Associate Professor of Law
IllinoisTech - Chicago Kent
Afﬁliated Faculty
Stanford CodeX
Center for Legal Informatics
Colle...

Fellow
Stanford CodeX
Center for Legal Informatics
Adjunct Professor
University of Michigan
Center for Study of Complex Sy...

Chief Strategy Ofﬁcer
LexPredict
Chief Executive Ofﬁcer
LexPredict

computationallegalstudies.com
Our
Blog
(since 2009)

@ computational

We are
#LegalInformatics
Researchers

Quantitative Legal Prediction
- or -
How I Learned to Stop Worrying and Start
Preparing for the Data Driven Future of the
...

The United States Tax Court
Cases and Dockets
#TaxLitigation

Measuring the Complexity of the Law:
The United States Code

Daniel Martin Katz, Joshua Gubler, Jon Zelner, Michael Bommarito, Eric Provins
& Eitan Ingall, Reproduction of Hierarchy...

Legal Language Explorer
Indexing 450,000+ Cases

#FreeTheLaw
#OpenSource

#ManagingFinancialRisk

Black
Reed
Frankfurter
Douglas
Jackson
Burton
Clark
Minton
Warren
Harlan
Brennan
Whittaker
Stewart
White
Goldberg
Fortas
M...

Acyclic digraphs arise in many natural and artiﬁcial processes. Among the
broader set, dynamic citation networks represent...

(2017 Forthcoming)
Legal Informatics
Ron DolinDaniel Martin Katz Michael Bommarito
35+ Contributors
(Edited Volume)
(Katz,...

#STEM + #LAW = Law’s Future

Daniel Martin Katz, The MIT
School of Law? A Perspective on
Legal Education in the 21st
Century, University of Illinois
La...

TheLawLab.com

Legal
Tech +
Innovation
Certiﬁcate

Quantitative Methods for Lawyers

http://www.quantitativemethodsclass.com/Professor Daniel Martin Katz
Intro Class

Legal Analytics
Professor Daniel Martin Katz
Professor Michael J Bommarito II

http://www.legalanalyticscourse.com/Professor Daniel Martin Katz
Professor Michael J. Bommarito II Advanced Class

The Age of
Data
Driven
Law
Practice

It Has Already Begun ...

implication is that every
organization needs a
data strategy
(including law ﬁrms & inside counsel)

Some Examples

The Age of
Quantitative Legal Prediction

The Age of
Quantitative Legal Prediction

The Age of
Quantitative Legal Prediction

The Age of
Quantitative Legal Prediction

Quantitative Legal Prediction
- or -
How I Learned to Stop Worrying and Start
Preparing for the Data Driven Future of the
...

Today we are going to
talk about one key
idea in prediction

There are 3 Known Ways
to Predict Something

Experts, Crowds, Algorithms

We could apply this to a
wide range of problems

For today we will apply
these approaches to the
decisions of the
Supreme Court of United States

Every year, law reviews, magazine and
newspaper articles, television and radio
time, conference panels, blog posts, and
tw...

There are only 3 ways  
to predict something
Experts
Crowds
Algorithms

Experts

Columbia Law Review
October, 2004
Theodore W. Ruger, Pauline T. Kim,
Andrew D. Martin, Kevin M. Quinn
Legal and Political ...

experts

Case Level Prediction
Justice Level Prediction
67.4% experts
58% experts
From the 68
Included
Cases
for the
2002-2003
Supr...

these experts probably
overﬁt

they ﬁt to the noise
and
not the signal

we need to
evaluate
experts and
somehow
benchmark
their
expertise

from a pure
forecasting
standpoint

the best
known
SCOTUS
predictor is

the law
version of
superforecasting

Crowds

crowds

https://fantasyscotus.lexpredict.com/case/list/
We can
generate
Crowd
Sourced
Predictions

however,
not all
members of
crowd are
made equal

we maintain
a ‘supercrowd’
which is the top n%
of predictors
up to time t

the
‘supercrowd’
outperforms
the overall
crowd
(and the best
single player)

(performance for the 2015 - 2016 term)

not
enough
crowd
based
decision
making in
(legal)
institutions

“Software developers were asked on two
separate days to estimate the completion
time for a given task, the hours they
proj...

in law
here
is our
commercial
offering

design
to
unlock
untapped
expertise
in
organizations
#Winning

Allowing
for
Frictionless
Crowdsourcing
#ManualUnderwriting

https://lexsemble.com/

https://lexsemble.com/

Algorithms

Black
Reed
Frankfurter
Douglas
Jackson
Burton
Clark
Minton
Warren
Harlan
Brennan
Whittaker
Stewart
White
Goldberg
Fortas
M...

Our approach is a special version
of random forest
Black
Reed
Frankfurter
Douglas
Jackson
Burton
Clark
Minton
Warren
Harla...

we have developed an
algorithm that we call
{Marshall}+
random forest

Benchmarking
since 1953
+
Using only data
available prior to
the decision
Mean Court Direction [FE]
Mean Court Direction 1...

Total Cases Predicted
Total Votes Predicted
7,700
68,964

Justice Prediction
Case Prediction
70.9% accuracy
69.6% accuracy
From 1953 - 2014
Version 2.0 is 1791 - 2015

Small Taste of How
Our Algorithm Works …

Breiman (1984) sets forth
the CART algorithm

Columbia Law Review
October, 2004
Theodore W. Ruger, Pauline T. Kim,
Andrew D. Martin, Kevin M. Quinn
Legal and Political ...

Given Some Data:
(X1, Y1), ... , (Xn, Yn)
Now We Have a New Set of X’s
We Want to Predict the Y

Form a BinaryTree that
Minimizes the Error
in each leaf of the tree
CART
(Classiﬁcation & RegressionTrees)

Observe the Correspondence
Between the Data andTrees

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
Adapted from Example
By Mathematical Monk

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
Adapted from Example
By Mathematical Monk
We want to build an
app...

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
Adapted from Example
By Mathematical Monk

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
Adapted from Example
By Mathematical Monk
L e t s B e g i n t o
P...

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
1 2
1
2
Adapted from Example
By Mathematical Monk
L e t s B e g i...

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
1 2
1
2
Adapted from Example
By Mathematical Monk
This Split Will...

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
1 2
1
2
Adapted from Example
By Mathematical Monk
We Ask the Ques...

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
1 2
1
2
Adapted from Example
By Mathematical Monk
If No - then we...

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
1 2
1
2
Adapted from Example
By Mathematical Monk
Here we Classif...

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
1 2
1
2
Adapted from Example
By Mathematical Monk
Using a Similar...

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
1 2
1
2
Adapted from Example
By Mathematical Monk
split 1
(a)
Xi1...

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0split 1
split 2
split 3
1 2 2.2
1
2
Xi1 > 1 ?
(0,5)
Xi2 > 1.45 ?
(...

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0split 1
split 2
split 3
split 4
1 2 2.2
1
2
Xi1 > 1 ?
(0,5)
Xi2 > ...

Okay Lets Add Back the ( )
which are new items
to be classiﬁed

For simplicity sake there
is one in each zone

We Will Use theTree Because
theTree Is Our Prediction Machine

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0split 1
split 2
split 3
split 4
1 2 2.2
1
2
Xi1 > 1 ?
(0,5)
Xi2 > ...

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0split 1
split 2
split 3
split 4
1 2 2.2
1
2
Xi1 > 1 ?
(0,5)
Xi2 > ...

1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0split 1
split 2
split 3
split 4
1 2 2.2
1
2
Xi1 > 1 ?
(0,5)
Xi2 > ...

Experts, Crowds, Algorithms

For most problems ...
ensembles of these streams
outperform any single stream

Humans
+
Machines

Humans
+
Machines
>

Humans
+
Machines
Humans
or
Machines
>

Ensembles come in
various forms

Here is a well known example

Poll Aggregation is one form of
ensemble where the learning question is
to determine how much weight (if any)
to assign to...

poll weighting

A Visual Depiction of
How to build an
ensemble method in our
judicial prediction example

expert crowd algorithm
ensemble method
learning problem is to discover when to use a given stream of intelligence

expert crowd algorithm
via back testing we can learn the
weights to apply for particular problems
ensemble method
learning...

{Marshall}+
algorithm

expert
crowd
algorithm

{Marshall}+ improvement
will likely come from
determining the optimal
weighting of experts,
crowds and algorithms
for vari...

ERISA cases
thus
might look like this

Patent cases
perhaps
might look like this

Search/Seizure cases
while
could look like this

this is one slice of our
research effort …

Given our ability to offer
forecasts of judicial
outcomes, we wondered
if this information could
inform an event-driven
tr...

Paper Released
August 24, 2015
http://arxiv.org/abs/1508.05751
available at
http://papers.ssrn.com/sol3/papers.cfm?abstrac...

We call this idea
“Law on the Market”
(LOTM)

A Motivating Example
Myriad Genetics
NASDAQ: MYGN
Market Cap of ~$3 billion+

Myriad Genetics
“Myriad employs a number of proprietary
technologies that permit doctors and patients
to understand the ge...

Myriad Genetics
“Myriad was the subject of scrutiny
after it became involved in a lengthy
lawsuit over its controversial p...

June 13, 2013
Supreme Court
offers this
decision
~10:05am

Initial Media
Reports and
Initial Trading
11:48am

Initial Media
Reports
Early
Afternoon
“In early afternoon trading
Thursday, Myriad shares
were up 5.4 percent, or
$2.36, a...

Final Media
Reports

Final Media
Reports

-0.050
-0.025
0.000
0.025
AverageCumulativeAbnormalReturns
NASDAQ: MYGN
Pegged to S&P 500
(Market Model)
June 12

-0.050
-0.025
0.000
0.025
AverageCumulativeAbnormalReturns
NASDAQ: MYGN
Pegged to S&P 500
(Market Model)
June 12
9:30am

-0.050
-0.025
0.000
0.025
AverageCumulativeAbnormalReturns
10:00am ET
NASDAQ: MYGN
Pegged to S&P 500
(Market Model)
June 1...

-0.050
-0.025
0.000
0.025
AverageCumulativeAbnormalReturns
10:00am ET
11:40am ET
NASDAQ: MYGN
Pegged to S&P 500
(Market Mo...

-0.050
-0.025
0.000
0.025
AverageCumulativeAbnormalReturns
10:00am ET
1:20pm ET
11:40am ET
NASDAQ: MYGN
Pegged to S&P 500
...

-0.050
-0.025
0.000
0.025
AverageCumulativeAbnormalReturns
10:00am ET
1:20pm ET
11:40am ET
NASDAQ: MYGN
Pegged to S&P 500
...

-0.050
-0.025
0.000
0.025
AverageCumulativeAbnormalReturns
10:00am ET
1:20pm ET
11:40am ET
NASDAQ: MYGN
Pegged to S&P 500
...

Paper Released
August 24, 2015

lots of litigation decisions
are just a version of this basic idea
law = ﬁnance

this is a part of the
industry where you
need rigorous
#LegalUnderwriting

but lots of litigation decisions
are actually implicit litigation ﬁnance
(or self insurance)
#ﬁn(legal)tech

however most implicit litigation
ﬁnance is not based upon  
rigorous underwriting …
law =! ﬁnance
(but it will)

we expand on this theme in this presentation
http://computationallegalstudies.com/2015/10/ﬁn-legal-tech-laws-future-from-ﬁ...

TheLawLab.com

FinLegalTechConference.comNovember 4, 2016

A Few Plugs …

LexPredict.com

ComputationalLegalStudies.com
BLOG

Michael J. Bommarito II
@ mjbommar
computationallegalstudies.com
lexpredict.com
bommaritollc.com
university of michigan ce...

Daniel Martin Katz
@ computational
computationallegalstudies.com
lexpredict.com
danielmartinkatz.com
illinois tech - chica...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

Fin (Legal) Tech – Law’s Future fro... by Daniel Katz 6891 views
Introduction to machine learning an... by Shishir Choudhary 967 views

View on Slideshare

1 of 187 Ad

View on Slideshare

1 of 187 Ad