Artificial Intelligence and Law -  A Primer

There has been lots of recent
interest in applying
artiﬁcial intelligence to law

and there is a bit of confusion
as to where we stand today
and where we are headed

A.I. in Law
A Quick Reset
Part I< >

data driven AI rules based AI
Competing Orientations in
Artiﬁcial Intelligence

expert
systems
Computational Law
Data Driven Rules Based
prediction
models
and
methods
network
analytic
methods
natural
language
processing
self
executing
law
visual
law
computable
codes

we see a decent amount of
rules based AI
in legal industry

Three Examples
of Rules Based A.I.

Among other things Neota
has been used to create
decision trees to support
lawyers / non lawyers

What do I do if there has been
An issue in Human Resources ?
A potential FCPA violation?
A potential data breach?

Decision Trees are a step by step
memorialization of best practices

At my home institution -
Illinois Tech Chicago-Kent Law
has a platform that allows
individuals to automate
various legal forms, etc.

used by a variety of
legal aid organizations

JUSTICE GAP
80%Civil legal needs of
low-income people in
the U.S. go unmet
For every 1 person
served in an LSC-
funded program, at
least 1 person is
turned away

LSC TECH
SUMMIT
“to explore the potential of
technology to move the United States
toward providing some form of
effective assistance to 100% of
persons otherwise unable to afford an
attorney for dealing with essential
civil legal needs.”

LSC TECH
SUMMIT
“to explore the potential of
technology to move the United States
toward providing some form of
effective assistance to 100% of
persons otherwise unable to afford an
attorney for dealing with essential
civil legal needs.”
some form of effective
assistance to 100% !

LSC TECH
SUMMIT
Technology leading to greater
access to legal information!

PROCESS
Guided
Interview
Completed
Document

JUSTICE &
TECHNOLOGY
PRACTICUM
STUDENT WORK
!
Fieldwork (e.g. Self-Help
Web Center)
Scope document
Research memo
Storyboard
A2J Guided Interview & HotDocs
template
Final presentation
Professor
Ron Staudt
IIT Chicago-
Kent College
of Law
Engage community partners: legal aid
organizations, courts

A2J AUTHOR
COURSE PROJECT
a2jclinic.classcaster.net

Used over
3.5
Million
times
2.1 Million
Documents generated
IMPACT

Expert Systems  
(together with data)  
will eventually
become Chatbots …

Client Intake
Client Acquisition
More Seamless Client Interaction
via Tech Platform
Providing Legal Information
to Non-Lawyers in Large
Organizations

so although we see a
decent amount of
rules based AI
in legal industry

I am pretty bearish
on Rules Based A.I.
for most applications …

my views are informed by
the history of A.I. in general

lots of issues
with expert systems
and/or
rules based A.I.
(without data or an evolutionary dynamic)

rules based A.I. data driven A.I.
1980’s, 1990’s, Early 2000’s
>

1980’s, 1990’s, Early 2000’s
>
2005 - Present
<
~

Ultimately we are trying to learn
the rules / dynamics that
underlie some class of activity

With that understanding we want to
be able to mimic / predict

There are some use cases
for Rules Based AI /
Expert Systems

Practically
ZERO
Top Tier
Companies
Building
Expert
Systems

A.I. State of the Art
purely data centric

purely data centric
augment expert forecasts w/ data

iterative data < > rules
purely data centric
augment expert forecasts w/ data

For example,
I like Chatbots because they
end up being a massive data
collection effort …
iterative data < > rules

#BigData pan everything
#Hashtag
Part II< >

In the Rules vs. Data
Debate in A.I.

Data Won
the War
(Terms of Surrender
are Available)

#DataScience
are already inﬂuencing
our lives in a variety of
meaningful ways
#BigData #Analytics
#A.I.

To date, the most
successful commercial
applications have massive
returns to scale and aim
for cross societal payoffs…

What is
powering
the
A.I.
revolution?

Increasing
Computing
Power
Decreasing
Data
Storage
Costs

Given large ﬁxed costs
infrastructure
+
human capital
(data scientists)

harder to successfully deploy
high quality enterprise
applications for relatively
narrow (sub)verticals

The Rise of #LegalAnalytics
Part II< >

Law is a relatively small vertical
and there is lots of diversity
among tasks lawyers undertake …

in addition
there is a
borderline
pathological
numerophobia
among lawyers

plus the implicit (explicit)
challenge of partnership as
the dominant form of the
organization within our market

taken together this has
challenged the deployment  
of analytics in legal

Analytics /
Quant Legal Prediction
has come to law
Notwithstanding these head winds—

#LegalAnalytics
Quantitative Legal Prediction

In a real sense, this represents
just a narrow set of products

#ContractAnalytics

#JudicialAnalytics

#PredictiveCoding #E-Discovery

General Counsels as Legal
Procurement Specialists
TyMetrix/ELM -
Using $50 billion+ in Legal
Spend Data to Help GC’s
Look for Arbitrage
Opportunities, Value
Propositions in Hiring Law
Firms
#LegalSpendAnalytics

#LegalAnalytics
https://lexsemble.com/

#NegotiationAnalytics

Here are just a subset of the
tasks that we are trying
to accomplish in legal …

#Predict Case Outcomes
Data Driven Legal Underwriting

#Predict Legal Costs
Data Driven Legal Operations

#Predict Relevant Documents
Data Driven EDiscovery/Due Diligence
(Predictive Coding)

(Predictive Coding)
#Predict Rogue Behavior
Data Driven Compliance

(Predictive Coding)
Data Driven Compliance
#Predict Contract Terms/Outcomes
Data Driven Transactional Work
#Predict Rogue Behavior

The Three Forms of
(Legal) Prediction
Part III< >

A Deeper Dive
on Predicting
Predicting Case Outcomes
(other problems can be
solved using similar methods)

Supreme Court of United States
#PredictSCOTUS

There are only 3 ways  
to predict something
Experts
Crowds
Algorithms

Columbia Law Review
October, 2004
Theodore W. Ruger, Pauline T. Kim,
Andrew D. Martin, Kevin M. Quinn
Legal and Political Science
Approaches to Predicting
Supreme Court Decision
Making
The Supreme Court
Forecasting Project:

Case Level Prediction
Justice Level Prediction
67.4% experts
58% experts
From the 68
Included
Cases
for the
2002-2003
Supreme
Court Term

these experts probably
overﬁt

they ﬁt to the noise
and
not the signal

if this were
ﬁnance this
would be
trading
worse than
S&P500

like many other forms
human endeavor
law is full of  
noise predictors …

we need to
evaluate
experts and
somehow
benchmark
their
expertise

from a pure
forecasting
standpoint

the best
known
SCOTUS
predictor is

the law
version of
superforecasting

https://fantasyscotus.lexpredict.com/case/list/
We can
generate
Crowd
Sourced
Predictions

Just like the
Market
the Crowd is
collectively
terrible …
< No Alpha >

however,
not all
members of
crowd are
made equal

we maintain
a ‘supercrowd’
which is the top n%
of predictors
up to time t

the
‘supercrowd’
outperforms
the overall
crowd
(and the best
single player)

not
enough
crowd
based
decision
making in
institutions
(law included)

“Software developers were asked on
two separate days to estimate the
completion time for a given task, the
hours they projected differed by 71%,
on average.
W h e n p a t h o l o g i s t s m a d e t wo
assessments of the severity of biopsy
results, the correlation between their
ratings was only .61 (out of a perfect
1.0), indicating that they made
inconsistent diagnoses quite frequently.
Judgments made by different people
are even more likely to diverge.”

not
enough
crowd
based
decision
making in
institutions

design
to
unlock
untapped
expertise
in
organizations

Brief Aside
About
Crowd
Sourced
Prediction
#LegalCrowdSourcing

(most pundits did not
identify as a serious
candidate him until
mid-January 2017)
Neil Gorsuch was #1
o n o u r F a n t a s y
Platform 12 Days after
Donald Trump was
elected President
(i.e Nov 20)

we have developed an
algorithm that we call
{Marshall}+
random forest

Ruger, et al (2004)
relied upon
Brieman(1984)
(as partially shown below)

Leo Brieman moved away from
CART in Brieman (2001)

Breiman, L.(2001). Random forests.
Machine learning, 45(1), 5-32.
Published in Machine Learning
(A Springer Science Journal)

One well-known problem with
standard classiﬁcation trees is
their tendency toward overﬁtting

http://machinelearning202.pbworks.com/w/ﬁle/fetch/37597425/
performanceCompSupervisedLearning-caruana.pdf
Random
Forest
(particularly
with special
conﬁg/
optimization)
have proven to
be unreasonably
effective

Random forest is an approach to
aggregate weak learners into
collective strong learners
(using a combo of bagging and random substrates)
(think of it as crowd sourcing of models)

Our algorithm is a special version
of random forest (time evolving)
http://journals.plos.org/
plosone/article?id=10.1371/
journal.pone.0174698
available at
RESEARCH ARTICLE
A general approach for predicting the
behavior of the Supreme Court of the United
States
Daniel Martin Katz1,2
*, Michael J. Bommarito II1,2
, Josh Blackman3
1 Illinois Tech - Chicago-Kent College of Law, Chicago, IL, United States of America, 2 CodeX - The Stanford
Center for Legal Informatics, Stanford, CA, United States of America, 3 South Texas College of Law Houston,
Houston, TX, United States of America
* dkatz3@kentlaw.iit.edu
Abstract
Building on developments in machine learning and prior work in the science of judicial pre-
diction, we construct a model designed to predict the behavior of the Supreme Court of the
United States in a generalized, out-of-sample context. To do so, we develop a time-evolving
random forest classifier that leverages unique feature engineering to predict more than
240,000 justice votes and 28,000 cases outcomes over nearly two centuries (1816-2015).
Using only data available prior to decision, our model outperforms null (baseline) models at
both the justice and case level under both parametric and non-parametric tests. Over nearly
two centuries, we achieve 70.2% accuracy at the case outcome level and 71.9% at the jus-
tice vote level. More recently, over the past century, we outperform an in-sample optimized
null model by nearly 5%. Our performance is consistent with, and improves on the general
level of prediction demonstrated by prior work; however, our model is distinctive because it
can be applied out-of-sample to the entire past and future of the Court, not a single term.
Our results represent an important advance for the science of quantitative legal prediction
and portend a range of other potential applications.
Introduction
As the leaves begin to fall each October, the first Monday marks the beginning of another term
for the Supreme Court of the United States. Each term brings with it a series of challenging,
important cases that cover legal questions as diverse as tax law, freedom of speech, patent law,
administrative law, equal protection, and environmental law. In many instances, the Court’s
decisions are meaningful not just for the litigants per se, but for society as a whole.
Unsurprisingly, predicting the behavior of the Court is one of the great pastimes for legal
and political observers. Every year, newspapers, television and radio pundits, academic jour-
nals, law reviews, magazines, blogs, and tweets predict how the Court will rule in a particular
case. Will the Justices vote based on the political preferences of the President who appointed
them or form a coalition along other dimensions? Will the Court counter expectations with an
unexpected ruling?
PLOS ONE | https://doi.org/10.1371/journal.pone.0174698 April 12, 2017 1 / 18
a1111111111
a1111111111
a1111111111
a1111111111
a1111111111
OPEN ACCESS
Citation: Katz DM, Bommarito MJ, II, Blackman J
(2017) A general approach for predicting the
behavior of the Supreme Court of the United
States. PLoS ONE 12(4): e0174698. https://doi.
org/10.1371/journal.pone.0174698
Editor: Luı´s A. Nunes Amaral, Northwestern
University, UNITED STATES
Received: January 17, 2017
Accepted: March 13, 2017
Published: April 12, 2017
Copyright: © 2017 Katz et al. This is an open
access article distributed under the terms of the
Creative Commons Attribution License, which
permits unrestricted use, distribution, and
reproduction in any medium, provided the original
author and source are credited.
Data Availability Statement: Data and replication
code are available on Github at the following URL:
https://github.com/mjbommar/scotus-predict-v2/.
Funding: The author(s) received no specific
funding for this work.
Competing interests: All Authors are Members of
a LexPredict, LLC which provides consulting
services to various legal industry stakeholders. We
received no financial contributions from LexPredict
or anyone else for this paper. This does not alter
our adherence to PLOS ONE policies on sharing
data and materials.

From a Pure
Machine Learning Perspective —
Much of this is not novel
EXCEPT the time evolving
element of the
Random Forest

https://github.com/mjbommar/
scotus-predict-v2/

243,882
28,009
Case Outcomes
JusticeVotes
Final Version of #PredictSCOTUS
1816-2015
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0174698

Final Version of #PredictSCOTUS
1816-2015
case accuracy
70.2%
71.9%
justice accuracy
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0174698

http://www.sciencemag.org/news/
2017/05/artiﬁcial-intelligence-prevails-
predicting-supreme-court-decisions
Professor Katz noted
that in the long term
…“We believe the
blend of experts,
crowds, and
algorithms is the
secret sauce for the
whole thing.”
May 2nd 2017

For most problems ...
ensembles of these streams
outperform any single stream

the non-trivial question
is how to optimally assemble
such streams for particular problems

Humans
+
Machines
Humans
or
Machines
>

Here is what we are
working on right now …

expert
forecast
crowd
forecast
learning problem is to discover how to blend streams of intelligence
algorithm
forecast
ensemble method
ensemble model

expert
forecast
crowd
forecast
learning problem is to discover how to blend streams of intelligence
algorithm
forecast
ensemble method
ensemble model
via back testing we can learn the weights
to apply for particular problems

By the way, you
might ask why does
one care about
marginal improvements
in prediction ?
#Fin(Legal)Tech

Given our ability to offer
forecasts of judicial
outcomes, we wondered
if this information could
inform an event driven
trading strategy ?

Revise + Resubmit @
http://arxiv.org/abs/1508.05751
available at
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726

The Killer App
is Financialization
#Fin(Legal)Tech

Once you can
engage in
#LegalPrediction,
that immediately
leads to thoughts
about ﬁnancialization
#Fin(Legal)Tech

https://computationallegalstudies.com/2016/02/27/ﬁn-legal-tech-laws-
future-from-ﬁnances-past-an-expanded-version-of-the-deck/
Here is a full presentation
of this idea … (it has overlap)

Claim:
ﬁn(tech)
offers
lessons
for many
areas
in law

FinLegalTechConference.comNovember 4, 2016

Fin(Legal)Tech Conference
finlegaltechconference.com
@Illinois Tech - Chicago Kent College of Law

With that background  
I would like to look over
the #LegalHorizon

Lots of folks ask me what is
next in legal analytics …

A big part of the answer
comes from one of the most
dominant vectors in tech

both those in positions of
leadership and those in technical
positions need to take stock

#MLaaS and the
Enterprise Open
Source Movement
Part IV< >

IBM WATSON
First major effort at #MLaaS
Machine Learning as a Service

IBM Watson is MLaaS and it
would have purported to be
among the biggest stories in
tech over the past few years

Turns out things would layout in
a slightly different fashion …

IBM Watson (per se)
IBM Watson (as early #MLaaS)
vs.

the democratization of
machine learning is underway

Emerging Business Model -
#MLaaS

#MLaaS

Machine
Learning as
a Service
#MLaaS

#MLaaS
Enterprise Open Source Movement
#OpenSource
+

Enterprise Open Source Movement
#OpenSource

https://techcrunch.com/2016/06/19/the-next-wave-in-software-is-open-adoption-software/

Part V< >
The Last Mile
Problem and the
New Dimension of
Competition

historically one needed to
build the full stack (i.e end to
end) for an application

Standing on  
the Shoulders of Giants

The (Emerging) Last Mile Problem
in (Legal) Analytics

Off the
Shelf
#MLaaS, etc.
(perhaps with some
conﬁguration
and/or
customization)
Unique Domain
Speciﬁc Offering

MLaas + Open Source
Decreases Cost of Production
Lowers the Cost of Protoyping

Workﬂow Across
the Machine
Learning Landscape

Piece together the
combinations of  
#MLaaS + open source

to build enterprise applications
which are unique combinations
drawn from across the
#MLaaS / open source spectrum

We are beginning
to see the ﬁrst wave
of #MLaaS
Implementation
Companies …

https://computationallegalstudies.com/2017/05/07/machine-learning-
service-mlaas-ecosystem-grows-bonsai-mlaas-implentation-company/
“AI startup Bonsai has raised $7.6
million to grow its platform that
simpliﬁes open-source machine
learning library TensorFlow to
help businesses construct their
own artiﬁcial intelligence models
and incorporate AI into their
business.”

Three Implications for
#LegalAnalytics
#LegalTech
#LegalAI
Part VI< >

every organization in law
needs a data strategy

Capture, Clean, Regularize Data
to support a range of tasks

Deploy Data for Speciﬁc
Enterprise Applications
Develop a
data roadmap

every organization in law
needs relevant human capital
#LegalAnalytics

Opening the Human
Capital Bottleneck

Probably going to
need homegrow
your own talent

http://www.quantitativemethodsclass.com/Professor Daniel Martin Katz
Intro Class

http://www.legalanalyticscourse.com/Professor Daniel Martin Katz
Professor Michael J. Bommarito II Advanced Class

First Wave vs.
Second Wave
Legal Tech

Second Movers can
catch up faster …

Second Movers
need less capital …

Second Movers
who start now
will have lower
ﬁxed costs …

probably will not
need to go for a
series z round of
funding

Major Implication
The Best Legal Tech
is Yet Be Built …

Prediction
on the #LegalHorizon

Prediction
More Legal Tech
More Legal Analytics

#MLaaS

ComputationalLegalStudies.com
BLOG

Daniel Martin Katz
@ computational
computationallegalstudies.com
lexpredict.com
danielmartinkatz.com
illinois tech - chicago kent college of law@
thelawlab.com

Artificial Intelligence and Law - A Primer

More Related Content

What's hot

Viewers also liked

Similar to Artificial Intelligence and Law - A Primer

More from Daniel Katz

Recently uploaded

In this document