Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Can Librarians Help
Law Become More
Data Driven ? 
an open question in need of a solution
daniel martin katz
blog | Comput...
A Long History
Innovation in Law
One of the First to Teach
Law with Computers
Helped Launch the Premier
State Wide Legal Aid Website
Helped 3.5 million+ Users
Seek Access to Justice
Guided
Interview
Completed
Document
A2J AUTHOR
www.a2jauthor.org
LOGIC
Us...
Building Upon
Our Tradition
thelawlab.com
#LegalScience
American
Federal
Judiciary
American
Law Professoriate
Building New Algorithms
Large Scale
Judicial Studies
Scientific
Resea...
3D HD Visualization of Supreme
Court Citation Network
Campaign Contributions and
Legislative Ecosystems
Six Degrees
of
Mar...
Scientific
Research
Scientific
Research
Polytechnic
Legal Training
http://www.quantitativemethodsclass.com/Professor Daniel Martin Katz
Intro Class
http://www.legalanalyticscourse.com/Professor Daniel Martin Katz
Professor Michael J. Bommarito II Advanced Class
Building
Ties with the
Legal Industry
TheLawLabChannel.com
FinLegalTechConference.comNovember 4, 2016
Fin(Legal)Tech Conference
finlegaltechconference.com
@Illinois Tech - Chicago Kent College of Law
Fin(Legal)Tech
Conference
October 19 2017
FinLegalTechConference.com
@LawLaboratory
Chicago, IL
Can Librarians Help
Law Become More
Data Driven ? 
an open question in need of a solution
daniel martin katz
blog | Comput...
Today —
a session
in five parts …
A Reset on Robot LawyersI.
The Rise of #LegalAnalyticsII.
The Killer Use Case(s) - Fin (Legal) Tech)III.
The Infrastructur...
A Reset on
Robolawyers
Part I< >
There has been lots of recent
interest in applying
artificial intelligence to law
and there is a bit of confusion
as to where we stand today
and where we are headed
data driven AI rules based AI
Competing Orientations in
Artificial Intelligence
expert
systems
Computational Law
Data Driven Rules Based
prediction
models
and
methods
network
analytic
methods
natural
la...
we see a decent amount of
rules based AI
in legal industry
Three Examples
of Rules Based A.I.
tax
preparation
software
Rules
Based
A.I.
Rules
Based
A.I.
Among other things Neota
has been used to create
decision trees to support
lawyers / non lawyers
What do I do if there has been
An issue in Human Resources ?
A potential FCPA violation?
A potential data breach?
Decision Trees are a step by step
memorialization of best practices
At my home institution -
Illinois Tech Chicago-Kent Law
has a platform that allows
individuals to automate
various legal f...
used by a variety of
legal aid organizations
A2J AUTHOR
www.a2jauthor.org
PROCESS
Guided
Interview
Completed
Document
LOGIC
DECISION TREE
Used over
3.5
Million
times
2.1 Million
Documents generated
IMPACT
Expert Systems 

(together with data) 

will eventually
become Chatbots …
Client Intake
More Seamless Client Interaction
via Tech Platform
Providing Legal Information
to Non-Lawyers in Large
Organ...
so although we see a
decent amount of
rules based AI
in legal industry
I am pretty bearish
on Rules Based A.I. for most
(but not all) applications …
my views are informed by
the history of A.I. in general
lots of issues
with expert systems
and/or
rules based A.I.
(without data or an evolutionary dynamic)
rules based A.I. data driven A.I.
1980’s, 1990’s, Early 2000’s
>
rules based A.I. data driven A.I.
1980’s, 1990’s, Early 2000’s
>
rules based A.I. data driven A.I.
2005 - Present
<
~
Ultimately we are trying to learn
the rules / dynamics that
underlie some class of activity
With that understanding we want to
be able to mimic / predict
There are some use cases
for Rules Based AI /
Expert Systems
Practically
ZERO
Top Tier
Companies
Building
Expert
Systems
expert
systems
Computational Law
Data Driven Rules Based
prediction
models
and
methods
network
analytic
methods
natural
la...
A.I. State of the Art
A.I. State of the Art
purely data centric
A.I. State of the Art
purely data centric
augment expert forecasts w/ data
iterative data < > rules
A.I. State of the Art
purely data centric
augment expert forecasts w/ data
Again -
I like Chatbots because they
end up being a massive data
collection effort …
iterative data < > rules
But as
a general
matter …
In the Rules vs. Data
Debate in A.I.
Data Won
the War
(Terms of Surrender
are Available)
The Rise of #LegalAnalytics
Part II< >
Law is a relatively small vertical
and there is lots of diversity
among tasks lawyers undertake …
Given large fixed costs
infrastructure
+
human capital
(data scientists)
harder to successfully deploy
high quality enterprise
applications for relatively
narrow (sub)verticals
in addition
there is a
borderline
pathological
numerophobia
among lawyers
plus the implicit (explicit)
challenge of partnership as
the dominant form of the
organization within our market
taken together this has
challenged the deployment 

of analytics in legal
Analytics /
Quant Legal Prediction
has come to law
Notwithstanding these head winds—
#LegalAnalytics
Quantitative Legal Prediction
#LegalAnalytics
Quantitative Legal Prediction
#LegalAnalytics
Quantitative Legal Prediction
#LegalAnalytics
Quantitative Legal Prediction
#LegalAnalytics
Quantitative Legal Prediction
#LegalAnalytics
Quantitative Legal Prediction
Some Commercial Applications
In a real sense, this represents
just a narrow set of products
#ContractAnalytics
Quantitative Legal Prediction
#JudicialAnalytics
Quantitative Legal Prediction
#PredictiveCoding #E-Discovery
Quantitative Legal Prediction
General Counsels as Legal
Procurement Specialists
TyMetrix/ELM -
Using $50 billion+ in Legal
Spend Data to Help GC’s
Look ...
#LegalAnalytics
Quantitative Legal Prediction
https://lexsemble.com/
#NegotiationAnalytics
Quantitative Legal Prediction
Here are just a subset of the
tasks that we are trying
to accomplish in legal …
#Predict Case Outcomes
Data Driven Legal Underwriting
#Predict Case Outcomes
Data Driven Legal Underwriting
#Predict Legal Costs
Data Driven Legal Operations
#Predict Relevant Documents
#Predict Case Outcomes
Data Driven Legal Underwriting
Data Driven EDiscovery/Due Diligence
(Pr...
#Predict Relevant Documents
#Predict Case Outcomes
Data Driven Legal Underwriting
Data Driven EDiscovery/Due Diligence
(Pr...
#Predict Relevant Documents
#Predict Case Outcomes
Data Driven Legal Underwriting
Data Driven EDiscovery/Due Diligence
(Pr...
A Deeper Dive
on Predicting
Predicting Case Outcomes
(other problems can be
solved using similar methods)
Supreme Court of United States
#PredictSCOTUS
There are only 3 ways 

to predict something
Experts
Crowds
Algorithms
Experts
Columbia Law Review
October, 2004
Theodore W. Ruger, Pauline T. Kim,
Andrew D. Martin, Kevin M. Quinn
Legal and Political ...
experts
Case Level Prediction
Justice Level Prediction
67.4% experts
58% experts
From the 68
Included
Cases
for the
2002-2003
Supr...
these experts probably
overfit
they fit to the noise
and
not the signal
if this were
finance this
would be
trading
worse than
S&P500
#NoiseTrading
#BuffetChallenge
#BuffetChallenge
like many other forms
human endeavor
law is full of 

noise predictors …
we need to
evaluate
experts and
somehow
benchmark
their
expertise
from a pure
forecasting
standpoint
the best
known
SCOTUS
predictor is
the law
version of
superforecasting
Crowds
crowds
https://fantasyscotus.lexpredict.com/case/list/
We can
generate
Crowd
Sourced
Predictions
Just like the
Market
the Crowd is
collectively
terrible …
< No Alpha >
however,
not all
members of
crowd are
made equal
we maintain
a ‘supercrowd’
which is the top n%
of predictors
up to time t
the
‘supercrowd’
outperforms
the overall
crowd
(and the best
single player)
For the 2015-2016 term
not
enough
crowd
based
decision
making in
institutions
(law included)
“Software developers were asked on
two separate days to estimate the
completion time for a given task, the
hours they proj...
not
enough
crowd
based
decision
making in
institutions
(aka manual
underwriting)
here
is a
commercial
offering
https://lexsemble.com/
Brief Aside
About
Crowd
Sourced
Prediction
#LegalCrowdSourcing
(most pundits did not
identify as a serious
candidate him until
mid-January 2017)
Neil Gorsuch was #1
o n o u r F a n t a ...
#FantasySCOTUS
Algorithms
Our algorithm is a special version
of random forest (time evolving)
http://journals.plos.org/
plosone/article?id=10.1371/
...
From a Pure
Machine Learning Perspective —
Much of this is not novel
EXCEPT the time evolving
element of the
Random Forest
https://github.com/mjbommar/
scotus-predict-v2/
243,882
28,009
Case Outcomes
JusticeVotes
Final Version of #PredictSCOTUS
1816-2015
http://journals.plos.org/plosone/artic...
Final Version of #PredictSCOTUS
1816-2015
case accuracy
70.2%
71.9%
justice accuracy
http://journals.plos.org/plosone/arti...
Experts, Crowds, Algorithms
http://www.sciencemag.org/news/
2017/05/artificial-intelligence-prevails-
predicting-supreme-court-decisions
Professor Katz...
For most problems ...
ensembles of these streams
outperform any single stream
the non-trivial question
is how to optimally assemble
such streams for particular problems
Humans
+
Machines
Humans
or
Machines
>
Here is what we are
working on right now …
expert
forecast
crowd
forecast
learning problem is to discover how to blend streams of intelligence
algorithm
forecast
ens...
expert
forecast
crowd
forecast
learning problem is to discover how to blend streams of intelligence
algorithm
forecast
ens...
By the way, you
might ask why does
one care about
marginal improvements
in prediction ?
#Fin(Legal)Tech
It is a fair question
because in the
private market …
improvements in
performance must
be linked up to an
actual business
...
Fin (Legal) Tech
is the killer use case
Part III< >
Given our ability to offer
forecasts of judicial
outcomes, we wondered
if this information could
inform an event driven
tr...
Revise + Resubmit @
http://arxiv.org/abs/1508.05751
available at
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726
lots of litigation decisions
are just a version of this basic idea
law = finance
from an asset valuation standpoint
lots of litigation decisions
are actually implicit litigation finance
(or self insurance...
Consider for example …
Litigation Reserves Setting Under
FASB ASC 450-20-25
#fin(legal)tech
But there are many
other places where …
law = finance
Fin (Legal) Tech
Three Types of Lawyers
(as described by paul lippe)
play “whack-a-mole”, reacting to
problems by creating fear and
friction within organizations and
the impression that there...
can help clients shape
(perhaps distort)
external perception of risk.
Merely Clever Lawyers
design systems that
balance risk and improve
transparency, helping clients
correctly price risk internally
Great Lawyers
law = finance
(insuranceaswell)
law < > finance
many elements in law look like
finance did 25 - 50 years ago
(on the long road from Black-Scholes to algorit...
Lawyer VALUE PROPOSITION
(From the Client’s Perspective)
(internal or external client)
help price risk /
help reduce information asymmetries
transactional =
litigation =
characterize (predict) risk/exposure
shift the expected value of a lawsuit
help price risk /
help reduce info...
litigation =
characterize (predict) risk/exposure
shift the expected value of a lawsuit
compliance = identify + prevent ro...
litigation =
characterize (predict) risk/exposure
shift the expected value of a lawsuit
compliance = identify + prevent ro...
Dominant Model in Law
expert centered pricing of risk
Dominant Model in Law
lots of unintentional self insurance
rarely (if ever) based upon
explicit risk models
Cult of one 

(or very small # of)
person(s) thinking drives
decisions with serious
financial consequences
Claim:
fin(tech) offers lessons
for many areas
in law
thesis statement:
the financialization
of the law will be
an important vector
of the next decade(s)
#Fin(Legal)Tech
application of those ideas and
technologies to a wide range of
law related spheres including
litigation, t...
Here are
just a few
of many
examples
www.burfordcapital.com/
http://www.gerchenkeller.com/
http://www.fulbrookmanagement.com/
http://www.longfordcapital.com/
h...
Litigation Finance
Litigation Finance
Event Driven Legal Trading
M&A Insurance
Outside of M+A
Requires Mapping of Deal Terms
to actual substantive outcomes
#legaldata
#legalanalytics
Being able to compute the
change in risk as a function
of a change in deal terms
Trading Desk is
all about alpha -
using data,
predictions,
process, etc.
Not about simply buying
tools off the shelf and
deploying them …
The Infrastructure
for Legal Analytics -
#MLaaS and the
Enterprise Open
Source Movement
Part IV< >
Lots of folks ask me what is
next in legal analytics …
A big part of the answer
comes from one of the most
dominant vectors in tech
both those in positions of
leadership and those in technical
positions need to take stock
the democratization of
machine learning is underway
Emerging Business Model -
Machine Learning as a Service
#MLaaS
IBM Watson (per se)
IBM Watson (as early #MLaaS)
vs.
IBM WATSON
First major effort at #MLaaS
Machine Learning as a Service
The
Cloud
Wars
Commercial Examples
Machine Learning as a Service
#MLaaS
Machine Learning as a Service
#MLaaS
Machine
Learning as
a Service
#MLaaS
Machine Learning as a Service
#MLaaS
But wait there is more …
Machine Learning as a Service
#MLaaS
Machine Learning as a Service
#MLaaS
Enterprise Open Source Movement
#OpenSource
+
Enterprise Open Source Movement
#OpenSource
https://techcrunch.com/2016/06/19/the-next-wave-in-software-is-open-adoption-software/
historically one needed to
build the full stack (i.e end to
end) for an application
Standing on 

the Shoulders of Giants
The (Emerging) Last Mile Problem
in (Legal) Analytics
Off the
Shelf
#MLaaS, etc.
(perhaps with some
configuration
and/or
customization)
Unique Domain
Specific Offering
The New Ball Game
Piece together the
combinations of 

#MLaaS + open source
to build enterprise applications
which are unique combinations
drawn from across the
#MLaaS / open source spectrum
First Wave vs.
Second Wave
Legal Tech
Second Movers can
catch up faster …
Second Movers
need less capital …
Second Movers
who start now
will have lower
fixed costs …
Major Implication
The Best Legal Tech
is Yet to Be Built …
We are beginning
to see the first wave
of #MLaaS
Implementation
Companies in
General
https://computationallegalstudies.com/2017/05/07/machine-learning-
service-mlaas-ecosystem-grows-bonsai-mlaas-implentation...
And this is in Part
What My Company
LexPredict
will be (already is)
doing within law …
https://www.slideshare.net/lexpredict/
contraxsuite-why-were-opensourcing-
contraxsuite-and-product-overview
#OpenSourceLe...
"We are increasingly thinking that there's room in
legal tech for a Red Hat in legal — companies that
really focus on deve...
contraxsuite.com
Will you resell the software to third parties?
YES%NO%
How much does ContraxSuite cost?
Will you keep derivative work open...
If you are just
buying tools from
vendors you likely
have no alpha
Building a Legal
Data Strategy
Part V< >
(A Role for Law Librarians?)
every organization in law
needs a data strategy
Capture, Clean, Regularize Data
to support a range of tasks
Deploy Data for Specific
Enterprise Applications
Develop a
data roadmap
What%is%a%data%strategy?%
Statement and Framework
Data Strategy: Defined
! ! ! ! A! data! strategy! combines! a! top2down! mission! statement!
ackno...
MAY–JUNE 2017 ISSUE!
D - I - K - W
From Data Strategy to Wisdom
Data$
Informa+on$
Knowledge$
Wisdom$
Direct'record'of'fact,'
signal,'symbol'
In...
When is Data Valuable?
Even when it’s not
LOW$ HIGH$
HIGH$LOW$
IMPACT'
FREQUENCY'
High3frequency,'high3impact'3'best'use'c...
Some%Organiza-ons%Have%Publically%
Commi8ed%Themselves%to%Use%Data%
to%Become%‘Best%in%Class’%%
Legal%Departments%%
33!
“Now! we! have! program! managers,!
data! analysts,! business! analysts,!
data! scien9sts,! opera9ons!
managers,!I!mea...
36!
“From!se)lement!informa0on!and!
contracts! to! sensi0ve! client! data!
and! beyond,! Liberty! Mutual!
creates! and! st...
34!
37!
“I"believe"strongly"that"data"analy2cs"is"
a"new"fron2er"in"the"legal"space.”"
Susie!Lees!
General!Counsel!!
Allstate!...
Why$a$legal%data$strategy?$
Five reasons to care
Can you answer these questions?
1. How&many&legal&ma.ers&did&you&handle&last&year?&
2. How&much&poten...
47!
Methods for Using (Legal) Data
Historical reporting in legal
Historical analytics in legal
Predictive analytics in leg...
48!
Historical reporting
in legal
Ques'on:+ What! did! we!
spend! on! se.lements! and!
legal!expenses!last!quarter?!
$1.2M...
49!
Ques&on:!What!
factors!drove!
se3lement!
amounts!last!
quarter?!
•  F o r ! l a b o r ! a n d!
employment!disputes,!th...
50!
Ques'on:!Should!we!se,le!this!dispute!at!outset?!
•  The!counterparty!is!expected!to!accept!an!ini8al!offer!
•  The!dis...
37!
Stages!of!Legal!!
Data!Strategy!Maturity!
Chaotic
Managed
Defined
Data-Driven Continuously Improvin
5!1! 2! 3! 4!
51!
(be able to do so without a herculean effort)
1. !Measure,!monitor,!and!manage!your!resources!and!service!providers.!
...
Deploying a Legal
Data Strategy for a
Discrete Problem
Y = βo +/- β1 ( X1 ) +/- β2 ( X2 ) +/- β3 ( X3 ) +/- β4 ( X3 ) +/- β5 ( X3 ) + ε
Y = $151 + $15 ( ) + 161 ( ) + 95 ( ) + 3...
1. Define the Parameter Space
3. Select a Model/Method
4. Validate Out of Sample
2. Collect / Normalize Data
(typically usi...
Work with experts to
define relevant variables
that drive outcomes on
some problem
(experts are strong at identifying
relev...
Figure out how to
collect or normalize
relevant data
Yes
No
f( )
Outcome?
binary
f( )
Outcome?
continuous
machine learning is the approach to
‘learn’ the best performing f ( )...
https://www.slideshare.net/lexpredict/
developing-a-legal-data-strategy-learning-
to-see-data-as-a-strategic-business-asse...
Daniel Martin Katz
@ computational
computationallegalstudies.com
lexpredict.com
danielmartinkatz.com
illinois tech - chica...
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Need of a Solution — Professor Daniel Martin Katz
Upcoming SlideShare
Loading in …5
×

Can Law Librarians Help Law Become More Data Driven ? An Open Question in Need of a Solution — Professor Daniel Martin Katz

1,401 views

Published on

Can Law Librarians Help Law Become More Data Driven ? An Open Question in Need of a Solution — Professor Daniel Martin Katz

Published in: Law
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Can Law Librarians Help Law Become More Data Driven ? An Open Question in Need of a Solution — Professor Daniel Martin Katz

  1. 1. Can Librarians Help Law Become More Data Driven ?  an open question in need of a solution daniel martin katz blog | ComputationalLegalStudies.com corp | LexPredict.com page | DanielMartinKatz.com edu | illinois tech - chicago kent law lab | TheLawLab.com
  2. 2. A Long History Innovation in Law
  3. 3. One of the First to Teach Law with Computers
  4. 4. Helped Launch the Premier State Wide Legal Aid Website
  5. 5. Helped 3.5 million+ Users Seek Access to Justice Guided Interview Completed Document A2J AUTHOR www.a2jauthor.org LOGIC Used over 3.5 Million times 2.1 Million Documents generated IMPACT
  6. 6. Building Upon Our Tradition
  7. 7. thelawlab.com
  8. 8. #LegalScience
  9. 9. American Federal Judiciary American Law Professoriate Building New Algorithms Large Scale Judicial Studies Scientific Research
  10. 10. 3D HD Visualization of Supreme Court Citation Network Campaign Contributions and Legislative Ecosystems Six Degrees of Marbury v. Madison Electronic World Treaty Index Radial SCOTUS Citation Network Scientific Research
  11. 11. Scientific Research
  12. 12. Scientific Research
  13. 13. Polytechnic Legal Training
  14. 14. http://www.quantitativemethodsclass.com/Professor Daniel Martin Katz Intro Class
  15. 15. http://www.legalanalyticscourse.com/Professor Daniel Martin Katz Professor Michael J. Bommarito II Advanced Class
  16. 16. Building Ties with the Legal Industry
  17. 17. TheLawLabChannel.com
  18. 18. FinLegalTechConference.comNovember 4, 2016
  19. 19. Fin(Legal)Tech Conference finlegaltechconference.com @Illinois Tech - Chicago Kent College of Law
  20. 20. Fin(Legal)Tech Conference October 19 2017 FinLegalTechConference.com @LawLaboratory Chicago, IL
  21. 21. Can Librarians Help Law Become More Data Driven ?  an open question in need of a solution daniel martin katz blog | ComputationalLegalStudies.com corp | LexPredict.com page | DanielMartinKatz.com edu | illinois tech - chicago kent law lab | TheLawLab.com
  22. 22. Today — a session in five parts …
  23. 23. A Reset on Robot LawyersI. The Rise of #LegalAnalyticsII. The Killer Use Case(s) - Fin (Legal) Tech)III. The Infrastructure for #LegalAnalyticsIV. Building a Legal Data StrategyV.
  24. 24. A Reset on Robolawyers Part I< >
  25. 25. There has been lots of recent interest in applying artificial intelligence to law
  26. 26. and there is a bit of confusion as to where we stand today and where we are headed
  27. 27. data driven AI rules based AI Competing Orientations in Artificial Intelligence
  28. 28. expert systems Computational Law Data Driven Rules Based prediction models and methods network analytic methods natural language processing self executing law visual law computable codes
  29. 29. we see a decent amount of rules based AI in legal industry
  30. 30. Three Examples of Rules Based A.I.
  31. 31. tax preparation software
  32. 32. Rules Based A.I.
  33. 33. Rules Based A.I.
  34. 34. Among other things Neota has been used to create decision trees to support lawyers / non lawyers
  35. 35. What do I do if there has been An issue in Human Resources ? A potential FCPA violation? A potential data breach?
  36. 36. Decision Trees are a step by step memorialization of best practices
  37. 37. At my home institution - Illinois Tech Chicago-Kent Law has a platform that allows individuals to automate various legal forms, etc.
  38. 38. used by a variety of legal aid organizations
  39. 39. A2J AUTHOR www.a2jauthor.org
  40. 40. PROCESS Guided Interview Completed Document
  41. 41. LOGIC
  42. 42. DECISION TREE
  43. 43. Used over 3.5 Million times 2.1 Million Documents generated IMPACT
  44. 44. Expert Systems 
 (together with data) 
 will eventually become Chatbots …
  45. 45. Client Intake More Seamless Client Interaction via Tech Platform Providing Legal Information to Non-Lawyers in Large Organizations
  46. 46. so although we see a decent amount of rules based AI in legal industry
  47. 47. I am pretty bearish on Rules Based A.I. for most (but not all) applications …
  48. 48. my views are informed by the history of A.I. in general
  49. 49. lots of issues with expert systems and/or rules based A.I. (without data or an evolutionary dynamic)
  50. 50. rules based A.I. data driven A.I. 1980’s, 1990’s, Early 2000’s >
  51. 51. rules based A.I. data driven A.I. 1980’s, 1990’s, Early 2000’s > rules based A.I. data driven A.I. 2005 - Present < ~
  52. 52. Ultimately we are trying to learn the rules / dynamics that underlie some class of activity
  53. 53. With that understanding we want to be able to mimic / predict
  54. 54. There are some use cases for Rules Based AI / Expert Systems
  55. 55. Practically ZERO Top Tier Companies Building Expert Systems
  56. 56. expert systems Computational Law Data Driven Rules Based prediction models and methods network analytic methods natural language processing self executing law visual law computable codes
  57. 57. A.I. State of the Art
  58. 58. A.I. State of the Art purely data centric
  59. 59. A.I. State of the Art purely data centric augment expert forecasts w/ data
  60. 60. iterative data < > rules A.I. State of the Art purely data centric augment expert forecasts w/ data
  61. 61. Again - I like Chatbots because they end up being a massive data collection effort … iterative data < > rules
  62. 62. But as a general matter …
  63. 63. In the Rules vs. Data Debate in A.I.
  64. 64. Data Won the War (Terms of Surrender are Available)
  65. 65. The Rise of #LegalAnalytics Part II< >
  66. 66. Law is a relatively small vertical and there is lots of diversity among tasks lawyers undertake …
  67. 67. Given large fixed costs infrastructure + human capital (data scientists)
  68. 68. harder to successfully deploy high quality enterprise applications for relatively narrow (sub)verticals
  69. 69. in addition there is a borderline pathological numerophobia among lawyers
  70. 70. plus the implicit (explicit) challenge of partnership as the dominant form of the organization within our market
  71. 71. taken together this has challenged the deployment 
 of analytics in legal
  72. 72. Analytics / Quant Legal Prediction has come to law Notwithstanding these head winds—
  73. 73. #LegalAnalytics Quantitative Legal Prediction
  74. 74. #LegalAnalytics Quantitative Legal Prediction
  75. 75. #LegalAnalytics Quantitative Legal Prediction
  76. 76. #LegalAnalytics Quantitative Legal Prediction
  77. 77. #LegalAnalytics Quantitative Legal Prediction
  78. 78. #LegalAnalytics Quantitative Legal Prediction
  79. 79. Some Commercial Applications
  80. 80. In a real sense, this represents just a narrow set of products
  81. 81. #ContractAnalytics Quantitative Legal Prediction
  82. 82. #JudicialAnalytics Quantitative Legal Prediction
  83. 83. #PredictiveCoding #E-Discovery Quantitative Legal Prediction
  84. 84. General Counsels as Legal Procurement Specialists TyMetrix/ELM - Using $50 billion+ in Legal Spend Data to Help GC’s Look for Arbitrage Opportunities, Value Propositions in Hiring Law Firms #LegalSpendAnalytics Quantitative Legal Prediction
  85. 85. #LegalAnalytics Quantitative Legal Prediction https://lexsemble.com/
  86. 86. #NegotiationAnalytics Quantitative Legal Prediction
  87. 87. Here are just a subset of the tasks that we are trying to accomplish in legal …
  88. 88. #Predict Case Outcomes Data Driven Legal Underwriting
  89. 89. #Predict Case Outcomes Data Driven Legal Underwriting #Predict Legal Costs Data Driven Legal Operations
  90. 90. #Predict Relevant Documents #Predict Case Outcomes Data Driven Legal Underwriting Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Legal Costs Data Driven Legal Operations
  91. 91. #Predict Relevant Documents #Predict Case Outcomes Data Driven Legal Underwriting Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Legal Costs #Predict Rogue Behavior Data Driven Legal Operations Data Driven Compliance
  92. 92. #Predict Relevant Documents #Predict Case Outcomes Data Driven Legal Underwriting Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Legal Costs Data Driven Legal Operations Data Driven Compliance #Predict Contract Terms/Outcomes Data Driven Transactional Work #Predict Rogue Behavior
  93. 93. A Deeper Dive on Predicting Predicting Case Outcomes (other problems can be solved using similar methods)
  94. 94. Supreme Court of United States #PredictSCOTUS
  95. 95. There are only 3 ways 
 to predict something Experts Crowds Algorithms
  96. 96. Experts
  97. 97. Columbia Law Review October, 2004 Theodore W. Ruger, Pauline T. Kim, Andrew D. Martin, Kevin M. Quinn Legal and Political Science Approaches to Predicting Supreme Court Decision Making The Supreme Court Forecasting Project:
  98. 98. experts
  99. 99. Case Level Prediction Justice Level Prediction 67.4% experts 58% experts From the 68 Included Cases for the 2002-2003 Supreme Court Term
  100. 100. these experts probably overfit
  101. 101. they fit to the noise and not the signal
  102. 102. if this were finance this would be trading worse than S&P500
  103. 103. #NoiseTrading
  104. 104. #BuffetChallenge
  105. 105. #BuffetChallenge
  106. 106. like many other forms human endeavor law is full of 
 noise predictors …
  107. 107. we need to evaluate experts and somehow benchmark their expertise
  108. 108. from a pure forecasting standpoint
  109. 109. the best known SCOTUS predictor is
  110. 110. the law version of superforecasting
  111. 111. Crowds
  112. 112. crowds
  113. 113. https://fantasyscotus.lexpredict.com/case/list/ We can generate Crowd Sourced Predictions
  114. 114. Just like the Market the Crowd is collectively terrible … < No Alpha >
  115. 115. however, not all members of crowd are made equal
  116. 116. we maintain a ‘supercrowd’ which is the top n% of predictors up to time t
  117. 117. the ‘supercrowd’ outperforms the overall crowd (and the best single player)
  118. 118. For the 2015-2016 term
  119. 119. not enough crowd based decision making in institutions (law included)
  120. 120. “Software developers were asked on two separate days to estimate the completion time for a given task, the hours they projected differed by 71%, on average. W h e n p a t h o l o g i s t s m a d e t wo assessments of the severity of biopsy results, the correlation between their ratings was only .61 (out of a perfect 1.0), indicating that they made inconsistent diagnoses quite frequently. Judgments made by different people are even more likely to diverge.”
  121. 121. not enough crowd based decision making in institutions (aka manual underwriting)
  122. 122. here is a commercial offering
  123. 123. https://lexsemble.com/
  124. 124. Brief Aside About Crowd Sourced Prediction #LegalCrowdSourcing
  125. 125. (most pundits did not identify as a serious candidate him until mid-January 2017) Neil Gorsuch was #1 o n o u r F a n t a s y Platform 12 Days after Donald Trump was elected President (i.e Nov 20)
  126. 126. #FantasySCOTUS
  127. 127. Algorithms
  128. 128. Our algorithm is a special version of random forest (time evolving) http://journals.plos.org/ plosone/article?id=10.1371/ journal.pone.0174698 available at RESEARCH ARTICLE A general approach for predicting the behavior of the Supreme Court of the United States Daniel Martin Katz1,2 *, Michael J. Bommarito II1,2 , Josh Blackman3 1 Illinois Tech - Chicago-Kent College of Law, Chicago, IL, United States of America, 2 CodeX - The Stanford Center for Legal Informatics, Stanford, CA, United States of America, 3 South Texas College of Law Houston, Houston, TX, United States of America * dkatz3@kentlaw.iit.edu Abstract Building on developments in machine learning and prior work in the science of judicial pre- diction, we construct a model designed to predict the behavior of the Supreme Court of the United States in a generalized, out-of-sample context. To do so, we develop a time-evolving random forest classifier that leverages unique feature engineering to predict more than 240,000 justice votes and 28,000 cases outcomes over nearly two centuries (1816-2015). Using only data available prior to decision, our model outperforms null (baseline) models at both the justice and case level under both parametric and non-parametric tests. Over nearly two centuries, we achieve 70.2% accuracy at the case outcome level and 71.9% at the jus- tice vote level. More recently, over the past century, we outperform an in-sample optimized null model by nearly 5%. Our performance is consistent with, and improves on the general level of prediction demonstrated by prior work; however, our model is distinctive because it can be applied out-of-sample to the entire past and future of the Court, not a single term. Our results represent an important advance for the science of quantitative legal prediction and portend a range of other potential applications. Introduction As the leaves begin to fall each October, the first Monday marks the beginning of another term for the Supreme Court of the United States. Each term brings with it a series of challenging, important cases that cover legal questions as diverse as tax law, freedom of speech, patent law, administrative law, equal protection, and environmental law. In many instances, the Court’s decisions are meaningful not just for the litigants per se, but for society as a whole. Unsurprisingly, predicting the behavior of the Court is one of the great pastimes for legal and political observers. Every year, newspapers, television and radio pundits, academic jour- nals, law reviews, magazines, blogs, and tweets predict how the Court will rule in a particular case. Will the Justices vote based on the political preferences of the President who appointed them or form a coalition along other dimensions? Will the Court counter expectations with an unexpected ruling? PLOS ONE | https://doi.org/10.1371/journal.pone.0174698 April 12, 2017 1 / 18 a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 OPEN ACCESS Citation: Katz DM, Bommarito MJ, II, Blackman J (2017) A general approach for predicting the behavior of the Supreme Court of the United States. PLoS ONE 12(4): e0174698. https://doi. org/10.1371/journal.pone.0174698 Editor: Luı´s A. Nunes Amaral, Northwestern University, UNITED STATES Received: January 17, 2017 Accepted: March 13, 2017 Published: April 12, 2017 Copyright: © 2017 Katz et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: Data and replication code are available on Github at the following URL: https://github.com/mjbommar/scotus-predict-v2/. Funding: The author(s) received no specific funding for this work. Competing interests: All Authors are Members of a LexPredict, LLC which provides consulting services to various legal industry stakeholders. We received no financial contributions from LexPredict or anyone else for this paper. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
  129. 129. From a Pure Machine Learning Perspective — Much of this is not novel EXCEPT the time evolving element of the Random Forest
  130. 130. https://github.com/mjbommar/ scotus-predict-v2/
  131. 131. 243,882 28,009 Case Outcomes JusticeVotes Final Version of #PredictSCOTUS 1816-2015 http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0174698
  132. 132. Final Version of #PredictSCOTUS 1816-2015 case accuracy 70.2% 71.9% justice accuracy http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0174698
  133. 133. Experts, Crowds, Algorithms
  134. 134. http://www.sciencemag.org/news/ 2017/05/artificial-intelligence-prevails- predicting-supreme-court-decisions Professor Katz noted that in the long term …“We believe the blend of experts, crowds, and algorithms is the secret sauce for the whole thing.” May 2nd 2017
  135. 135. For most problems ... ensembles of these streams outperform any single stream
  136. 136. the non-trivial question is how to optimally assemble such streams for particular problems
  137. 137. Humans + Machines Humans or Machines >
  138. 138. Here is what we are working on right now …
  139. 139. expert forecast crowd forecast learning problem is to discover how to blend streams of intelligence algorithm forecast ensemble method ensemble model
  140. 140. expert forecast crowd forecast learning problem is to discover how to blend streams of intelligence algorithm forecast ensemble method ensemble model via back testing we can learn the weights to apply for particular problems
  141. 141. By the way, you might ask why does one care about marginal improvements in prediction ? #Fin(Legal)Tech
  142. 142. It is a fair question because in the private market … improvements in performance must be linked up to an actual business model …
  143. 143. Fin (Legal) Tech is the killer use case Part III< >
  144. 144. Given our ability to offer forecasts of judicial outcomes, we wondered if this information could inform an event driven trading strategy ?
  145. 145. Revise + Resubmit @ http://arxiv.org/abs/1508.05751 available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726
  146. 146. lots of litigation decisions are just a version of this basic idea law = finance
  147. 147. from an asset valuation standpoint lots of litigation decisions are actually implicit litigation finance (or self insurance) #fin(legal)tech
  148. 148. Consider for example … Litigation Reserves Setting Under FASB ASC 450-20-25 #fin(legal)tech
  149. 149. But there are many other places where … law = finance
  150. 150. Fin (Legal) Tech
  151. 151. Three Types of Lawyers (as described by paul lippe)
  152. 152. play “whack-a-mole”, reacting to problems by creating fear and friction within organizations and the impression that there is a legal risk around every corner. Mediocre Lawyers
  153. 153. can help clients shape (perhaps distort) external perception of risk. Merely Clever Lawyers
  154. 154. design systems that balance risk and improve transparency, helping clients correctly price risk internally Great Lawyers
  155. 155. law = finance (insuranceaswell)
  156. 156. law < > finance many elements in law look like finance did 25 - 50 years ago (on the long road from Black-Scholes to algorithmic trading)
  157. 157. Lawyer VALUE PROPOSITION (From the Client’s Perspective) (internal or external client)
  158. 158. help price risk / help reduce information asymmetries transactional =
  159. 159. litigation = characterize (predict) risk/exposure shift the expected value of a lawsuit help price risk / help reduce information asymmetries transactional =
  160. 160. litigation = characterize (predict) risk/exposure shift the expected value of a lawsuit compliance = identify + prevent rogue behavior monitor behavior in (near) real time help price risk / help reduce information asymmetries transactional =
  161. 161. litigation = characterize (predict) risk/exposure shift the expected value of a lawsuit compliance = identify + prevent rogue behavior monitor behavior in (near) real time help price risk / help reduce information asymmetries transactional = regulatory = help identify (predict) the decisions of regulators / law makers and the risk associated with various outcomes
  162. 162. Dominant Model in Law expert centered pricing of risk
  163. 163. Dominant Model in Law lots of unintentional self insurance rarely (if ever) based upon explicit risk models
  164. 164. Cult of one 
 (or very small # of) person(s) thinking drives decisions with serious financial consequences
  165. 165. Claim: fin(tech) offers lessons for many areas in law
  166. 166. thesis statement: the financialization of the law will be an important vector of the next decade(s)
  167. 167. #Fin(Legal)Tech application of those ideas and technologies to a wide range of law related spheres including litigation, transactional work and compliance.
  168. 168. Here are just a few of many examples
  169. 169. www.burfordcapital.com/ http://www.gerchenkeller.com/ http://www.fulbrookmanagement.com/ http://www.longfordcapital.com/ http://www.benthamimf.com/ Litigation Finance
  170. 170. Litigation Finance
  171. 171. Litigation Finance
  172. 172. Event Driven Legal Trading
  173. 173. M&A Insurance
  174. 174. Outside of M+A Requires Mapping of Deal Terms to actual substantive outcomes #legaldata #legalanalytics
  175. 175. Being able to compute the change in risk as a function of a change in deal terms
  176. 176. Trading Desk is all about alpha - using data, predictions, process, etc.
  177. 177. Not about simply buying tools off the shelf and deploying them …
  178. 178. The Infrastructure for Legal Analytics - #MLaaS and the Enterprise Open Source Movement Part IV< >
  179. 179. Lots of folks ask me what is next in legal analytics …
  180. 180. A big part of the answer comes from one of the most dominant vectors in tech
  181. 181. both those in positions of leadership and those in technical positions need to take stock
  182. 182. the democratization of machine learning is underway
  183. 183. Emerging Business Model - Machine Learning as a Service #MLaaS
  184. 184. IBM Watson (per se) IBM Watson (as early #MLaaS) vs.
  185. 185. IBM WATSON First major effort at #MLaaS Machine Learning as a Service
  186. 186. The Cloud Wars
  187. 187. Commercial Examples
  188. 188. Machine Learning as a Service #MLaaS
  189. 189. Machine Learning as a Service #MLaaS
  190. 190. Machine Learning as a Service #MLaaS
  191. 191. Machine Learning as a Service #MLaaS
  192. 192. But wait there is more …
  193. 193. Machine Learning as a Service #MLaaS
  194. 194. Machine Learning as a Service #MLaaS Enterprise Open Source Movement #OpenSource +
  195. 195. Enterprise Open Source Movement #OpenSource
  196. 196. https://techcrunch.com/2016/06/19/the-next-wave-in-software-is-open-adoption-software/
  197. 197. historically one needed to build the full stack (i.e end to end) for an application
  198. 198. Standing on 
 the Shoulders of Giants
  199. 199. The (Emerging) Last Mile Problem in (Legal) Analytics
  200. 200. Off the Shelf #MLaaS, etc. (perhaps with some configuration and/or customization) Unique Domain Specific Offering
  201. 201. The New Ball Game
  202. 202. Piece together the combinations of 
 #MLaaS + open source
  203. 203. to build enterprise applications which are unique combinations drawn from across the #MLaaS / open source spectrum
  204. 204. First Wave vs. Second Wave Legal Tech
  205. 205. Second Movers can catch up faster …
  206. 206. Second Movers need less capital …
  207. 207. Second Movers who start now will have lower fixed costs …
  208. 208. Major Implication The Best Legal Tech is Yet to Be Built …
  209. 209. We are beginning to see the first wave of #MLaaS Implementation Companies in General
  210. 210. https://computationallegalstudies.com/2017/05/07/machine-learning- service-mlaas-ecosystem-grows-bonsai-mlaas-implentation-company/ “AI startup Bonsai has raised $7.6 million to grow its platform that simplifies open-source machine learning library TensorFlow to help businesses construct their own artificial intelligence models and incorporate AI into their business.”
  211. 211. And this is in Part What My Company LexPredict will be (already is) doing within law …
  212. 212. https://www.slideshare.net/lexpredict/ contraxsuite-why-were-opensourcing- contraxsuite-and-product-overview #OpenSourceLegal
  213. 213. "We are increasingly thinking that there's room in legal tech for a Red Hat in legal — companies that really focus on development of software by providing wraparound services, but offer their software open source," Michael J Bommarito II said. Michael J. Bommarito Co-Founder CEO @ LexPredict
  214. 214. contraxsuite.com
  215. 215. Will you resell the software to third parties? YES%NO% How much does ContraxSuite cost? Will you keep derivative work open?Free% YES%NO% Free%$12K/year% 50% in trust for open source grants ! 50% for ContraxSuite, LLC!
  216. 216. If you are just buying tools from vendors you likely have no alpha
  217. 217. Building a Legal Data Strategy Part V< > (A Role for Law Librarians?)
  218. 218. every organization in law needs a data strategy
  219. 219. Capture, Clean, Regularize Data to support a range of tasks
  220. 220. Deploy Data for Specific Enterprise Applications Develop a data roadmap
  221. 221. What%is%a%data%strategy?%
  222. 222. Statement and Framework Data Strategy: Defined ! ! ! ! A! data! strategy! combines! a! top2down! mission! statement! acknowledging! the! value! of! an! organiza(on’s+ data! with! a! framework!for!developing!data.driven+capabili(es.+ ! ! ! ! ! While! data! strategies! are! built! on! lists! of! principles! and! technologies,! they! address! much! more:! strategic! communica=on! and! change! management,! process! improvement,!knowledge!management,!and!risk!management,! to!name!a!few.!
  223. 223. MAY–JUNE 2017 ISSUE!
  224. 224. D - I - K - W From Data Strategy to Wisdom Data$ Informa+on$ Knowledge$ Wisdom$ Direct'record'of'fact,' signal,'symbol' Indirect'record'or' descrip6on$ Interpreta6on'of' informa6on$ Ac6onable'inference'or' heuris6c$ Data-Information-Knowledge-Wisdom Data$ Readings'from'a'temperature' sensor'in'Tahoe.$ Informa+on$ The'average'temperature'in'the' month'of'December'is'32.2F.$ Knowledge$ Snow'is'likely'to'accumulate'in' December.' Wisdom$ January'is'a'good'month'to'plan' a'ski'trip'to'Tahoe.$
  225. 225. When is Data Valuable? Even when it’s not LOW$ HIGH$ HIGH$LOW$ IMPACT' FREQUENCY' High3frequency,'high3impact'3'best'use'case'for'data' •  Systema/c'understanding'and'treatment' •  Standardized$reporEng'and'sta/s/cal$treatment' •  PotenEal'for'automaEon'and'predicEon' Example:' •  Labor'&'Employment'for'a'large'employer' •  Patent'Defense'for'a'large'tech'company'
  226. 226. Some%Organiza-ons%Have%Publically% Commi8ed%Themselves%to%Use%Data% to%Become%‘Best%in%Class’%% Legal%Departments%%
  227. 227. 33! “Now! we! have! program! managers,! data! analysts,! business! analysts,! data! scien9sts,! opera9ons! managers,!I!mean,!we!have!a!ton!of! stuff.! That's! the! key! for! me,! is! thinking! about! the! right! people! doing! the! right! tasks.! That's! the! people!part.!And!then!how!they!do! them,! is! the! process,! and! then,! automa9ng! parts,! is! kind! of! that! next,!final!step.!! " And$ all$ of$ that$ is$ underpinned$ by$ d a t a ." Y o u$ c a n ' t$ d o$ a n y$ improvements$ unless$ you$ have$ data.$ You$ can't$ automate$ unless$ you$have$good$data.”!
  228. 228. 36! “From!se)lement!informa0on!and! contracts! to! sensi0ve! client! data! and! beyond,! Liberty! Mutual! creates! and! stores! ever:growing! volumes! of! unorganized! data! across! its! worldwide! offices! and! databases.”! “I've!seen!a!real!transforma0on!in! the! legal! department! just! having! t h a t! i n f o r m a 0 o n! v i s u a l l y! available."! “The' legal' department' is' now' w o r k i n g' p r e d i c 7 v e' a n d' prescrip7ve' analy7cs,"' i.e.' ways' to' analyze' data' that' enable' forecas7ng'for'legal'issues.”'
  229. 229. 34!
  230. 230. 37! “I"believe"strongly"that"data"analy2cs"is" a"new"fron2er"in"the"legal"space.”" Susie!Lees! General!Counsel!! Allstate!! “Leveraging" data," not" only" that" we" possess" but" that" our" law" firms" have" amassed"over"the"years,"offers"a"plethora" of" un<tapped" opportuni=es—not" simply" to" help" us" forecast" and" manage" legal" expenses," but" also" to" help" our" clients" make"more"informed"business"decisions.”"
  231. 231. Why$a$legal%data$strategy?$
  232. 232. Five reasons to care Can you answer these questions? 1. How&many&legal&ma.ers&did&you&handle&last&year?& 2. How&much&poten:al&legal&liability&did&you&handle&last&year?& 3. How&many&hours&per&legal&ma.er&did&you&spend&last&year?& 4. How&many&dollars&per&legal&ma.er&did&you&spend&last&year?& 5. How&much&value&did&you&protect&or&create&last&year?&
  233. 233. 47! Methods for Using (Legal) Data Historical reporting in legal Historical analytics in legal Predictive analytics in legal
  234. 234. 48! Historical reporting in legal Ques'on:+ What! did! we! spend! on! se.lements! and! legal!expenses!last!quarter?! $1.2M+ Ques'on:+ On! average,! how! many!effort!hours!does!staff! counsel! spend! on! the! discovery! phase! of! a! non> compete!dispute?! 25+ hours+
  235. 235. 49! Ques&on:!What! factors!drove! se3lement! amounts!last! quarter?! •  F o r ! l a b o r ! a n d! employment!disputes,!the! length! of! employment! a n d! p r e s e n c e! o f! retaliatory! or! sexual! harassment! claims! are! posi&vely! related! to! se3lement!amount! •  Disputes! origina&ng! in! region!X!have!abnormally! higher! se3lements! than! expected,! given! their! facts! Ques&on:!What! factors!drove! legal!expenses! last!quarter?! •  An! increase! in! ma3ers! in! highCcost! jurisdic&ons! is! posi&vely! related! to! total! legal!expenses! •  A! decrease! in! arbitra&on/ media&on! u&liza&on! is! posi&vely! related! to! total! legal!expenses! Historical analytics in legal
  236. 236. 50! Ques'on:!Should!we!se,le!this!dispute!at!outset?! •  The!counterparty!is!expected!to!accept!an!ini8al!offer! •  The!dispute!is!predicted!to!se,le!for!$100k,!with!legal! expenses!of!$15k! •  If!an!ini8al!offer!is!not!made,!this!dispute!is!expected!to! cost!$50k!in!legal!expenses!and!has!a!25%!chance!of!going! to!jury!trial.! Ques'on:!How!many!effort!hours!will! we!spend!on!this!ma,er?! •  An!es8mate!of!18!hours,!with!90%!confidence!that!the! dispute!will!fall!between!13!and!30!hours! Predictive analytics in legal
  237. 237. 37! Stages!of!Legal!! Data!Strategy!Maturity! Chaotic Managed Defined Data-Driven Continuously Improvin 5!1! 2! 3! 4!
  238. 238. 51! (be able to do so without a herculean effort) 1. !Measure,!monitor,!and!manage!your!resources!and!service!providers.! ! 2. !Using!data!+!experts,!model!and!improve!the!processes!you!execute.! 3. !Allocate!tasks!across!internal/external!resources!and!assess!cost!and!quality.! 4. !Manage!risk!and!be!able!to!formally!characterize!the!risks!avoided.! 5. !Jus&fy)and)explain)performance)to)the)clients.! Five Goals for Every Legal Organization
  239. 239. Deploying a Legal Data Strategy for a Discrete Problem
  240. 240. Y = βo +/- β1 ( X1 ) +/- β2 ( X2 ) +/- β3 ( X3 ) +/- β4 ( X3 ) +/- β5 ( X3 ) + ε Y = $151 + $15 ( ) + 161 ( ) + 95 ( ) + 34 ( ) +/- β5 ( ) + ε Per 100 Lawyers If Tier 1 Market is True Partner Status is True Per 10 Years Practice Area
  241. 241. 1. Define the Parameter Space 3. Select a Model/Method 4. Validate Out of Sample 2. Collect / Normalize Data (typically using experts)
  242. 242. Work with experts to define relevant variables that drive outcomes on some problem (experts are strong at identifying relevant variables but have trouble applying weights)
  243. 243. Figure out how to collect or normalize relevant data
  244. 244. Yes No f( ) Outcome? binary f( ) Outcome? continuous machine learning is the approach to ‘learn’ the best performing f ( ) select a model/method then validated out of sample
  245. 245. https://www.slideshare.net/lexpredict/ developing-a-legal-data-strategy-learning- to-see-data-as-a-strategic-business-asset https://www.slideshare.net/lexpredict/ legal-data-strategy-maturity-assessing- capabilities-and-planning-improvements
  246. 246. Daniel Martin Katz @ computational computationallegalstudies.com lexpredict.com danielmartinkatz.com illinois tech - chicago kent college of law@ thelawlab.com

×