Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
measure twice, cut once
Solving the Legal Profession's Biggest Problems Together
daniel martin katz
blog | ComputationalLe...
collectively this industry
faces some real challenges …
challenges that have been 

well documented …
But I believe that
we are going to persist
indeed, I think we can thrive…
my resolution is
not related to the notion that
the world owes us anything
But rather it is related
to this group and groups like it
working together help solve 

the Legal Profession's
Biggest Problems
But we have
real work to do
So today I would like to
focus my comments …
on how we together might build
a more perfect supply chain
financially rigorous
measurement of the value
proposition associated
with various legal services
centered upon
moving items from the
‘art’ column and to the
‘science’ column …
so today 

a presentation
in five parts …
the economics of law
the industrialization
of the artisan
toward an enterprise
data strategy in legal
fin (legal) tech
Lega...
the economics of law
Part One
I would like to take a step back
When we look at the industry…
under alternative conditions
its structure might have differed
there are fundamental economic
principles which have yielded
the current industrial organization 

of the legal industry
why do we have lawyers?
(in other words what do they solve for …)
help navigate complexity
manage enterprise (legal) risk
+
Social, Economic and
Political Complexity
Which for our
purposes manifests
in legal complexity
In the face of ever
growing legal complexity
we have applied greater
and greater numbers of
human experts to solve
the und...
Lawyer as Complexity Engineer
complexity keeps growing ...
and so has total expenditures
on legal services
Legal Expenditures as
a function of GDP
(some disagreement between these
plots but they project a similar trend)
Cobb Douglas is
the traditional way
to describe a
production
function
LaborCapital
Cobb Douglas is
the traditional way
to describe a
production
function
Capital
Cobb Douglas is
the traditional way
to describe a
production
function
Labor
historically we have
turned this dial
Where is are large
scale complexity filled
opportunities in law?
Where is are large
scale complexity filled
opportunities in law?
BANKS AS CLIENTS
(and TECH AS CLIENTS)
manage enterprise (legal) risk
Three Types of Lawyers
(as described by paul lippe)
play “whack-a-mole”, reacting to
problems by creating fear and
friction within organizations and
the impression that there...
can help clients shape
(perhaps distort)
external perception of risk.
Merely Clever Lawyers
design systems that
balance risk and improve
transparency, helping clients
correctly price risk internally
Great Lawyers
when it comes to risk …
one challenge with identifying
their value proposition
is the counterfactual
why do we have law firms?
(in other words what do they solve for …)
The Enterprise
Consumer (client)
always faces the
decision of 

make versus buy
Solving the Peak
Load Problem
Provide High Value but
Rarely Utilized or Hard
to Acquire Expertise
Without sufficient volume
it is not sensible to make
but rather to buy …
lawyers and law firms
provide substantial value
BUT the
problem of
agency costs
always looms
an economic concept concerning the fee to a
“principal” (an organization, person or group of
persons), when the principal ...
agency costs
turn
allies (friends)
into
frenemies
frenemy
agency costs muddy
the inside vs outside counsel
relationship
(Partially) solving the industry’s
requires engineering around
these agency issues?
Measurement,
Standards,
Metrics,
#DATA
this allows for a partial
solution to the problem
Many at this conference
and at conferences like this
are working on the problem
there are successes from other
sectors of the economy from
whom we can learn
where the supply chain thrived
in a metrics heavy environment…
the industrialization
of the artisan
Part Two
across the economy there are
many effort to convert an
artisanal process into an
industrial process
as we move toward a more metrics
centered field we want to ensure
that we can maintain the artisan
elements that DO ADD VAL...
My favorite non-legal example
Sal Consiglio - (Sally’s APizza) Domino’s Ad Circa 1990’s
ARTISAN INDUSTRIAL
the industrialization of the artisan
lets focus on these two because
for now this is where 

process improvement and 

data should be directed
so …
remaining mindful
of the lawyer and law firm
value proposition
but with an eye toward reducing
the agency costs issues …
every organization in law
needs a data strategy
Toward an Enterprise
Data Strategy in Legal
Part Three
every organization in law
needs a data strategy
Capture, Clean, Regularize Data
to support a range of tasks
Deploy Data for Specific
Enterprise Applications
Develop a
data roadmap
We want data to help support
two major things …
We want data to help support
two major things …
substantive
predictions
procedural
predictions
there is often a significant
spread between
Kim Craig @ Seyfarth Lean Consulting
Chicago Legal Technology + Innovation MEET...
so think of the process map as a
first order estimation of your
actual processes
rich / granular data
can help illuminate
the actual processes
present in various
(legal) organizations
for each node in a process
we want to be able to render
a prediction about things
such as duration,cost, etc.
each unit of time linked + logged
to a node on the process map
if there is not a node than it can
be added
and thus the map becomes more
reflective of reality
just be careful not to create a
#ridiculogram
with predictions about
individual nodes
we can then sum to generate
predictions about the
distributional moments of an
overall matter (or phase)
(i.e. mean, varia...
this matter should take …
between 9-15 months
in 85% of the similar matters
(what about the long tail?)
this matter will cost…
most common range 275k - 345k
but the second mode is 555k - 625k
(and that second
mode typically is...
#LegalData Collaboration Point
transparency
as the relationship glue
(and trust that comes with transparency)
how could you facilitate
data sharing / transparency?
sharing data between
customer and client
(real time, no filter?)
are law firms
AND
corporate counsel
willing to engage in a 

two way data exchange ?
We want data to help support
two major things …
substantive
predictions
procedural
predictions
Here are just a subset of the
substantive predictions we are
trying to undertake in legal …
#Predict Relevant Documents
Data Driven EDiscovery/Due Diligence
(Predictive Coding)
SUBSTANTIVE LEGAL PREDICTIONS
#Predict Relevant Documents
Data Driven EDiscovery/Due Diligence
(Predictive Coding)
#Predict Contract Terms/Outcomes
Data...
#Predict Relevant Documents
Data Driven EDiscovery/Due Diligence
(Predictive Coding)
#Predict Rouge Behavior
Data Driven C...
#Predict Relevant Documents
#Predict Case Outcomes
Data Driven Legal Underwriting
Data Driven EDiscovery/Due Diligence
(Pr...
Data Driven Transactional Work
Meet Bob
Meet Bob
lawyer on
a major
corporate
transaction
Meet Bob Bob is about to
engage in yet
another round of
markup on deal terms
lawyer on
a major
corporate
transaction
Meet Bob Bob is about to
engage in yet
another round of
markup on deal terms
this round is likely to
generate a delay on
t...
how much value is created
by these modifications?
how much delay
will be introduced?
vs.
Need a better understanding
of the actual drivers of risk
Being able to compute the
change in risk as a function
of a change in deal terms
Requires Mapping of Deal Terms
to actual substantive outcomes
#legaldata
#legalanalytics
this is particularly important
when non-lawyers are
doing the negotiation
(for example your global sales force)
Data Driven Compliance
80%+ of the world’s data
is unstructured data
Solution is to either let
tech or human process that data
And humans are actually pretty
good pattern detectors
But only for
certain types of problems
Trading (HFT in particular)
is about looking for anomalies
the discovery + compliance convergence
a hard #bigdata problem in law
(near real time) compliance
FCPA, Product Defect, etc.
the goal is
near real time monitoring
defect w/5 ‘airbag’
version 1.0
backdate w/5 ‘option’
etc.
near real time monitoring of
version 2.0
a massive volume of communications
Corp Security Beginning to
mirror today’s NSA
Behavior will change
(i.e. rogue action will be done offline)
Corp Security Beginning to
mirror today’s NSA
Behavior will change
But Behavior Change will lag
(i.e. rogue action will be done offline)
(i.e. folks will craft incrimina...
thus, discovery (in part)
becomes compliance and some
(only some) litigation is avoided
legal standards will still shift
r...
#Predict Case Outcomes
Data Driven Legal Underwriting
A Deeper Dive
on Predicting
Predicting Case Outcomes
(other problems can be
solved using similar methods)
Supreme Court of United States
#PredictSCOTUS
There are only 3 ways 

to predict something
Experts
Crowds
Algorithms
Experts
Columbia Law Review
October, 2004
Theodore W. Ruger, Pauline T. Kim,
Andrew D. Martin, Kevin M. Quinn
Legal and Political ...
experts
Case Level Prediction
Justice Level Prediction
67.4% experts
58% experts
From the 68
Included
Cases
for the
2002-2003
Supr...
these experts probably
overfit
they fit to the noise
and
not the signal
we need to
evaluate
experts and
somehow
benchmark
their
expertise
from a pure
forecasting
standpoint
the best
known
SCOTUS
predictor is
the law
version of
superforecasting
Crowds
crowds
https://fantasyscotus.lexpredict.com/case/list/
We can
generate
Crowd
Sourced
Predictions
however,
not all
members of
crowd are
made equal
we maintain
a ‘supercrowd’
which is the top n%
of predictors
up to time t
the
‘supercrowd’
outperforms
the overall
crowd
(and the best
single player)
not
enough
crowd
based
decision
making in
institutions
“Software developers were asked on two
separate days to estimate the completion
time for a given task, the hours they
proj...
in law
here
is a
commercial
offering
design
to
unlock
untapped
expertise
in
organizations
Allowing
for
Frictionless
Crowdsourcing
#ManualUnderwriting
https://lexsemble.com/
https://lexsemble.com/
Algorithms
Black
Reed
Frankfurter
Douglas
Jackson
Burton
Clark
Minton
Warren
Harlan
Brennan
Whittaker
Stewart
White
Goldberg
Fortas
M...
we have developed an
algorithm that we call
{Marshall}+
random forest
Benchmarking
since 1953
+
Using only data
available prior to
the decision
Mean Court Direction [FE]
Mean Court Direction 1...
Total Cases Predicted
Total Votes Predicted
7,700
68,964
Justice Prediction
Case Prediction
70.9% accuracy
69.6% accuracy
From 1953 - 2014
Our algorithm is a special version
of random forest
Black
Reed
Frankfurter
Douglas
Jackson
Burton
Clark
Minton
Warren
Harl...
Experts, Crowds, Algorithms
For most problems ...
ensembles of these streams
outperform any single stream
Humans
+
Machines
Humans
+
Machines
>
Humans
+
Machines
Humans
or
Machines
>
Ensembles come in
various forms
Here is a well known example
Poll Aggregation is one form of
ensemble where the learning question is
to determine how much weight (if any)
to assign to...
poll weighting
A Visual Depiction of
How to build an
ensemble method in our
judicial prediction example
expert crowd algorithm
ensemble method
learning problem is to discover when to use a given stream of intelligence
expert crowd algorithm
via back testing we can learn the
weights to apply for particular problems
ensemble method
learning...
Legal Analytics +
#MLaaS
Act Four
Given large fixed costs
infrastructure
+
human capital
(data scientists)
Historically speaking
harder to successfully deploy
high quality enterprise
applications for relatively
narrow (sub)verticals
Law is a relatively small vertical
and there is lots of diversity
among tasks lawyers undertake …
in addition
there is a
borderline
pathological
numerophobia
among lawyers
Analytics /
Quant Legal Prediction
has come to law
Notwithstanding these head winds—
I predict some very
interesting economic
forces will impact the
#legalanalytics space
And applications
are about to get far
cheaper to develop
Emerging Business Model -
Machine Learning as a Service
#MLaaS
The
Cloud
Wars
Commercial Examples
Machine
Learning as
a Service
#MLaaS
Machine Learning as a Service
#MLaaS
Machine Learning as a Service
#MLaaS
Machine Learning as a Service
#MLaaS
historically one needed to
build the full stack (i.e end to
end) for an application
Standing on 

the Shoulders of Giants
The (Emerging) Last Mile Problem
in (Legal) Analytics
Off the
Shelf
#MLaaS, etc.
(perhaps with some
configuration
and/or
customization)
Unique Domain
Specific Offering
MLaas + Open Source
Decreases Cost of Production
Lowers the Cost of Protoyping
Fin (Legal) Tech
Part Five
Today I have encouraged
collaboration
and one reason is
that both sides
(firms / clients) can
unlock more
enterprise value
by working together
because
past is merely prelude
because the biggest change in legal
not robots
because the biggest change in legal
financialization
not robots
developing a data strategy
developing a data strategy
leveraging #MLaaS
developing a data strategy
leveraging #MLaaS
we get better at predicting
developing a data strategy
leveraging #MLaaS
we get better at predicting
which opens the door for…
#FinTech
#FinTech
removing
socially
meaningless
frictions
characterizing
(pricing)
increasingly exotic
forms of risk
#Fin(Legal)Tech
application of those ideas and
technology to a wide range of
law related spheres including
litigation, tra...
if we can predict we can
develop insurance
if we can predict we can
develop insurance
if we can predict
we can develop trading strategies
if we can predict we can
develop insurance
if we can predict
we can develop trading strategies
if we can predict
we have a...
Just a Few Examples of
Fin(Legal)Tech
#fin(legal)tech
pricing
it is *not* predicting cost of
this particular matter where n=1
correctly characterize the
distributional properties of
a portfolio of matters
both + and -
including identification of outliers
apply portfolio theory
to take n=1
and
scale to n=many
#fin(legal)tech
#self insurance
today this is how you
would run a more
rigorous version of
tomorrow?
learn from legal ops service
offering to build a commercial
insurance product offering
legal cost insurance ?
ot...
AIG to Launch Data-
Driven Legal Ops
Business in 2016
https://bol.bna.com/aig-to-
launch-data-driven-legal-
ops-business-i...
#fin(legal)tech
In such a world,
Law Firm is *not* interfacing
with client but rather insurance
company regarding fees
Earlier I discussed the
application of
experts, crowds +
algorithms
as applied
to predicting
case outcomes
that was an example
of manual underwriting
Given our ability to offer
forecasts of judicial
outcomes, we wondered
if this information could
inform an event driven
tr...
Paper Released
August 24, 2015
http://arxiv.org/abs/1508.05751
available at
http://papers.ssrn.com/sol3/papers.cfm?abstrac...
lots of litigation decisions
are just a version of this basic idea
law = finance
lots of litigation decisions
are actually implicit litigation finance
(or self insurance)
#fin(legal)tech
But most implicit litigation
finance is not based upon 

rigorous underwriting …
law =! finance
(but it will)
http://www.slideshare.net/
Danielkatz/fin-legal-tech-laws-
future-from-finances-past-
professors-daniel-martin-katz-
michael...
TheLawLab.com
FinLegalTechConference.comNovember 4, 2016
In sum, I believe …
Over the coming years,
we are going to be able
financialize large elements
of the legal industry
By which I mean —-
apply the tools of finance
and insurance to measure /
predict a wide range of
procedural + substantive
o...
it will help better establish
the value proposition
associated with a wide
range of legal tasks …
As we move items from
the ‘art’ column to the
‘science’ column …
There will be impacts on
the industrial organization
of the legal industry
But what remains
there after will be a
better industry …
it will help focus every individual
and every organization on the
places where they actually provide
a return on investmen...
Daniel Martin Katz
@ computational
computationallegalstudies.com
lexpredict.com
danielmartinkatz.com
illinois tech - chica...
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz
Upcoming SlideShare
Loading in …5
×

Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz

3,957 views

Published on

Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz

Published in: Law

Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Together - Professor Daniel Martin Katz

  1. 1. measure twice, cut once Solving the Legal Profession's Biggest Problems Together daniel martin katz blog | ComputationalLegalStudies.com corp | LexPredict.com page | DanielMartinKatz.com edu | chicago kent college of law
  2. 2. collectively this industry faces some real challenges …
  3. 3. challenges that have been 
 well documented …
  4. 4. But I believe that we are going to persist
  5. 5. indeed, I think we can thrive…
  6. 6. my resolution is not related to the notion that the world owes us anything
  7. 7. But rather it is related to this group and groups like it
  8. 8. working together help solve 
 the Legal Profession's Biggest Problems
  9. 9. But we have real work to do
  10. 10. So today I would like to focus my comments …
  11. 11. on how we together might build a more perfect supply chain
  12. 12. financially rigorous measurement of the value proposition associated with various legal services centered upon
  13. 13. moving items from the ‘art’ column and to the ‘science’ column …
  14. 14. so today 
 a presentation in five parts …
  15. 15. the economics of law the industrialization of the artisan toward an enterprise data strategy in legal fin (legal) tech Legal Analytics + #MLaaS part 1: part 2: part 3: part 4: part 5:
  16. 16. the economics of law Part One
  17. 17. I would like to take a step back
  18. 18. When we look at the industry…
  19. 19. under alternative conditions its structure might have differed
  20. 20. there are fundamental economic principles which have yielded
  21. 21. the current industrial organization 
 of the legal industry
  22. 22. why do we have lawyers? (in other words what do they solve for …)
  23. 23. help navigate complexity manage enterprise (legal) risk +
  24. 24. Social, Economic and Political Complexity
  25. 25. Which for our purposes manifests in legal complexity
  26. 26. In the face of ever growing legal complexity we have applied greater and greater numbers of human experts to solve the underlying problem
  27. 27. Lawyer as Complexity Engineer
  28. 28. complexity keeps growing ...
  29. 29. and so has total expenditures on legal services
  30. 30. Legal Expenditures as a function of GDP (some disagreement between these plots but they project a similar trend)
  31. 31. Cobb Douglas is the traditional way to describe a production function
  32. 32. LaborCapital Cobb Douglas is the traditional way to describe a production function
  33. 33. Capital Cobb Douglas is the traditional way to describe a production function Labor historically we have turned this dial
  34. 34. Where is are large scale complexity filled opportunities in law?
  35. 35. Where is are large scale complexity filled opportunities in law? BANKS AS CLIENTS (and TECH AS CLIENTS)
  36. 36. manage enterprise (legal) risk
  37. 37. Three Types of Lawyers (as described by paul lippe)
  38. 38. play “whack-a-mole”, reacting to problems by creating fear and friction within organizations and the impression that there is a legal risk around every corner. Mediocre Lawyers
  39. 39. can help clients shape (perhaps distort) external perception of risk. Merely Clever Lawyers
  40. 40. design systems that balance risk and improve transparency, helping clients correctly price risk internally Great Lawyers
  41. 41. when it comes to risk … one challenge with identifying their value proposition is the counterfactual
  42. 42. why do we have law firms? (in other words what do they solve for …)
  43. 43. The Enterprise Consumer (client) always faces the decision of 
 make versus buy
  44. 44. Solving the Peak Load Problem
  45. 45. Provide High Value but Rarely Utilized or Hard to Acquire Expertise
  46. 46. Without sufficient volume it is not sensible to make but rather to buy …
  47. 47. lawyers and law firms provide substantial value
  48. 48. BUT the problem of agency costs always looms
  49. 49. an economic concept concerning the fee to a “principal” (an organization, person or group of persons), when the principal chooses or hires an "agent" to act on its behalf. Because the two parties have different interests and the agent has more information, the principal cannot directly ensure that its agent is always acting in its (the principal's) best interests.
  50. 50. agency costs turn allies (friends) into frenemies
  51. 51. frenemy
  52. 52. agency costs muddy the inside vs outside counsel relationship
  53. 53. (Partially) solving the industry’s requires engineering around these agency issues?
  54. 54. Measurement, Standards, Metrics, #DATA
  55. 55. this allows for a partial solution to the problem
  56. 56. Many at this conference and at conferences like this are working on the problem
  57. 57. there are successes from other sectors of the economy from whom we can learn
  58. 58. where the supply chain thrived in a metrics heavy environment…
  59. 59. the industrialization of the artisan Part Two
  60. 60. across the economy there are many effort to convert an artisanal process into an industrial process
  61. 61. as we move toward a more metrics centered field we want to ensure that we can maintain the artisan elements that DO ADD VALUE
  62. 62. My favorite non-legal example Sal Consiglio - (Sally’s APizza) Domino’s Ad Circa 1990’s ARTISAN INDUSTRIAL
  63. 63. the industrialization of the artisan
  64. 64. lets focus on these two because for now this is where 
 process improvement and 
 data should be directed
  65. 65. so … remaining mindful of the lawyer and law firm value proposition
  66. 66. but with an eye toward reducing the agency costs issues …
  67. 67. every organization in law needs a data strategy
  68. 68. Toward an Enterprise Data Strategy in Legal Part Three
  69. 69. every organization in law needs a data strategy
  70. 70. Capture, Clean, Regularize Data to support a range of tasks
  71. 71. Deploy Data for Specific Enterprise Applications Develop a data roadmap
  72. 72. We want data to help support two major things …
  73. 73. We want data to help support two major things … substantive predictions procedural predictions
  74. 74. there is often a significant spread between Kim Craig @ Seyfarth Lean Consulting Chicago Legal Technology + Innovation MEETUP
  75. 75. so think of the process map as a first order estimation of your actual processes
  76. 76. rich / granular data can help illuminate the actual processes present in various (legal) organizations
  77. 77. for each node in a process we want to be able to render a prediction about things such as duration,cost, etc.
  78. 78. each unit of time linked + logged to a node on the process map
  79. 79. if there is not a node than it can be added and thus the map becomes more reflective of reality
  80. 80. just be careful not to create a #ridiculogram
  81. 81. with predictions about individual nodes
  82. 82. we can then sum to generate predictions about the distributional moments of an overall matter (or phase) (i.e. mean, variance, skewness, kurtosis)
  83. 83. this matter should take … between 9-15 months in 85% of the similar matters (what about the long tail?)
  84. 84. this matter will cost… most common range 275k - 345k but the second mode is 555k - 625k (and that second mode typically is achieved when the following factors are present … )
  85. 85. #LegalData Collaboration Point
  86. 86. transparency as the relationship glue (and trust that comes with transparency)
  87. 87. how could you facilitate data sharing / transparency?
  88. 88. sharing data between customer and client (real time, no filter?)
  89. 89. are law firms AND corporate counsel willing to engage in a 
 two way data exchange ?
  90. 90. We want data to help support two major things … substantive predictions procedural predictions
  91. 91. Here are just a subset of the substantive predictions we are trying to undertake in legal …
  92. 92. #Predict Relevant Documents Data Driven EDiscovery/Due Diligence (Predictive Coding) SUBSTANTIVE LEGAL PREDICTIONS
  93. 93. #Predict Relevant Documents Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Contract Terms/Outcomes Data Driven Transactional Work SUBSTANTIVE LEGAL PREDICTIONS
  94. 94. #Predict Relevant Documents Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Rouge Behavior Data Driven Compliance #Predict Contract Terms/Outcomes Data Driven Transactional Work SUBSTANTIVE LEGAL PREDICTIONS
  95. 95. #Predict Relevant Documents #Predict Case Outcomes Data Driven Legal Underwriting Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Rouge Behavior Data Driven Compliance #Predict Contract Terms/Outcomes Data Driven Transactional Work SUBSTANTIVE LEGAL PREDICTIONS
  96. 96. Data Driven Transactional Work
  97. 97. Meet Bob
  98. 98. Meet Bob lawyer on a major corporate transaction
  99. 99. Meet Bob Bob is about to engage in yet another round of markup on deal terms lawyer on a major corporate transaction
  100. 100. Meet Bob Bob is about to engage in yet another round of markup on deal terms this round is likely to generate a delay on the expected close of the deal lawyer on a major corporate transaction
  101. 101. how much value is created by these modifications? how much delay will be introduced? vs.
  102. 102. Need a better understanding of the actual drivers of risk
  103. 103. Being able to compute the change in risk as a function of a change in deal terms
  104. 104. Requires Mapping of Deal Terms to actual substantive outcomes #legaldata #legalanalytics
  105. 105. this is particularly important when non-lawyers are doing the negotiation (for example your global sales force)
  106. 106. Data Driven Compliance
  107. 107. 80%+ of the world’s data is unstructured data
  108. 108. Solution is to either let tech or human process that data
  109. 109. And humans are actually pretty good pattern detectors
  110. 110. But only for certain types of problems
  111. 111. Trading (HFT in particular) is about looking for anomalies
  112. 112. the discovery + compliance convergence
  113. 113. a hard #bigdata problem in law (near real time) compliance FCPA, Product Defect, etc.
  114. 114. the goal is near real time monitoring
  115. 115. defect w/5 ‘airbag’ version 1.0 backdate w/5 ‘option’ etc.
  116. 116. near real time monitoring of version 2.0 a massive volume of communications
  117. 117. Corp Security Beginning to mirror today’s NSA
  118. 118. Behavior will change (i.e. rogue action will be done offline) Corp Security Beginning to mirror today’s NSA
  119. 119. Behavior will change But Behavior Change will lag (i.e. rogue action will be done offline) (i.e. folks will craft incriminating communications at least for a while) Corp Security Beginning to mirror today’s NSA
  120. 120. thus, discovery (in part) becomes compliance and some (only some) litigation is avoided legal standards will still shift real time monitoring will generate lots of false positives
  121. 121. #Predict Case Outcomes Data Driven Legal Underwriting
  122. 122. A Deeper Dive on Predicting Predicting Case Outcomes (other problems can be solved using similar methods)
  123. 123. Supreme Court of United States #PredictSCOTUS
  124. 124. There are only 3 ways 
 to predict something Experts Crowds Algorithms
  125. 125. Experts
  126. 126. Columbia Law Review October, 2004 Theodore W. Ruger, Pauline T. Kim, Andrew D. Martin, Kevin M. Quinn Legal and Political Science Approaches to Predicting Supreme Court Decision Making The Supreme Court Forecasting Project:
  127. 127. experts
  128. 128. Case Level Prediction Justice Level Prediction 67.4% experts 58% experts From the 68 Included Cases for the 2002-2003 Supreme Court Term
  129. 129. these experts probably overfit
  130. 130. they fit to the noise and not the signal
  131. 131. we need to evaluate experts and somehow benchmark their expertise
  132. 132. from a pure forecasting standpoint
  133. 133. the best known SCOTUS predictor is
  134. 134. the law version of superforecasting
  135. 135. Crowds
  136. 136. crowds
  137. 137. https://fantasyscotus.lexpredict.com/case/list/ We can generate Crowd Sourced Predictions
  138. 138. however, not all members of crowd are made equal
  139. 139. we maintain a ‘supercrowd’ which is the top n% of predictors up to time t
  140. 140. the ‘supercrowd’ outperforms the overall crowd (and the best single player)
  141. 141. not enough crowd based decision making in institutions
  142. 142. “Software developers were asked on two separate days to estimate the completion time for a given task, the hours they projected differed by 71%, on average. When pathologists made two assessments of the severity of biopsy results, the correlation between their ratings was only .61 (out of a perfect 1.0), indicating that they made inconsistent diagnoses quite frequently. Judgments made by different people are even more likely to diverge.”
  143. 143. in law here is a commercial offering
  144. 144. design to unlock untapped expertise in organizations
  145. 145. Allowing for Frictionless Crowdsourcing #ManualUnderwriting
  146. 146. https://lexsemble.com/
  147. 147. https://lexsemble.com/
  148. 148. Algorithms
  149. 149. Black Reed Frankfurter Douglas Jackson Burton Clark Minton Warren Harlan Brennan Whittaker Stewart White Goldberg Fortas Marshall Burger Blackmun Powell Rehnquist Stevens OConnor Scalia Kennedy Souter Thomas Ginsburg Breyer Roberts Alito Sotomayor Kagan 1953 1963 1973 1983 1993 2003 2013 9-0 Reverse 8-1, 7-2, 6-3 19 19 19 19 19 20 20 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 - Reverse 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 - 8-1, 7-2, 6-3 9-0 19 19 19 19 19 20 20 algorithms
  150. 150. we have developed an algorithm that we call {Marshall}+ random forest
  151. 151. Benchmarking since 1953 + Using only data available prior to the decision Mean Court Direction [FE] Mean Court Direction 10 [FE] Mean Court Direction Issue [FE] Mean Court Direction Issue 10 [FE] Mean Court Direction Petitioner [FE] Mean Court Direction Petitioner 10 [FE] Mean Court Direction Respondent [FE] Mean Court Direction Respondent 10 [FE] Mean Court Direction Circuit Origin [FE] Mean Court Direction Circuit Origin 10 [FE] Mean Court Direction Circuit Source [FE] Mean Court Direction Circuit Source 10 [FE] Difference Justice Court Direction [FE] Abs. Difference Justice Court Direction [FE] Difference Justice Court Direction Issue [FE] Abs. Difference Justice Court Direction Issue [FE] Z Score Difference Justice Court Direction Issue [FE] Difference Justice Court Direction Petitioner [FE] Abs. Difference Justice Court Direction Petitioner [FE] Difference Justice Court Direction Respondent [FE] Abs. Difference Justice Court Direction Respondent [FE] Z Score Justice Court Direction Difference [FE] Justice Lower Court Direction Difference [FE] Justice Lower Court Direction Abs. Difference [FE] Justice Lower Court Direction Z Score [FE] Z Score Justice Lower Court Direction Difference [FE] Agreement of Justice with Majority [FE] Agreement of Justice with Majority 10 [FE] Difference Court and Lower Ct Direction [FE] Abs. Difference Court and Lower Ct Direction [FE] Z-Score Difference Court and Lower Ct Direction [FE] Z-Score Abs. Difference Court and Lower Ct Direction [FE] Justice [S] Justice Gender [FE] Is Chief [FE] Party President [FE] Natural Court [S] Segal Cover Score [SC] Year of Birth [FE] Mean Lower Court Direction Circuit Source [FE] Mean Lower Court Direction Circuit Source 10 [FE] Mean Lower Court Direction Issue [FE] Mean Lower Court Direction Issue 10 [FE] Mean Lower Court Direction Petitioner [FE] Mean Lower Court Direction Petitioner 10 [FE] Mean Lower Court Direction Respondent [FE] Mean Lower Court Direction Respondent 10 [FE] Mean Justice Direction [FE] Mean Justice Direction 10 [FE] Mean Justice Direction Z Score [FE] Mean Justice Direction Petitioner [FE] Mean Justice Direction Petitioner 10 [FE] Mean Justice Direction Respondent [FE] Mean Justice Direction Respondent 10 [FE] Mean Justice Direction for Circuit Origin [FE] Mean Justice Direction for Circuit Origin 10 [FE] Mean Justice Direction for Circuit Source [FE] Mean Justice Direction for Circuit Source 10 [FE] Mean Justice Direction by Issue [FE] Mean Justice Direction by Issue 10 [FE] Mean Justice Direction by Issue Z Score [FE] Admin Action [S] Case Origin [S] Case Origin Circuit [S] Case Source [S] Case Source Circuit [S] Law Type [S] Lower Court Disposition Direction [S] Lower Court Disposition [S] Lower Court Disagreement [S] Issue [S] Issue Area [S] Jurisdiction Manner [S] Month Argument [FE] Month Decision [FE] Petitioner [S] Petitioner Binned [FE] Respondent [S] Respondent Binned [FE] Cert Reason [S] Mean Agreement Level of Current Court [FE] Std. Dev. of Agreement Level of Current Court [FE] Mean Current Court Direction Circuit Origin [FE] Std. Dev. Current Court Direction Circuit Origin [FE] Mean Current Court Direction Circuit Source [FE] Std. Dev. Current Court Direction Circuit Source [FE] Mean Current Court Direction Issue [FE] Z-Score Current Court Direction Issue [FE] Std. Dev. Current Court Direction Issue [FE] Mean Current Court Direction [FE] Std. Dev. Current Court Direction [FE] Mean Current Court Direction Petitioner [FE] Std. Dev. Current Court Direction Petitioner [FE] Mean Current Court Direction Respondent [FE] Std. Dev. Current Court Direction Respondent [FE] 0.00781 0.00205 0.00283 0.00604 0.00764 0.00971 0.00793 TOTAL 0.04403 Justice and Court Background Information Case Information 0.00978 0.00971 0.00845 0.00953 0.01015 0.01370 0.01190 0.01125 0.00706 0.01541 0.01469 0.00595 0.02014 0.01349 0.01406 0.01199 0.01490 0.01179 0.01408 TOTAL 0.22814 Overall Historic Supreme Court Trends 0.00988 0.01997 0.01546 0.00938 0.00863 0.00904 0.00875 0.00925 0.00791 0.00864 0.00951 0.01017 TOTAL 0.12663 Lower Court Trends 0.00962 0.01017 0.01334 0.00933 0.00949 0.00874 0.00973 0.00900 TOTAL 0.07946 0.00955 0.00936 0.00789 0.00850 0.00945 0.01021 0.01469 0.00832 0.01266 0.00918 0.00942 0.00863 0.00894 0.00882 0.00888 Current Supreme Court Trends TOTAL 0.14456 Individual Supreme Court Justice Trends 0.01248 0.01530 0.00826 0.00732 0.01027 0.00724 0.01030 0.00792 0.00945 0.00891 0.00970 0.01881 0.00950 0.00771 TOTAL 0.14323 0.01210 0.00929 0.01167 0.00968 0.01055 0.00705 0.00708 0.00690 0.00699 0.01280 0.01922 0.02494 0.01126 0.00992 0.00866 0.01483 0.01522 0.01199 0.01217 0.01150 TOTAL 0.23391 Differences in Trends
  152. 152. Total Cases Predicted Total Votes Predicted 7,700 68,964
  153. 153. Justice Prediction Case Prediction 70.9% accuracy 69.6% accuracy From 1953 - 2014
  154. 154. Our algorithm is a special version of random forest Black Reed Frankfurter Douglas Jackson Burton Clark Minton Warren Harlan Brennan Whittaker Stewart White Goldberg Fortas Marshall Burger Blackmun Powell Rehnquist Stevens OConnor Scalia Kennedy Souter Thomas Ginsburg Breyer Roberts Alito Sotomayor Kagan 1953 1963 1973 1983 1993 2003 2013 9-0 Reverse 8-1, 7-2, 6-3 19 19 19 19 19 20 20 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 - Reverse 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 - 8-1, 7-2, 6-3 9-0 19 19 19 19 19 20 20 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2463244 http://arxiv.org/abs/1407.6333 available at Revise and Resubmit @ PloS One
  155. 155. Experts, Crowds, Algorithms
  156. 156. For most problems ... ensembles of these streams outperform any single stream
  157. 157. Humans + Machines
  158. 158. Humans + Machines >
  159. 159. Humans + Machines Humans or Machines >
  160. 160. Ensembles come in various forms
  161. 161. Here is a well known example
  162. 162. Poll Aggregation is one form of ensemble where the learning question is to determine how much weight (if any) to assign to each individual poll
  163. 163. poll weighting
  164. 164. A Visual Depiction of How to build an ensemble method in our judicial prediction example
  165. 165. expert crowd algorithm ensemble method learning problem is to discover when to use a given stream of intelligence
  166. 166. expert crowd algorithm via back testing we can learn the weights to apply for particular problems ensemble method learning problem is to discover when to use a given stream of intelligence
  167. 167. Legal Analytics + #MLaaS Act Four
  168. 168. Given large fixed costs infrastructure + human capital (data scientists) Historically speaking
  169. 169. harder to successfully deploy high quality enterprise applications for relatively narrow (sub)verticals
  170. 170. Law is a relatively small vertical and there is lots of diversity among tasks lawyers undertake …
  171. 171. in addition there is a borderline pathological numerophobia among lawyers
  172. 172. Analytics / Quant Legal Prediction has come to law Notwithstanding these head winds—
  173. 173. I predict some very interesting economic forces will impact the #legalanalytics space
  174. 174. And applications are about to get far cheaper to develop
  175. 175. Emerging Business Model - Machine Learning as a Service #MLaaS
  176. 176. The Cloud Wars
  177. 177. Commercial Examples
  178. 178. Machine Learning as a Service #MLaaS
  179. 179. Machine Learning as a Service #MLaaS
  180. 180. Machine Learning as a Service #MLaaS
  181. 181. Machine Learning as a Service #MLaaS
  182. 182. historically one needed to build the full stack (i.e end to end) for an application
  183. 183. Standing on 
 the Shoulders of Giants
  184. 184. The (Emerging) Last Mile Problem in (Legal) Analytics
  185. 185. Off the Shelf #MLaaS, etc. (perhaps with some configuration and/or customization) Unique Domain Specific Offering
  186. 186. MLaas + Open Source Decreases Cost of Production Lowers the Cost of Protoyping
  187. 187. Fin (Legal) Tech Part Five
  188. 188. Today I have encouraged collaboration
  189. 189. and one reason is that both sides (firms / clients) can unlock more enterprise value by working together
  190. 190. because past is merely prelude
  191. 191. because the biggest change in legal not robots
  192. 192. because the biggest change in legal financialization not robots
  193. 193. developing a data strategy
  194. 194. developing a data strategy leveraging #MLaaS
  195. 195. developing a data strategy leveraging #MLaaS we get better at predicting
  196. 196. developing a data strategy leveraging #MLaaS we get better at predicting which opens the door for… #FinTech
  197. 197. #FinTech removing socially meaningless frictions characterizing (pricing) increasingly exotic forms of risk
  198. 198. #Fin(Legal)Tech application of those ideas and technology to a wide range of law related spheres including litigation, transactional work and compliance.
  199. 199. if we can predict we can develop insurance
  200. 200. if we can predict we can develop insurance if we can predict we can develop trading strategies
  201. 201. if we can predict we can develop insurance if we can predict we can develop trading strategies if we can predict we have assets under mgmt.
  202. 202. Just a Few Examples of Fin(Legal)Tech
  203. 203. #fin(legal)tech pricing
  204. 204. it is *not* predicting cost of this particular matter where n=1
  205. 205. correctly characterize the distributional properties of a portfolio of matters
  206. 206. both + and - including identification of outliers
  207. 207. apply portfolio theory
  208. 208. to take n=1 and scale to n=many #fin(legal)tech
  209. 209. #self insurance today this is how you would run a more rigorous version of
  210. 210. tomorrow? learn from legal ops service offering to build a commercial insurance product offering legal cost insurance ? other exotic insurance offerings?
  211. 211. AIG to Launch Data- Driven Legal Ops Business in 2016 https://bol.bna.com/aig-to- launch-data-driven-legal- ops-business-in-2016/
  212. 212. #fin(legal)tech In such a world, Law Firm is *not* interfacing with client but rather insurance company regarding fees
  213. 213. Earlier I discussed the application of experts, crowds + algorithms
  214. 214. as applied to predicting case outcomes
  215. 215. that was an example of manual underwriting
  216. 216. Given our ability to offer forecasts of judicial outcomes, we wondered if this information could inform an event driven trading strategy ?
  217. 217. Paper Released August 24, 2015 http://arxiv.org/abs/1508.05751 available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726
  218. 218. lots of litigation decisions are just a version of this basic idea law = finance
  219. 219. lots of litigation decisions are actually implicit litigation finance (or self insurance) #fin(legal)tech
  220. 220. But most implicit litigation finance is not based upon 
 rigorous underwriting … law =! finance (but it will)
  221. 221. http://www.slideshare.net/ Danielkatz/fin-legal-tech-laws- future-from-finances-past- professors-daniel-martin-katz- michael-j-bommarito-ii
  222. 222. TheLawLab.com
  223. 223. FinLegalTechConference.comNovember 4, 2016
  224. 224. In sum, I believe …
  225. 225. Over the coming years, we are going to be able financialize large elements of the legal industry
  226. 226. By which I mean —- apply the tools of finance and insurance to measure / predict a wide range of procedural + substantive outcomes
  227. 227. it will help better establish the value proposition associated with a wide range of legal tasks …
  228. 228. As we move items from the ‘art’ column to the ‘science’ column …
  229. 229. There will be impacts on the industrial organization of the legal industry
  230. 230. But what remains there after will be a better industry …
  231. 231. it will help focus every individual and every organization on the places where they actually provide a return on investment (ROI)
  232. 232. Daniel Martin Katz @ computational computationallegalstudies.com lexpredict.com danielmartinkatz.com illinois tech - chicago kent college of law@

×