Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Complexity
& Prediction
daniel martin katz
michael j bommarito
Toward a Characterization of
Law +
Legal Systems as Complex...
Complexity &
Prediction
daniel martin katz
blog | ComputationalLegalStudies.com
corp | LexPredict.com
page | DanielMartinK...
Today we are excited to
return to Ann Arbor …
for us -
this is the place
where it all began …
cut & paste
safety first
west hall
stephen
wolfram’s
robot
presence
Here Are a Few
Things from Our Time
@ UM CSCS
3D HD Visualization of Supreme
Court Citation Network
Campaign Contributions...
American
Federal
Judiciary
American
Law Professoriate
Building New Algorithms
Large Scale
Judicial Studies
Here Are a Few
...
Mathematical Formalization of
the United States Code
Congressional Bills
Law as a Seamless Web?
SCOTUS + TAX
Here Are a Fe...
ComputationalLegalStudies.com
BLOG
Here Are a Few
Things from Our Time
@ UM CSCS
For Me —
Culminating in this …
And for both of us
ultimately laying a
foundation for this …
Which Became This …
thelawlab.com
And there is also this …
LexPredict.com
So thanks to the
organizers for inviting
us to participate…
http://lsa.umich.edu/cscs/news-events/all-events/complexity-an...
Today we would like to
sweep across a bunch
of our projects …
in the hopes that it might
spur a conversation
in the hopes that it might
spur a conversation
and
in the hopes that it might
spur a conversation
and
perhaps a collaboration .?
Today —
Three Areas of Our
Research that would
like to highlight
Network Science + Law
Exploring the ‘Evolution’ of the Law and its Institutions
Network Science + Law
Exploring the ‘Evolution’ of the Law and its Institutions
Experts, Crowds + Algorithms
Three Forms o...
Network Science + Law
Exploring the ‘Evolution’ of the Law and its Institutions
Three Forms of (Legal) Prediction
Experts,...
Network Science + Law
Exploring the ‘Evolution’ of the Law
Network Science has been
one of the fastest growing
areas of science over the
past 15+ years …
Networks are one
manner in which we
might formalize the
interdependencies
between units in
complex systems
Physical and
Biological Sciences
Computer
Science
Mapping
of the
Code
The Internet
The Iranian Blogosphere
code
dependencies
Hiring and Placement of
Political Science PhD’s
Co-Sponsorship in Congress
(Fowler et al)
Spread of Obesity
(christakis an...
Among others, we
undertook some of the early
work (exploring merely a
small subset) of possible
applications for law
Network Science
in Legal Studies:
An Overview
The American Federal Judiciary
D. Katz, D. Stafford, Hustle and Flow: A Social
Network Analysis of the American Federal
Judiciary. Ohio St. L. J. 71, 457...
The American Law Professoriate


Computational Model
of Intellectual Diffusion
Katz, D. Gubler J. Zelner J. Bommarito M.
Provins E. & Ingall E. (2011). Reproduction
of Hierarchy? A Social Network Analy...
Tracking the Development
of Common Law Jurisprudence
Views of the SCOTUS Citation Network
Radial
Layout
SCOTUS
Citation
Network
Horizontal
Layout
SCOTUS
Citation
Network
Six Degrees of Marbury v. Madison
Judicial Citation Networks
are Acyclic Digraphs
There has
Cases Decided by
the Supreme Court
Citations in the
Current Year
Citations from
prior years
watch full video at ...
Early Growth of the 

SCOTUS Giant Component
This is still a significant
area for potential
scientific exploration …
#LegalScience
#STEMplusLawEqualsFuture
Not unlike many other
Human Socio-Technical systems
we observe is the
highly skewed distribution
of authority in legal systems
one of the common features —
This is some evidence
supporting the thesis that
Complex SystemsLAW =
Katz, et al (2011)
American Legal Academy
Katz & Stafford (2010)
American Federal Judges
Geist (2009)
Austrian Supreme Cou...
https://computationallegalstudies.com/network-analysis-and-law-tutorial/
For those who
might want to
learn more …
Given ti...
In 2017, 2018 and beyond -
we hope to revisit and expand
upon our earlier work in a
variety of important ways …
(we are op...
Experts, Crowds + Algorithms
Three Forms of (Legal) Prediction
Beyond the policy making
sphere where there is
a real case to be made that
prediction > causal inference …
There has been
growing interest in
rigorous
There has been
growing interest in
rigorous
There has been
growing interest in
out of sample
rigorous
There has been
growing interest in
prediction in law
out of sample
rigorous
#AI #LegalTech
#Machine Learning
#LegalAnalytics
There has been
growing interest in
prediction in law
out of samp...
Here are just a few
predictions
that lawyers are trying to
accomplish on a daily basis
#Predict Relevant Documents
Data Driven EDiscovery/Due Diligence
(Predictive Coding)
#Predict Relevant Documents
Data Driven EDiscovery/Due Diligence
(Predictive Coding)
#Predict Contract Terms/Outcomes
Data...
#Predict Relevant Documents
Data Driven EDiscovery/Due Diligence
(Predictive Coding)
#Predict Rouge Behavior
Data Driven C...
#Predict Relevant Documents
#Predict Case Outcomes
Data Driven Legal Underwriting
Data Driven EDiscovery/Due Diligence
(Pr...
#Predict Relevant Documents
#Predict Case Outcomes
Data Driven Legal Underwriting
Data Driven EDiscovery/Due Diligence
(Pr...
In so much as prediction is
the task in question …
#LegalTech #FinTech
#Fin(Legal)Tech
“The real roll-up of all this isn’t robot lawyers,
it’s financialization, with law becoming an
applied branch of finance and...
#Fin(Legal)Tech
https://computationallegalstudies.com/2016/02/27/fin-legal-tech-
laws-future-from-finances-past-an-expanded-...
Borrowing in part from fin(tech)
There are 3 Known Ways
to Predict Something
Borrowing in part from fin(tech)
Experts, Crowds, Algorithms
example from our own work
predicting the decisions of the
Supreme Court of the United States
#SCOTUS
Experts
Columbia Law Review
October, 2004
Theodore W. Ruger, Pauline T. Kim,
Andrew D. Martin, Kevin M. Quinn
Legal and Political
...
experts
Case Level Prediction
Justice Level Prediction
67.4% experts
58% experts
From the 68
Included
Cases
for the
2002-2003
Supr...
these experts probably
overfit
they fit to the noise
and
not the signal
if this were
finance this
would be
trading
worse than
S&P500
#NoiseTrading
#BuffetChallenge
like many other forms
human endeavor
law is full of 

noise predictors …
we need to
evaluate
legal experts
and
somehow
benchmark
their
expertise
from a pure
forecasting
standpoint
the best
known
SCOTUS
predictor is
the law
version of
superforecasting
Crowds
crowds
https://fantasyscotus.lexpredict.com/case/list/
We can
generate
Crowd
Sourced
Predictions
not all
members of
crowd are
made equal
we maintain
a ‘supercrowd’
which is the top n
of predictors
up to time t-1
the
‘supercrowd’
outperforms
the overall
crowd
(and even the
best single player)
not
enough
crowd
based
decision
making in
institutions
(law included)
“Software developers were asked on
two separate days to estimate the
completion time for a given task, the
hours they proj...
Brief Aside
About
Crowd
Sourced
Prediction
#LegalCrowdSourcing
(most pundits did not
identify as a serious
candidate him until
mid-January 2017)
Neil Gorsuch was #1
o n o u r F a n t a ...
#FantasySCOTUS
Algorithms
Columbia Law Review
October, 2004
Theodore W. Ruger, Pauline T. Kim,
Andrew D. Martin, Kevin M. Quinn
Legal and Political ...
Ruger, et al (2004)
relied upon
Brieman(1984)
(as partially shown below)
Leo Brieman moved away from
CART in Brieman (2001)
Breiman, L.(2001). Random forests.
Machine learning, 45(1), 5-32.
Published in Machine Learning
(A Springer Science Journa...
One well-known problem with
standard classification trees is
their tendency toward overfitting
http://machinelearning202.pbworks.com/w/file/fetch/37597425/
performanceCompSupervisedLearning-caruana.pdf
Random
Forest
(p...
Random forest is an approach to
aggregate weak learners into
collective strong learners
(using a combo of bagging and rand...
Our algorithm is a special version
of random forest (time evolving)
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=24...
We call this a ‘general’ model
of #SCOTUS Prediction
available at
https://arxiv.org/pdf/1612.03473
Not just interested in accuracy
over a short time window
available at
https://arxiv.org/pdf/1612.03473
A locally tuned model will
typically lead to overfitting
as the dynamics shift
available at
https://arxiv.org/pdf/1612.03473
We want a model that is
robust to a large number
of known dynamics …
available at
https://arxiv.org/pdf/1612.03473
Version 2.02
January 16, 2017
243,882
28,009
Case Outcomes
JusticeVotes
Current Version of #PredictSCOTUS
1816-2015
Version 2.02
January 16, 2017
Current Version of #PredictSCOTUS
1816-2015
case accuracy
70.2%
71.9%
justice accuracy
But are these results ‘good’ ?
What constitutes ‘good’
performance in this context?
We Craft
Three
Alternative
‘Null’
Models
Our Model Against the Null Models
Some commentators had suggested using a heuristic rule of

‘always guess reverse’ as a b...
Our Model Against the Null Models
(Null Model 2 ) memory window = inf
This is our model against Null Model 2
What about me...
Our Model Against the Null Models
(Null Model 3 ) finite memory window = 10
We in-sample optimize using future information ...
Over past century, we outperform
M=10 by nearly 5% and have
significant temporal stability at both
the justice, case, term ...
Experts, Crowds, Algorithms
For most problems ...
ensembles of these streams
outperform any single stream
the non-trivial question
is how to optimally assemble
such streams for particular problems
Humans
+
Machines
Humans
+
Machines
>
Humans
+
Machines
Humans
or
Machines
>
Here is what we are
working on right now …
expert
forecast
crowd
forecast
learning problem is to discover how to blend streams of intelligence
algorithm
forecast
ens...
expert
forecast
crowd
forecast
learning problem is to discover how to blend streams of intelligence
algorithm
forecast
ens...
By the way, you
might ask why does
one care about
marginal improvements
in prediction ?
#Fin(Legal)Tech
Given our ability to offer
forecasts of judicial
outcomes, we wondered
if this information could
inform an event driven
tr...
Revise + Resubmit @
http://arxiv.org/abs/1508.05751
available at
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726
Mean-Field Measurement of Risk + Legal Complexity
Measuring Complex Systems
For years, we have been arguing
social, economic & political
complexity
is generating
legal complexity …
This complexity is challenging
our (legal and other)
long standing institutions
Measuring Complexity
in Complex Systems
is a major challenge …
mean-field legal science
This is the United States Code
The Titles in US Code
+
Hierarchical
Structure
Citation
Network
M. Bommarito. & D. Katz. A Mathematical Approach to the
Study of the United States Code. Physica A: Statistical Mechanics
...
D. Katz & M. Bommarito. Measuring the complexity of the law: the United
States Code. Artificial intelligence and law, 22(4)...


Mean-Field Measurement Projects -
Mining 10k’s / 8k’s
Exploring Regultory Risk
One of our current
Paper 1
(Currently Under Review)
M. Bommarito., D. Katz,
Measuring the Temperature
and Diversity of the U.S.
Regulatory Ec...
Summary
Available
Here
https://www.law.ox.ac.uk/business-law-
blog/blog/2017/02/measuring-temperature-
and-diversity-us-re...
“A Form 10-K is an annual report
required by the U.S. Securities and
Exchange Commission (SEC), that gives
a comprehensive...
Data Collection and Pre-Processing
Form 10-K contains
a range of relevant
information about
regulatory exposure

(and othe...
Data Collection and Pre-Processing
“…the Financial Executives
Research Foundation reveals
a mean and median 2015
expense o...
Data Collection and Pre-Processing
Countervailing Incentives:
Report to get a form of
‘securities fraud insurance’
Data Collection and Pre-Processing
Countervailing Incentives:
Report to get a form of
‘securities fraud insurance’
Do not ...
Data Collection and Pre-Processing
data is *not* perfect
but it is large scale
characterization of
the manner in which
reg...
Text of the
10-k for
Company i
in year y
Synonym +
Fuzzy String
Matching to Act,
Agency, U.S. Code
and CFR Masterlist
Gram...
Data
Publically Traded Companies34,000+
Data
Publically Traded Companies
Total Number of 10-K’s
34,000+
160,000+
Data
Publically Traded Companies
Total Number of 10-K’s
Total Number of
Act / Agency References
34,000+
160,000+
4.5 milli...
A Mean - Field Characterization
Temperature
What is
the global
average
temperature?
Has this changed
over time?
Paper 1
(Currently Under Review)
Paper 1
(Currently Under Review)
Temperature =
# of
Acts / Agency
References
as a Function
of Time
Paper 1
(Currently Under Review)
Note:
Norms,
Incentives,
Requirements
have changed
over 20+ years…
A Mean - Field Characterization
Diversity
Use BitString Encoding
as an Information Theoretic
Representation for the Self-
Identified Regulatory Exposure
0
1
1
0
0
0
0
0
1
…
Encoding
Regulatory
Bitstring for
Company i
in year y
For each
{company, year} pair
Encode as 1 for th...
0
1
1
0
0
0
0
0
1
…
Encoding
Regulatory
Bitstring for
Company i
in year y
Text of the
10-k for
Company i
in year y
Synonym...
We have 160,000+ of these
{year, company} bitstrings
Is the Diversity of the System
Growing or Contracting?
We use Hamming Distance
as a Mean-Field Characterization
Diversity of the Overall
Regulatory Ecosystem
Regulatory Dynamics Revealed
by the Securities Filings of
Publicly Traded Companies
Paper 2
(Currently In Progress)
In Bommarito & Katz 2016,
we explored the mean-field
temperature and diversity of
the regulatory ecosystem
and mentioned that there
were likely to be differential
regulatory dynamics and
microclimates therein
Classification Using
Time Series Signatures
Characterizing Dynamics:
In the spirit of
complex systems /
physics, we would
like to try offer a
generalization …
We are
undertaking
a version of
this approach
Riley Crane and Didier Sornette. "Robust dynamic
classes revealed by measuri...
Some
Motivating
Examples
Clean Water Act
Sarbanes Oxley
Y2K
Again,
leveraging
the time series
signature …
μ
σ
τ auto-correlation
(can be thought of as memory)
set each parameter to base zero at
time = and indext 0
variance2
mean...
taxonomy
of
behavioral
signatures
(‘M’ ‘H’, ‘+’)
Anti Kickback
Fairness In Asbestos Injury Resolution
American Clean Energy And Security
Pension Funding Equ...
In sum, we are able to use this
framework to classify any
regulation by its behavior
upon the broader
regulatory ecosystem
Regulatory ‘Profiles’
and Mircoclimates
Once we have a distance metric
or some manner to encode edges…
we can generate a network
projection of the overall
company landscape …
Network Generation Procedure
a) Calculate the Hamming distance matrix as described
in Paper #1 over Acts
b) Threshold the ...
Sequencing
of
Regulatory
Bitstrings
Universal Profiles
versus
Specific Profiles
(Growing Speciation)
Project is Still in Progress
So More to Come Soon …
Today we have
provided an
overview of three
areas of our
research
Network Science + Law
Exploring the ‘Evolution’ of the Law and its Institutions
Three Forms of (Legal) Prediction
Experts,...
We hope to work with
some of you to advance
the state of the science
in this field
Associate Professor of Law
The Law Lab @ Illinois-Tech
Affiliated Faculty
Stanford CodeX
Center for Legal Informatics
Found...
Fellow
Stanford CodeX
Center for Legal Informatics
Director of Research
The Law Lab
IllinoisTech - Chicago Kent Law
Chief Strategy Officer
LexPredict
Chief Executive Officer
LexPredict
LexPredict.com
thelawlab.com
ComputationalLegalStudies.com
BLOG
Michael J. Bommarito II
@ mjbommar
computationallegalstudies.com
lexpredict.com
bommaritollc.com
illinois tech - chicago k...
Daniel Martin Katz
@ computational
computationallegalstudies.com
lexpredict.com
danielmartinkatz.com
illinois tech - chica...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems  - Professors Daniel Martin K...
Upcoming SlideShare
Loading in …5
×

Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems - Professors Daniel Martin Katz + Michael J Bommarito

4,609 views

Published on

Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems - Professors Daniel Martin Katz + Michael J Bommarito

Published in: Law
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems - Professors Daniel Martin Katz + Michael J Bommarito

  1. 1. Complexity & Prediction daniel martin katz michael j bommarito Toward a Characterization of Law + Legal Systems as Complex Systems
  2. 2. Complexity & Prediction daniel martin katz blog | ComputationalLegalStudies.com corp | LexPredict.com page | DanielMartinKatz.com michael j bommarito blog | ComputationalLegalStudies.com corp | LexPredict.com page | bommaritollc.com edu | illinois tech - chicago kent law edu | illinois tech - chicago kent law Toward a Characterization of Legal Systems as Complex Systems Law + lab | theLawLab.comlab | theLawLab.com
  3. 3. Today we are excited to return to Ann Arbor …
  4. 4. for us - this is the place where it all began …
  5. 5. cut & paste safety first west hall stephen wolfram’s robot presence
  6. 6. Here Are a Few Things from Our Time @ UM CSCS 3D HD Visualization of Supreme Court Citation Network Campaign Contributions and Legislative Ecosystems Six Degrees of Marbury v. Madison Electronic World Treaty Index Radial SCOTUS Citation Network
  7. 7. American Federal Judiciary American Law Professoriate Building New Algorithms Large Scale Judicial Studies Here Are a Few Things from Our Time @ UM CSCS
  8. 8. Mathematical Formalization of the United States Code Congressional Bills Law as a Seamless Web? SCOTUS + TAX Here Are a Few Things from Our Time @ UM CSCS
  9. 9. ComputationalLegalStudies.com BLOG Here Are a Few Things from Our Time @ UM CSCS
  10. 10. For Me — Culminating in this …
  11. 11. And for both of us ultimately laying a foundation for this …
  12. 12. Which Became This …
  13. 13. thelawlab.com
  14. 14. And there is also this …
  15. 15. LexPredict.com
  16. 16. So thanks to the organizers for inviting us to participate… http://lsa.umich.edu/cscs/news-events/all-events/complexity-and-the-law.html
  17. 17. Today we would like to sweep across a bunch of our projects …
  18. 18. in the hopes that it might spur a conversation
  19. 19. in the hopes that it might spur a conversation and
  20. 20. in the hopes that it might spur a conversation and perhaps a collaboration .?
  21. 21. Today — Three Areas of Our Research that would like to highlight
  22. 22. Network Science + Law Exploring the ‘Evolution’ of the Law and its Institutions
  23. 23. Network Science + Law Exploring the ‘Evolution’ of the Law and its Institutions Experts, Crowds + Algorithms Three Forms of (Legal) Prediction
  24. 24. Network Science + Law Exploring the ‘Evolution’ of the Law and its Institutions Three Forms of (Legal) Prediction Experts, Crowds + Algorithms Mean-Field Measurement of Risk + Legal Complexity Measuring Complex Systems
  25. 25. Network Science + Law Exploring the ‘Evolution’ of the Law
  26. 26. Network Science has been one of the fastest growing areas of science over the past 15+ years …
  27. 27. Networks are one manner in which we might formalize the interdependencies between units in complex systems
  28. 28. Physical and Biological Sciences
  29. 29. Computer Science Mapping of the Code The Internet The Iranian Blogosphere code dependencies
  30. 30. Hiring and Placement of Political Science PhD’s Co-Sponsorship in Congress (Fowler et al) Spread of Obesity (christakis and fowler) High School Friendship (Moody) Roll Call Votes in Congress (Mucha, et al) Social Science The Political Blogosphere (Adamic & Glance)
  31. 31. Among others, we undertook some of the early work (exploring merely a small subset) of possible applications for law
  32. 32. Network Science in Legal Studies: An Overview
  33. 33. The American Federal Judiciary
  34. 34. D. Katz, D. Stafford, Hustle and Flow: A Social Network Analysis of the American Federal Judiciary. Ohio St. L. J. 71, 457 (2010) available at https://papers.ssrn.com/sol3/papers2.cfm?abstract_id=1103573
  35. 35. The American Law Professoriate
  36. 36. 
 Computational Model of Intellectual Diffusion
  37. 37. Katz, D. Gubler J. Zelner J. Bommarito M. Provins E. & Ingall E. (2011). Reproduction of Hierarchy? A Social Network Analysis of the American Law Professoriate. Journal of Legal Education, 61(1), 76-103. https://papers.ssrn.com/sol3/papers2.cfm?abstract_id=1352656available at
  38. 38. Tracking the Development of Common Law Jurisprudence
  39. 39. Views of the SCOTUS Citation Network Radial Layout SCOTUS Citation Network Horizontal Layout SCOTUS Citation Network
  40. 40. Six Degrees of Marbury v. Madison
  41. 41. Judicial Citation Networks are Acyclic Digraphs
  42. 42. There has Cases Decided by the Supreme Court Citations in the Current Year Citations from prior years watch full video at https://vimeo.com/9427420
  43. 43. Early Growth of the 
 SCOTUS Giant Component
  44. 44. This is still a significant area for potential scientific exploration … #LegalScience #STEMplusLawEqualsFuture
  45. 45. Not unlike many other Human Socio-Technical systems
  46. 46. we observe is the highly skewed distribution of authority in legal systems one of the common features —
  47. 47. This is some evidence supporting the thesis that Complex SystemsLAW =
  48. 48. Katz, et al (2011) American Legal Academy Katz & Stafford (2010) American Federal Judges Geist (2009) Austrian Supreme Court Smith (2007) U.S. Supreme Court Smith (2007) U.S. Law Reviews Post & Eisen (2000) NY Ct of Appeals
  49. 49. https://computationallegalstudies.com/network-analysis-and-law-tutorial/ For those who might want to learn more … Given time is limited for today …
  50. 50. In 2017, 2018 and beyond - we hope to revisit and expand upon our earlier work in a variety of important ways … (we are open to collaborating in appropriate instances with those who are interested)
  51. 51. Experts, Crowds + Algorithms Three Forms of (Legal) Prediction
  52. 52. Beyond the policy making sphere where there is a real case to be made that prediction > causal inference …
  53. 53. There has been growing interest in
  54. 54. rigorous There has been growing interest in
  55. 55. rigorous There has been growing interest in out of sample
  56. 56. rigorous There has been growing interest in prediction in law out of sample
  57. 57. rigorous #AI #LegalTech #Machine Learning #LegalAnalytics There has been growing interest in prediction in law out of sample
  58. 58. Here are just a few predictions that lawyers are trying to accomplish on a daily basis
  59. 59. #Predict Relevant Documents Data Driven EDiscovery/Due Diligence (Predictive Coding)
  60. 60. #Predict Relevant Documents Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Contract Terms/Outcomes Data Driven Transactional Work
  61. 61. #Predict Relevant Documents Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Rouge Behavior Data Driven Compliance #Predict Contract Terms/Outcomes Data Driven Transactional Work
  62. 62. #Predict Relevant Documents #Predict Case Outcomes Data Driven Legal Underwriting Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Rouge Behavior Data Driven Compliance #Predict Contract Terms/Outcomes Data Driven Transactional Work
  63. 63. #Predict Relevant Documents #Predict Case Outcomes Data Driven Legal Underwriting Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Rouge Behavior Data Driven Compliance #Predict Contract Terms/Outcomes Data Driven Transactional Work #Predict Regulatory Outcomes Data Driven Lobbying, etc.
  64. 64. In so much as prediction is the task in question … #LegalTech #FinTech #Fin(Legal)Tech
  65. 65. “The real roll-up of all this isn’t robot lawyers, it’s financialization, with law becoming an applied branch of finance and insurance.” Daniel Martin Katz, professor, Illinois Tech’s Chicago Kent College of Law http://www.ozy.com/fast-forward/why-artificial-intelligence-might-replace-your-lawyer/75435
  66. 66. #Fin(Legal)Tech https://computationallegalstudies.com/2016/02/27/fin-legal-tech- laws-future-from-finances-past-an-expanded-version-of-the-deck/ GO HERE FOR A DETAILED TREATMENT OF THE QUESTION
  67. 67. Borrowing in part from fin(tech)
  68. 68. There are 3 Known Ways to Predict Something Borrowing in part from fin(tech)
  69. 69. Experts, Crowds, Algorithms
  70. 70. example from our own work
  71. 71. predicting the decisions of the Supreme Court of the United States #SCOTUS
  72. 72. Experts
  73. 73. Columbia Law Review October, 2004 Theodore W. Ruger, Pauline T. Kim, Andrew D. Martin, Kevin M. Quinn Legal and Political Science Approaches to Predicting Supreme Court Decision Making The Supreme Court Forecasting Project:
  74. 74. experts
  75. 75. Case Level Prediction Justice Level Prediction 67.4% experts 58% experts From the 68 Included Cases for the 2002-2003 Supreme Court Term
  76. 76. these experts probably overfit
  77. 77. they fit to the noise and not the signal
  78. 78. if this were finance this would be trading worse than S&P500
  79. 79. #NoiseTrading
  80. 80. #BuffetChallenge
  81. 81. like many other forms human endeavor law is full of 
 noise predictors …
  82. 82. we need to evaluate legal experts and somehow benchmark their expertise
  83. 83. from a pure forecasting standpoint
  84. 84. the best known SCOTUS predictor is
  85. 85. the law version of superforecasting
  86. 86. Crowds
  87. 87. crowds
  88. 88. https://fantasyscotus.lexpredict.com/case/list/ We can generate Crowd Sourced Predictions
  89. 89. not all members of crowd are made equal
  90. 90. we maintain a ‘supercrowd’ which is the top n of predictors up to time t-1
  91. 91. the ‘supercrowd’ outperforms the overall crowd (and even the best single player)
  92. 92. not enough crowd based decision making in institutions (law included)
  93. 93. “Software developers were asked on two separate days to estimate the completion time for a given task, the hours they projected differed by 71%, on average. W h e n p a t h o l o g i s t s m a d e t wo assessments of the severity of biopsy results, the correlation between their ratings was only .61 (out of a perfect 1.0), indicating that they made inconsistent diagnoses quite frequently. Judgments made by different people are even more likely to diverge.”
  94. 94. Brief Aside About Crowd Sourced Prediction #LegalCrowdSourcing
  95. 95. (most pundits did not identify as a serious candidate him until mid-January 2017) Neil Gorsuch was #1 o n o u r F a n t a s y Platform 12 Days after Donald Trump was elected President (i.e Nov 20)
  96. 96. #FantasySCOTUS
  97. 97. Algorithms
  98. 98. Columbia Law Review October, 2004 Theodore W. Ruger, Pauline T. Kim, Andrew D. Martin, Kevin M. Quinn Legal and Political Science Approaches to Predicting Supreme Court Decision Making The Supreme Court Forecasting Project:
  99. 99. Ruger, et al (2004) relied upon Brieman(1984) (as partially shown below)
  100. 100. Leo Brieman moved away from CART in Brieman (2001)
  101. 101. Breiman, L.(2001). Random forests. Machine learning, 45(1), 5-32. Published in Machine Learning (A Springer Science Journal)
  102. 102. One well-known problem with standard classification trees is their tendency toward overfitting
  103. 103. http://machinelearning202.pbworks.com/w/file/fetch/37597425/ performanceCompSupervisedLearning-caruana.pdf Random Forest (particularly with special config/ optimization) have proven to be unreasonably effective
  104. 104. Random forest is an approach to aggregate weak learners into collective strong learners (using a combo of bagging and random substrates) (think of it as crowd sourcing of models)
  105. 105. Our algorithm is a special version of random forest (time evolving) https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2463244 available at Revise + Resubmit Version 2.02 - January 16, 2017
  106. 106. We call this a ‘general’ model of #SCOTUS Prediction available at https://arxiv.org/pdf/1612.03473
  107. 107. Not just interested in accuracy over a short time window available at https://arxiv.org/pdf/1612.03473
  108. 108. A locally tuned model will typically lead to overfitting as the dynamics shift available at https://arxiv.org/pdf/1612.03473
  109. 109. We want a model that is robust to a large number of known dynamics … available at https://arxiv.org/pdf/1612.03473
  110. 110. Version 2.02 January 16, 2017 243,882 28,009 Case Outcomes JusticeVotes Current Version of #PredictSCOTUS 1816-2015
  111. 111. Version 2.02 January 16, 2017 Current Version of #PredictSCOTUS 1816-2015 case accuracy 70.2% 71.9% justice accuracy
  112. 112. But are these results ‘good’ ?
  113. 113. What constitutes ‘good’ performance in this context?
  114. 114. We Craft Three Alternative ‘Null’ Models
  115. 115. Our Model Against the Null Models Some commentators had suggested using a heuristic rule of
 ‘always guess reverse’ as a baseline (Null Model 1 ) the always guess Reverse model Turns out it is a lousy model prior to ~1950 Because the reversal rate is not stable over time
  116. 116. Our Model Against the Null Models (Null Model 2 ) memory window = inf This is our model against Null Model 2 What about memory window that selects the most frequent historical outcome? (Green = our model out performs)
  117. 117. Our Model Against the Null Models (Null Model 3 ) finite memory window = 10 We in-sample optimize using future information to select a null model that is among the best performing of all null models as it is using in-sample info this is a deeply unfair null
  118. 118. Over past century, we outperform M=10 by nearly 5% and have significant temporal stability at both the justice, case, term level
  119. 119. Experts, Crowds, Algorithms
  120. 120. For most problems ... ensembles of these streams outperform any single stream
  121. 121. the non-trivial question is how to optimally assemble such streams for particular problems
  122. 122. Humans + Machines
  123. 123. Humans + Machines >
  124. 124. Humans + Machines Humans or Machines >
  125. 125. Here is what we are working on right now …
  126. 126. expert forecast crowd forecast learning problem is to discover how to blend streams of intelligence algorithm forecast ensemble method ensemble model
  127. 127. expert forecast crowd forecast learning problem is to discover how to blend streams of intelligence algorithm forecast ensemble method ensemble model via back testing we can learn the weights to apply for particular problems
  128. 128. By the way, you might ask why does one care about marginal improvements in prediction ? #Fin(Legal)Tech
  129. 129. Given our ability to offer forecasts of judicial outcomes, we wondered if this information could inform an event driven trading strategy ?
  130. 130. Revise + Resubmit @ http://arxiv.org/abs/1508.05751 available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726
  131. 131. Mean-Field Measurement of Risk + Legal Complexity Measuring Complex Systems
  132. 132. For years, we have been arguing social, economic & political complexity is generating legal complexity …
  133. 133. This complexity is challenging our (legal and other) long standing institutions
  134. 134. Measuring Complexity in Complex Systems is a major challenge …
  135. 135. mean-field legal science
  136. 136. This is the United States Code
  137. 137. The Titles in US Code
  138. 138. + Hierarchical Structure Citation Network
  139. 139. M. Bommarito. & D. Katz. A Mathematical Approach to the Study of the United States Code. Physica A: Statistical Mechanics and its Applications, 389(19), 4195-4200 (2010).
  140. 140. D. Katz & M. Bommarito. Measuring the complexity of the law: the United States Code. Artificial intelligence and law, 22(4), 337-374. (2014)
  141. 141. 
 Mean-Field Measurement Projects - Mining 10k’s / 8k’s Exploring Regultory Risk One of our current
  142. 142. Paper 1 (Currently Under Review) M. Bommarito., D. Katz, Measuring the Temperature and Diversity of the U.S. Regulatory Ecosystem (2016) V available at https://arxiv.org/pdf/1612.09244.pdf https://papers.ssrn.com/sol3/papers2.cfm?abstract_id=2891494 2
  143. 143. Summary Available Here https://www.law.ox.ac.uk/business-law- blog/blog/2017/02/measuring-temperature- and-diversity-us-regulatory-ecosystem
  144. 144. “A Form 10-K is an annual report required by the U.S. Securities and Exchange Commission (SEC), that gives a comprehensive summary of a company's financial performance.”
  145. 145. Data Collection and Pre-Processing Form 10-K contains a range of relevant information about regulatory exposure
 (and other risks)
  146. 146. Data Collection and Pre-Processing “…the Financial Executives Research Foundation reveals a mean and median 2015 expense of $1.8M and $522,205, respectively.” Real Resources are used to produced these reports:
  147. 147. Data Collection and Pre-Processing Countervailing Incentives: Report to get a form of ‘securities fraud insurance’
  148. 148. Data Collection and Pre-Processing Countervailing Incentives: Report to get a form of ‘securities fraud insurance’ Do not enumerate all risks under the sun because it may scare investors
  149. 149. Data Collection and Pre-Processing data is *not* perfect but it is large scale characterization of the manner in which regulations impact companies
  150. 150. Text of the 10-k for Company i in year y Synonym + Fuzzy String Matching to Act, Agency, U.S. Code and CFR Masterlist Gramm Leach Bliley Financial Services Modernization Act GLBA Graham Leach Bliley Financial Services Modernization” Act Gramm Leach Bliley Financial Services Modernization Act of 1999 Data Collection and Pre-Processing
  151. 151. Data Publically Traded Companies34,000+
  152. 152. Data Publically Traded Companies Total Number of 10-K’s 34,000+ 160,000+
  153. 153. Data Publically Traded Companies Total Number of 10-K’s Total Number of Act / Agency References 34,000+ 160,000+ 4.5 million
  154. 154. A Mean - Field Characterization Temperature
  155. 155. What is the global average temperature? Has this changed over time? Paper 1 (Currently Under Review)
  156. 156. Paper 1 (Currently Under Review) Temperature = # of Acts / Agency References as a Function of Time
  157. 157. Paper 1 (Currently Under Review) Note: Norms, Incentives, Requirements have changed over 20+ years…
  158. 158. A Mean - Field Characterization Diversity
  159. 159. Use BitString Encoding as an Information Theoretic Representation for the Self- Identified Regulatory Exposure
  160. 160. 0 1 1 0 0 0 0 0 1 … Encoding Regulatory Bitstring for Company i in year y For each {company, year} pair Encode as 1 for the presence of an act / agency Pseudocode for BitString Encoding Encode as 0 otherwise
  161. 161. 0 1 1 0 0 0 0 0 1 … Encoding Regulatory Bitstring for Company i in year y Text of the 10-k for Company i in year y Synonym + Fuzzy String Matching to Act, Agency, U.S. Code and CFR Masterlist Gramm Leach Bliley Financial Services Modernization Act GLBA Graham Leach Bliley Financial Services Modernization” Act Gramm Leach Bliley Financial Services Modernization Act of 1999 A Mean - Field Characterization
  162. 162. We have 160,000+ of these {year, company} bitstrings
  163. 163. Is the Diversity of the System Growing or Contracting?
  164. 164. We use Hamming Distance as a Mean-Field Characterization
  165. 165. Diversity of the Overall Regulatory Ecosystem
  166. 166. Regulatory Dynamics Revealed by the Securities Filings of Publicly Traded Companies Paper 2 (Currently In Progress)
  167. 167. In Bommarito & Katz 2016, we explored the mean-field temperature and diversity of the regulatory ecosystem
  168. 168. and mentioned that there were likely to be differential regulatory dynamics and microclimates therein
  169. 169. Classification Using Time Series Signatures Characterizing Dynamics:
  170. 170. In the spirit of complex systems / physics, we would like to try offer a generalization …
  171. 171. We are undertaking a version of this approach Riley Crane and Didier Sornette. "Robust dynamic classes revealed by measuring the response function of a social system." Proceedings of the National Academy of Sciences 105, no. 41 (2008): 15649-15653.
  172. 172. Some Motivating Examples Clean Water Act Sarbanes Oxley Y2K
  173. 173. Again, leveraging the time series signature …
  174. 174. μ σ τ auto-correlation (can be thought of as memory) set each parameter to base zero at time = and indext 0 variance2 mean (H, M, L) (+, -) (H, M, L)
  175. 175. taxonomy of behavioral signatures
  176. 176. (‘M’ ‘H’, ‘+’) Anti Kickback Fairness In Asbestos Injury Resolution American Clean Energy And Security Pension Funding Equity Medicare, Medicaid, And Schip Benefits Improvement Clusters that have a Similar Behaviorial Signature (Not necessariy topically similar) (‘H’ ‘H’, ‘+’) Patient Protection And Affordable Care Secure And Fair Enforcement For Mortgage Licensing Dodd Frank Wall Street Reform And Consumer Protection Energy Independence And Security Tax Relief, Unemployment Insurance Reauthorization
  177. 177. In sum, we are able to use this framework to classify any regulation by its behavior upon the broader regulatory ecosystem
  178. 178. Regulatory ‘Profiles’ and Mircoclimates
  179. 179. Once we have a distance metric or some manner to encode edges…
  180. 180. we can generate a network projection of the overall company landscape …
  181. 181. Network Generation Procedure a) Calculate the Hamming distance matrix as described in Paper #1 over Acts b) Threshold the matrix by removing all elements whose distances are greater than D (D=3 in this figure) c) Generate graph from resulting edge-weighted adjacency matrix, where edge weight = 1/(1 + d) d) Layout is Fruchterman-Reingold weighted showing only giant component
  182. 182. Sequencing of Regulatory Bitstrings Universal Profiles versus Specific Profiles (Growing Speciation)
  183. 183. Project is Still in Progress So More to Come Soon …
  184. 184. Today we have provided an overview of three areas of our research
  185. 185. Network Science + Law Exploring the ‘Evolution’ of the Law and its Institutions Three Forms of (Legal) Prediction Experts, Crowds + Algorithms Mean-Field Measurement of Risk + Legal Complexity Measuring Complex Systems
  186. 186. We hope to work with some of you to advance the state of the science in this field
  187. 187. Associate Professor of Law The Law Lab @ Illinois-Tech Affiliated Faculty Stanford CodeX Center for Legal Informatics Founder + Director
  188. 188. Fellow Stanford CodeX Center for Legal Informatics Director of Research The Law Lab IllinoisTech - Chicago Kent Law
  189. 189. Chief Strategy Officer LexPredict Chief Executive Officer LexPredict
  190. 190. LexPredict.com
  191. 191. thelawlab.com
  192. 192. ComputationalLegalStudies.com BLOG
  193. 193. Michael J. Bommarito II @ mjbommar computationallegalstudies.com lexpredict.com bommaritollc.com illinois tech - chicago kent college of law@ thelawlab.com
  194. 194. Daniel Martin Katz @ computational computationallegalstudies.com lexpredict.com danielmartinkatz.com illinois tech - chicago kent college of law@ thelawlab.com

×