Artificial Intelligence and Law - 
A Primer

6,172 views

Published on

Artificial Intelligence in Law (and beyond) including Machine Learning as a Service, Quantitative Legal Prediction / Legal Analytics, Experts + Crowds + Algorithms

Published in: Law

Artificial Intelligence and Law - 
A Primer

  1. 1. [ a.i. + law ] 
 a six part primer artificial intelligence in law (and beyond) daniel martin katz blog | ComputationalLegalStudies.com corp | LexPredict.com page | DanielMartinKatz.com edu | chicago kent college of law
  2. 2. There has been lots of recent interest in applying artificial intelligence to law
  3. 3. and there is a bit of confusion as to where we stand today and where we are headed
  4. 4. A.I. in Law A Quick Reset Part I< >
  5. 5. data driven AI rules based AI Competing Orientations in Artificial Intelligence
  6. 6. expert systems Computational Law Data Driven Rules Based prediction models and methods network analytic methods natural language processing self executing law visual law computable codes
  7. 7. we see a decent amount of rules based AI in legal industry
  8. 8. Three Examples of Rules Based A.I.
  9. 9. tax preparation software
  10. 10. Illinois Tech Chicago-Kent Law has a platform that allows individuals to automate various legal forms, etc.
  11. 11. used by a variety of legal aid organizations
  12. 12. What is A2J Author? An online tool from Chicago-Kent & CALI to build graphical interfaces for low-income, self-represented individuals.
  13. 13. 2,524,639 1,529,205 A2J Guided Interviews
  14. 14. Rules Based A.I.
  15. 15. Rules Based A.I.
  16. 16. Among other things Neota has been used to create decision trees to support lawyers / non lawyers
  17. 17. What do I do if there has been An issue in Human Resources ? A potential FCPA violation? A potential data breach?
  18. 18. Decision Trees are a step by step memorialization of best practices
  19. 19. so again we see a decent amount of rules based AI in legal industry
  20. 20. that is actually pretty consistent with path of A.I. in general
  21. 21. lots of issues with expert systems and/or rules based A.I. (without data or an evolutionary dynamic)
  22. 22. rules based A.I. data driven A.I. 1980’s, 1990’s, Early 2000’s >
  23. 23. rules based A.I. data driven A.I. 1980’s, 1990’s, Early 2000’s > rules based A.I. data driven A.I. 2005 - Present < ~
  24. 24. Ultimately we are trying to learn the rules / dynamics that underlie some class of activity
  25. 25. With that understanding we want to be able to mimic / predict
  26. 26. A.I. State of the Art
  27. 27. A.I. State of the Art purely data centric
  28. 28. A.I. State of the Art purely data centric augment expert forecasts w/ data
  29. 29. iterative data < > rules A.I. State of the Art purely data centric augment expert forecasts w/ data
  30. 30. #BigData pan everything #Hashtag Part II< >
  31. 31. #DataScience are already influencing our lives in a variety of meaningful ways #BigData #Analytics #A.I.
  32. 32. To date, the most successful commercial applications have massive returns to scale and aim for cross societal payoffs…
  33. 33. Medicine
  34. 34. Finance
  35. 35. Logistics
  36. 36. Agriculture
  37. 37. Transportation
  38. 38. Retail
  39. 39. Given large fixed costs
  40. 40. Given large fixed costs infrastructure + human capital (data scientists)
  41. 41. harder to successfully deploy high quality enterprise applications for relatively narrow (sub)verticals
  42. 42. The Rise of #LegalAnalytics Part II< >
  43. 43. Law is a relatively small vertical and there is lots of diversity among tasks lawyers undertake …
  44. 44. in addition there is a borderline pathological numerophobia among lawyers
  45. 45. plus the implicit (explicit) challenge of partnership as the dominant form of the organization within our market
  46. 46. taken together this has challenged the deployment 
 of analytics in legal
  47. 47. Analytics / Quant Legal Prediction has come to law Notwithstanding these head winds—
  48. 48. #LegalAnalytics Quantitative Legal Prediction
  49. 49. #LegalAnalytics Quantitative Legal Prediction
  50. 50. #LegalAnalytics Quantitative Legal Prediction
  51. 51. #LegalAnalytics Quantitative Legal Prediction
  52. 52. #LegalAnalytics Quantitative Legal Prediction
  53. 53. #LegalAnalytics Quantitative Legal Prediction
  54. 54. Some Commercial Applications
  55. 55. In a real sense, this represents just a narrow set of products
  56. 56. #ContractAnalytics Quantitative Legal Prediction
  57. 57. #JudicialAnalytics Quantitative Legal Prediction
  58. 58. #PredictiveCoding #E-Discovery Quantitative Legal Prediction
  59. 59. General Counsels as Legal Procurement Specialists TyMetrix/ELM - Using $50 billion+ in Legal Spend Data to Help GC’s Look for Arbitrage Opportunities, Value Propositions in Hiring Law Firms #LegalSpendAnalytics Quantitative Legal Prediction
  60. 60. #LegalAnalytics Quantitative Legal Prediction https://lexsemble.com/
  61. 61. #NegotiationAnalytics Quantitative Legal Prediction
  62. 62. Here are just a subset of the tasks that we are trying to accomplish in legal …
  63. 63. #Predict Case Outcomes Data Driven Legal Underwriting
  64. 64. #Predict Case Outcomes Data Driven Legal Underwriting #Predict Legal Costs Data Driven Legal Operations
  65. 65. #Predict Relevant Documents #Predict Case Outcomes Data Driven Legal Underwriting Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Legal Costs Data Driven Legal Operations
  66. 66. #Predict Relevant Documents #Predict Case Outcomes Data Driven Legal Underwriting Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Legal Costs #Predict Rouge Behavior Data Driven Legal Operations Data Driven Compliance
  67. 67. #Predict Relevant Documents #Predict Case Outcomes Data Driven Legal Underwriting Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Legal Costs #Predict Rouge Behavior Data Driven Legal Operations Data Driven Compliance #Predict Contract Terms/Outcomes Data Driven Transactional Work
  68. 68. The Three Forms of (Legal) Prediction Part III< >
  69. 69. A Deeper Dive on Predicting Predicting Case Outcomes (other problems can be solved using similar methods)
  70. 70. Supreme Court of United States #PredictSCOTUS
  71. 71. There are only 3 ways 
 to predict something Experts Crowds Algorithms
  72. 72. Experts
  73. 73. Columbia Law Review October, 2004 Theodore W. Ruger, Pauline T. Kim, Andrew D. Martin, Kevin M. Quinn Legal and Political Science Approaches to Predicting Supreme Court Decision Making The Supreme Court Forecasting Project:
  74. 74. experts
  75. 75. Case Level Prediction Justice Level Prediction 67.4% experts 58% experts From the 68 Included Cases for the 2002-2003 Supreme Court Term
  76. 76. these experts probably overfit
  77. 77. they fit to the noise and not the signal
  78. 78. we need to evaluate experts and somehow benchmark their expertise
  79. 79. from a pure forecasting standpoint
  80. 80. the best known SCOTUS predictor is
  81. 81. the law version of superforecasting
  82. 82. Crowds
  83. 83. crowds
  84. 84. https://fantasyscotus.lexpredict.com/case/list/ We can generate Crowd Sourced Predictions
  85. 85. however, not all members of crowd are made equal
  86. 86. we maintain a ‘supercrowd’ which is the top n% of predictors up to time t
  87. 87. the ‘supercrowd’ outperforms the overall crowd (and the best single player)
  88. 88. not enough crowd based decision making in institutions
  89. 89. here is a commercial offering
  90. 90. design to unlock untapped expertise in organizations
  91. 91. https://lexsemble.com/
  92. 92. https://lexsemble.com/
  93. 93. Algorithms
  94. 94. Black Reed Frankfurter Douglas Jackson Burton Clark Minton Warren Harlan Brennan Whittaker Stewart White Goldberg Fortas Marshall Burger Blackmun Powell Rehnquist Stevens OConnor Scalia Kennedy Souter Thomas Ginsburg Breyer Roberts Alito Sotomayor Kagan 1953 1963 1973 1983 1993 2003 2013 9-0 Reverse 8-1, 7-2, 6-3 19 19 19 19 19 20 20 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 - Reverse 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 - 8-1, 7-2, 6-3 9-0 19 19 19 19 19 20 20 algorithms
  95. 95. we have developed an algorithm that we call {Marshall}+ random forest
  96. 96. Benchmarking since 1953 + Using only data available prior to the decision Mean Court Direction [FE] Mean Court Direction 10 [FE] Mean Court Direction Issue [FE] Mean Court Direction Issue 10 [FE] Mean Court Direction Petitioner [FE] Mean Court Direction Petitioner 10 [FE] Mean Court Direction Respondent [FE] Mean Court Direction Respondent 10 [FE] Mean Court Direction Circuit Origin [FE] Mean Court Direction Circuit Origin 10 [FE] Mean Court Direction Circuit Source [FE] Mean Court Direction Circuit Source 10 [FE] Difference Justice Court Direction [FE] Abs. Difference Justice Court Direction [FE] Difference Justice Court Direction Issue [FE] Abs. Difference Justice Court Direction Issue [FE] Z Score Difference Justice Court Direction Issue [FE] Difference Justice Court Direction Petitioner [FE] Abs. Difference Justice Court Direction Petitioner [FE] Difference Justice Court Direction Respondent [FE] Abs. Difference Justice Court Direction Respondent [FE] Z Score Justice Court Direction Difference [FE] Justice Lower Court Direction Difference [FE] Justice Lower Court Direction Abs. Difference [FE] Justice Lower Court Direction Z Score [FE] Z Score Justice Lower Court Direction Difference [FE] Agreement of Justice with Majority [FE] Agreement of Justice with Majority 10 [FE] Difference Court and Lower Ct Direction [FE] Abs. Difference Court and Lower Ct Direction [FE] Z-Score Difference Court and Lower Ct Direction [FE] Z-Score Abs. Difference Court and Lower Ct Direction [FE] Justice [S] Justice Gender [FE] Is Chief [FE] Party President [FE] Natural Court [S] Segal Cover Score [SC] Year of Birth [FE] Mean Lower Court Direction Circuit Source [FE] Mean Lower Court Direction Circuit Source 10 [FE] Mean Lower Court Direction Issue [FE] Mean Lower Court Direction Issue 10 [FE] Mean Lower Court Direction Petitioner [FE] Mean Lower Court Direction Petitioner 10 [FE] Mean Lower Court Direction Respondent [FE] Mean Lower Court Direction Respondent 10 [FE] Mean Justice Direction [FE] Mean Justice Direction 10 [FE] Mean Justice Direction Z Score [FE] Mean Justice Direction Petitioner [FE] Mean Justice Direction Petitioner 10 [FE] Mean Justice Direction Respondent [FE] Mean Justice Direction Respondent 10 [FE] Mean Justice Direction for Circuit Origin [FE] Mean Justice Direction for Circuit Origin 10 [FE] Mean Justice Direction for Circuit Source [FE] Mean Justice Direction for Circuit Source 10 [FE] Mean Justice Direction by Issue [FE] Mean Justice Direction by Issue 10 [FE] Mean Justice Direction by Issue Z Score [FE] Admin Action [S] Case Origin [S] Case Origin Circuit [S] Case Source [S] Case Source Circuit [S] Law Type [S] Lower Court Disposition Direction [S] Lower Court Disposition [S] Lower Court Disagreement [S] Issue [S] Issue Area [S] Jurisdiction Manner [S] Month Argument [FE] Month Decision [FE] Petitioner [S] Petitioner Binned [FE] Respondent [S] Respondent Binned [FE] Cert Reason [S] Mean Agreement Level of Current Court [FE] Std. Dev. of Agreement Level of Current Court [FE] Mean Current Court Direction Circuit Origin [FE] Std. Dev. Current Court Direction Circuit Origin [FE] Mean Current Court Direction Circuit Source [FE] Std. Dev. Current Court Direction Circuit Source [FE] Mean Current Court Direction Issue [FE] Z-Score Current Court Direction Issue [FE] Std. Dev. Current Court Direction Issue [FE] Mean Current Court Direction [FE] Std. Dev. Current Court Direction [FE] Mean Current Court Direction Petitioner [FE] Std. Dev. Current Court Direction Petitioner [FE] Mean Current Court Direction Respondent [FE] Std. Dev. Current Court Direction Respondent [FE] 0.00781 0.00205 0.00283 0.00604 0.00764 0.00971 0.00793 TOTAL 0.04403 Justice and Court Background Information Case Information 0.00978 0.00971 0.00845 0.00953 0.01015 0.01370 0.01190 0.01125 0.00706 0.01541 0.01469 0.00595 0.02014 0.01349 0.01406 0.01199 0.01490 0.01179 0.01408 TOTAL 0.22814 Overall Historic Supreme Court Trends 0.00988 0.01997 0.01546 0.00938 0.00863 0.00904 0.00875 0.00925 0.00791 0.00864 0.00951 0.01017 TOTAL 0.12663 Lower Court Trends 0.00962 0.01017 0.01334 0.00933 0.00949 0.00874 0.00973 0.00900 TOTAL 0.07946 0.00955 0.00936 0.00789 0.00850 0.00945 0.01021 0.01469 0.00832 0.01266 0.00918 0.00942 0.00863 0.00894 0.00882 0.00888 Current Supreme Court Trends TOTAL 0.14456 Individual Supreme Court Justice Trends 0.01248 0.01530 0.00826 0.00732 0.01027 0.00724 0.01030 0.00792 0.00945 0.00891 0.00970 0.01881 0.00950 0.00771 TOTAL 0.14323 0.01210 0.00929 0.01167 0.00968 0.01055 0.00705 0.00708 0.00690 0.00699 0.01280 0.01922 0.02494 0.01126 0.00992 0.00866 0.01483 0.01522 0.01199 0.01217 0.01150 TOTAL 0.23391 Differences in Trends
  97. 97. Total Cases Predicted Total Votes Predicted 7,700 68,964
  98. 98. Justice Prediction Case Prediction 70.9% accuracy 69.6% accuracy From 1953 - 2014
  99. 99. Our algorithm is a special version of random forest Black Reed Frankfurter Douglas Jackson Burton Clark Minton Warren Harlan Brennan Whittaker Stewart White Goldberg Fortas Marshall Burger Blackmun Powell Rehnquist Stevens OConnor Scalia Kennedy Souter Thomas Ginsburg Breyer Roberts Alito Sotomayor Kagan 1953 1963 1973 1983 1993 2003 2013 9-0 Reverse 8-1, 7-2, 6-3 19 19 19 19 19 20 20 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 - Reverse 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 - 8-1, 7-2, 6-3 9-0 19 19 19 19 19 20 20 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2463244 http://arxiv.org/abs/1407.6333 available at Revise and Resubmit @ PloS One
  100. 100. Experts, Crowds, Algorithms
  101. 101. For most problems ... ensembles of these streams outperform any single stream
  102. 102. Humans + Machines
  103. 103. Humans + Machines >
  104. 104. Humans + Machines Humans or Machines >
  105. 105. Ensembles come in various forms
  106. 106. Here is a well known example
  107. 107. Poll Aggregation is one form of ensemble where the learning question is to determine how much weight (if any) to assign to each individual poll
  108. 108. poll weighting
  109. 109. A Visual Depiction of How to build an ensemble method in our judicial prediction example
  110. 110. expert crowd algorithm ensemble method learning problem is to discover when to use a given stream of intelligence
  111. 111. expert crowd algorithm via back testing we can learn the weights to apply for particular problems ensemble method learning problem is to discover when to use a given stream of intelligence
  112. 112. With that background 
 I would like to look over the #LegalHorizon
  113. 113. Lots of folks ask me what is next in legal analytics …
  114. 114. A big part of the answer comes from one of the most dominant vectors in tech
  115. 115. both those in positions of leadership and those in technical positions need to take stock
  116. 116. #MLaaS and the Enterprise Open Source Movement Part IV< >
  117. 117. IBM WATSON First major effort at #MLaaS Machine Learning as a Service
  118. 118. IBM Watson is MLaaS and it would have purported to be among the biggest stories in tech over the past few years
  119. 119. Turns out things would layout in a slightly different fashion …
  120. 120. IBM Watson (per se) IBM Watson (as early #MLaaS) vs.
  121. 121. the democratization of machine learning is underway
  122. 122. Emerging Business Model - Machine Learning as a Service #MLaaS
  123. 123. The Cloud Wars
  124. 124. Commercial Examples
  125. 125. Machine Learning as a Service #MLaaS
  126. 126. Machine Learning as a Service #MLaaS
  127. 127. Machine Learning as a Service #MLaaS
  128. 128. Machine Learning as a Service #MLaaS
  129. 129. But wait there is more …
  130. 130. Machine Learning as a Service #MLaaS
  131. 131. Machine Learning as a Service #MLaaS Enterprise Open Source Movement #OpenSource +
  132. 132. Enterprise Open Source Movement #OpenSource
  133. 133. https://techcrunch.com/2016/06/19/the-next-wave-in-software-is-open-adoption-software/
  134. 134. Part V< > The Last Mile Problem and the New Dimension of Competition
  135. 135. historically one needed to build the full stack (i.e end to end) for an application
  136. 136. Standing on 
 the Shoulders of Giants
  137. 137. The (Emerging) Last Mile Problem in (Legal) Analytics
  138. 138. Off the Shelf #MLaaS, etc. (perhaps with some configuration and/or customization) Unique Domain Specific Offering
  139. 139. MLaas + Open Source Decreases Cost of Production Lowers the Cost of Protoyping
  140. 140. The New Ball Game
  141. 141. Workflow Across the Machine Learning Landscape
  142. 142. Piece together the combinations of 
 #MLaaS + open source
  143. 143. to build enterprise applications which are unique combinations drawn from across the #MLaaS / open source spectrum
  144. 144. Three Implications for #LegalAnalytics #LegalTech #LegalAI Part VI< >
  145. 145. Implication #1< >
  146. 146. every organization in law needs a data strategy
  147. 147. Capture, Clean, Regularize Data to support a range of tasks
  148. 148. Deploy Data for Specific Enterprise Applications Develop a data roadmap
  149. 149. Implication #2< >
  150. 150. every organization in law needs a relevant human captial #LegalAnalytics
  151. 151. Opening the Human Capital Bottleneck
  152. 152. Probably going to need homegrow your own talent
  153. 153. http://www.quantitativemethodsclass.com/Professor Daniel Martin Katz Intro Class
  154. 154. http://www.legalanalyticscourse.com/Professor Daniel Martin Katz Professor Michael J. Bommarito II Advanced Class
  155. 155. Implication #3< >
  156. 156. First Wave vs. Second Wave Legal Tech
  157. 157. Second Movers can catch up faster …
  158. 158. Second Movers need less capital …
  159. 159. Second Movers who start now will have lower fixed costs …
  160. 160. probably will not need to go for a series z round of funding
  161. 161. In Conclusion< >
  162. 162. Prediction on the #LegalHorizon
  163. 163. Prediction More Legal Tech More Legal Analytics
  164. 164. Leveraging (in part) …
  165. 165. #MLaaS Machine Learning as a Service
  166. 166. Daniel Martin Katz @ computational computationallegalstudies.com lexpredict.com danielmartinkatz.com illinois tech - chicago kent college of law@

×