Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Artificial Intelligence and Law - 
A Primer

21,916 views

Published on

Artificial Intelligence in Law (and beyond) including Machine Learning as a Service, Quantitative Legal Prediction / Legal Analytics, Experts + Crowds + Algorithms

Published in: Law

Artificial Intelligence and Law - 
A Primer

  1. 1. 
 a six part primer artificial intelligence in law (and beyond) daniel martin katz blog | ComputationalLegalStudies.com corp | LexPredict.com page | DanielMartinKatz.com edu | chicago kent college of law lab | TheLawLab.com [ a.i. + law ]
  2. 2. There has been lots of recent interest in applying artificial intelligence to law
  3. 3. and there is a bit of confusion as to where we stand today and where we are headed
  4. 4. A.I. in Law A Quick Reset Part I< >
  5. 5. data driven AI rules based AI Competing Orientations in Artificial Intelligence
  6. 6. expert systems Computational Law Data Driven Rules Based prediction models and methods network analytic methods natural language processing self executing law visual law computable codes
  7. 7. we see a decent amount of rules based AI in legal industry
  8. 8. Three Examples of Rules Based A.I.
  9. 9. tax preparation software
  10. 10. Rules Based A.I.
  11. 11. Rules Based A.I.
  12. 12. Among other things Neota has been used to create decision trees to support lawyers / non lawyers
  13. 13. What do I do if there has been An issue in Human Resources ? A potential FCPA violation? A potential data breach?
  14. 14. Decision Trees are a step by step memorialization of best practices
  15. 15. At my home institution - Illinois Tech Chicago-Kent Law has a platform that allows individuals to automate various legal forms, etc.
  16. 16. used by a variety of legal aid organizations
  17. 17. JUSTICE GAP 80%Civil legal needs of low-income people in the U.S. go unmet For every 1 person served in an LSC- funded program, at least 1 person is turned away
  18. 18. LSC TECH SUMMIT “to explore the potential of technology to move the United States toward providing some form of effective assistance to 100% of persons otherwise unable to afford an attorney for dealing with essential civil legal needs.”
  19. 19. LSC TECH SUMMIT “to explore the potential of technology to move the United States toward providing some form of effective assistance to 100% of persons otherwise unable to afford an attorney for dealing with essential civil legal needs.” some form of effective assistance to 100% !
  20. 20. LSC TECH SUMMIT Technology leading to greater access to legal information!
  21. 21. A2J AUTHOR www.a2jauthor.org
  22. 22. PROCESS Guided Interview Completed Document
  23. 23. LOGIC
  24. 24. DECISION TREE
  25. 25. JUSTICE & TECHNOLOGY PRACTICUM STUDENT WORK ! Fieldwork (e.g. Self-Help Web Center) Scope document Research memo Storyboard A2J Guided Interview & HotDocs template Final presentation Professor Ron Staudt IIT Chicago- Kent College of Law Engage community partners: legal aid organizations, courts
  26. 26. A2J AUTHOR COURSE PROJECT a2jclinic.classcaster.net
  27. 27. Used over 3.5 Million times 2.1 Million Documents generated IMPACT
  28. 28. Expert Systems 
 (together with data) 
 will eventually become Chatbots …
  29. 29. Client Intake Client Acquisition More Seamless Client Interaction via Tech Platform Providing Legal Information to Non-Lawyers in Large Organizations
  30. 30. so although we see a decent amount of rules based AI in legal industry
  31. 31. I am pretty bearish on Rules Based A.I. for most applications …
  32. 32. my views are informed by the history of A.I. in general
  33. 33. lots of issues with expert systems and/or rules based A.I. (without data or an evolutionary dynamic)
  34. 34. rules based A.I. data driven A.I. 1980’s, 1990’s, Early 2000’s >
  35. 35. rules based A.I. data driven A.I. 1980’s, 1990’s, Early 2000’s > rules based A.I. data driven A.I. 2005 - Present < ~
  36. 36. Ultimately we are trying to learn the rules / dynamics that underlie some class of activity
  37. 37. With that understanding we want to be able to mimic / predict
  38. 38. There are some use cases for Rules Based AI / Expert Systems
  39. 39. Practically ZERO Top Tier Companies Building Expert Systems
  40. 40. expert systems Computational Law Data Driven Rules Based prediction models and methods network analytic methods natural language processing self executing law visual law computable codes
  41. 41. A.I. State of the Art
  42. 42. A.I. State of the Art purely data centric
  43. 43. A.I. State of the Art purely data centric augment expert forecasts w/ data
  44. 44. iterative data < > rules A.I. State of the Art purely data centric augment expert forecasts w/ data
  45. 45. For example, I like Chatbots because they end up being a massive data collection effort … iterative data < > rules
  46. 46. #BigData pan everything #Hashtag Part II< >
  47. 47. But as a general matter …
  48. 48. In the Rules vs. Data Debate in A.I.
  49. 49. Data Won the War (Terms of Surrender are Available)
  50. 50. #DataScience are already influencing our lives in a variety of meaningful ways #BigData #Analytics #A.I.
  51. 51. To date, the most successful commercial applications have massive returns to scale and aim for cross societal payoffs…
  52. 52. Medicine
  53. 53. Finance
  54. 54. Logistics
  55. 55. Agriculture
  56. 56. Transportation
  57. 57. Retail
  58. 58. What is powering the A.I. revolution?
  59. 59. Increasing Computing Power Decreasing Data Storage Costs
  60. 60. Moore’s law !
  61. 61. Kryder’s law !
  62. 62. And How Big is ‘Big’?
  63. 63. How Much Data Is a Petabyte?
  64. 64. How Much Data Is a Petabyte?
  65. 65. How Much Data Is a Petabyte?
  66. 66. How Much Data Is a Petabyte?
  67. 67. Given large fixed costs
  68. 68. Given large fixed costs infrastructure + human capital (data scientists)
  69. 69. harder to successfully deploy high quality enterprise applications for relatively narrow (sub)verticals
  70. 70. The Rise of #LegalAnalytics Part II< >
  71. 71. Law is a relatively small vertical and there is lots of diversity among tasks lawyers undertake …
  72. 72. in addition there is a borderline pathological numerophobia among lawyers
  73. 73. plus the implicit (explicit) challenge of partnership as the dominant form of the organization within our market
  74. 74. taken together this has challenged the deployment 
 of analytics in legal
  75. 75. Analytics / Quant Legal Prediction has come to law Notwithstanding these head winds—
  76. 76. #LegalAnalytics Quantitative Legal Prediction
  77. 77. #LegalAnalytics Quantitative Legal Prediction
  78. 78. #LegalAnalytics Quantitative Legal Prediction
  79. 79. #LegalAnalytics Quantitative Legal Prediction
  80. 80. #LegalAnalytics Quantitative Legal Prediction
  81. 81. #LegalAnalytics Quantitative Legal Prediction
  82. 82. Some Commercial Applications
  83. 83. In a real sense, this represents just a narrow set of products
  84. 84. #ContractAnalytics Quantitative Legal Prediction
  85. 85. #JudicialAnalytics Quantitative Legal Prediction
  86. 86. #PredictiveCoding #E-Discovery Quantitative Legal Prediction
  87. 87. General Counsels as Legal Procurement Specialists TyMetrix/ELM - Using $50 billion+ in Legal Spend Data to Help GC’s Look for Arbitrage Opportunities, Value Propositions in Hiring Law Firms #LegalSpendAnalytics Quantitative Legal Prediction
  88. 88. #LegalAnalytics Quantitative Legal Prediction https://lexsemble.com/
  89. 89. #NegotiationAnalytics Quantitative Legal Prediction
  90. 90. Here are just a subset of the tasks that we are trying to accomplish in legal …
  91. 91. #Predict Case Outcomes Data Driven Legal Underwriting
  92. 92. #Predict Case Outcomes Data Driven Legal Underwriting #Predict Legal Costs Data Driven Legal Operations
  93. 93. #Predict Relevant Documents #Predict Case Outcomes Data Driven Legal Underwriting Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Legal Costs Data Driven Legal Operations
  94. 94. #Predict Relevant Documents #Predict Case Outcomes Data Driven Legal Underwriting Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Legal Costs #Predict Rogue Behavior Data Driven Legal Operations Data Driven Compliance
  95. 95. #Predict Relevant Documents #Predict Case Outcomes Data Driven Legal Underwriting Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Legal Costs Data Driven Legal Operations Data Driven Compliance #Predict Contract Terms/Outcomes Data Driven Transactional Work #Predict Rogue Behavior
  96. 96. The Three Forms of (Legal) Prediction Part III< >
  97. 97. A Deeper Dive on Predicting Predicting Case Outcomes (other problems can be solved using similar methods)
  98. 98. Supreme Court of United States #PredictSCOTUS
  99. 99. There are only 3 ways 
 to predict something Experts Crowds Algorithms
  100. 100. Experts
  101. 101. Columbia Law Review October, 2004 Theodore W. Ruger, Pauline T. Kim, Andrew D. Martin, Kevin M. Quinn Legal and Political Science Approaches to Predicting Supreme Court Decision Making The Supreme Court Forecasting Project:
  102. 102. experts
  103. 103. Case Level Prediction Justice Level Prediction 67.4% experts 58% experts From the 68 Included Cases for the 2002-2003 Supreme Court Term
  104. 104. these experts probably overfit
  105. 105. they fit to the noise and not the signal
  106. 106. if this were finance this would be trading worse than S&P500
  107. 107. #NoiseTrading
  108. 108. #BuffetChallenge
  109. 109. #BuffetChallenge
  110. 110. like many other forms human endeavor law is full of 
 noise predictors …
  111. 111. we need to evaluate experts and somehow benchmark their expertise
  112. 112. from a pure forecasting standpoint
  113. 113. the best known SCOTUS predictor is
  114. 114. the law version of superforecasting
  115. 115. Crowds
  116. 116. crowds
  117. 117. https://fantasyscotus.lexpredict.com/case/list/ We can generate Crowd Sourced Predictions
  118. 118. Just like the Market the Crowd is collectively terrible … < No Alpha >
  119. 119. however, not all members of crowd are made equal
  120. 120. we maintain a ‘supercrowd’ which is the top n% of predictors up to time t
  121. 121. the ‘supercrowd’ outperforms the overall crowd (and the best single player)
  122. 122. For the 2015-2016 term
  123. 123. not enough crowd based decision making in institutions (law included)
  124. 124. “Software developers were asked on two separate days to estimate the completion time for a given task, the hours they projected differed by 71%, on average. W h e n p a t h o l o g i s t s m a d e t wo assessments of the severity of biopsy results, the correlation between their ratings was only .61 (out of a perfect 1.0), indicating that they made inconsistent diagnoses quite frequently. Judgments made by different people are even more likely to diverge.”
  125. 125. not enough crowd based decision making in institutions
  126. 126. here is a commercial offering
  127. 127. design to unlock untapped expertise in organizations
  128. 128. https://lexsemble.com/
  129. 129. Brief Aside About Crowd Sourced Prediction #LegalCrowdSourcing
  130. 130. (most pundits did not identify as a serious candidate him until mid-January 2017) Neil Gorsuch was #1 o n o u r F a n t a s y Platform 12 Days after Donald Trump was elected President (i.e Nov 20)
  131. 131. #FantasySCOTUS
  132. 132. Algorithms
  133. 133. we have developed an algorithm that we call {Marshall}+ random forest
  134. 134. Columbia Law Review October, 2004 Theodore W. Ruger, Pauline T. Kim, Andrew D. Martin, Kevin M. Quinn Legal and Political Science Approaches to Predicting Supreme Court Decision Making The Supreme Court Forecasting Project:
  135. 135. Ruger, et al (2004) relied upon Brieman(1984) (as partially shown below)
  136. 136. Leo Brieman moved away from CART in Brieman (2001)
  137. 137. Breiman, L.(2001). Random forests. Machine learning, 45(1), 5-32. Published in Machine Learning (A Springer Science Journal)
  138. 138. One well-known problem with standard classification trees is their tendency toward overfitting
  139. 139. http://machinelearning202.pbworks.com/w/file/fetch/37597425/ performanceCompSupervisedLearning-caruana.pdf Random Forest (particularly with special config/ optimization) have proven to be unreasonably effective
  140. 140. Random forest is an approach to aggregate weak learners into collective strong learners (using a combo of bagging and random substrates) (think of it as crowd sourcing of models)
  141. 141. Our algorithm is a special version of random forest (time evolving) http://journals.plos.org/ plosone/article?id=10.1371/ journal.pone.0174698 available at RESEARCH ARTICLE A general approach for predicting the behavior of the Supreme Court of the United States Daniel Martin Katz1,2 *, Michael J. Bommarito II1,2 , Josh Blackman3 1 Illinois Tech - Chicago-Kent College of Law, Chicago, IL, United States of America, 2 CodeX - The Stanford Center for Legal Informatics, Stanford, CA, United States of America, 3 South Texas College of Law Houston, Houston, TX, United States of America * dkatz3@kentlaw.iit.edu Abstract Building on developments in machine learning and prior work in the science of judicial pre- diction, we construct a model designed to predict the behavior of the Supreme Court of the United States in a generalized, out-of-sample context. To do so, we develop a time-evolving random forest classifier that leverages unique feature engineering to predict more than 240,000 justice votes and 28,000 cases outcomes over nearly two centuries (1816-2015). Using only data available prior to decision, our model outperforms null (baseline) models at both the justice and case level under both parametric and non-parametric tests. Over nearly two centuries, we achieve 70.2% accuracy at the case outcome level and 71.9% at the jus- tice vote level. More recently, over the past century, we outperform an in-sample optimized null model by nearly 5%. Our performance is consistent with, and improves on the general level of prediction demonstrated by prior work; however, our model is distinctive because it can be applied out-of-sample to the entire past and future of the Court, not a single term. Our results represent an important advance for the science of quantitative legal prediction and portend a range of other potential applications. Introduction As the leaves begin to fall each October, the first Monday marks the beginning of another term for the Supreme Court of the United States. Each term brings with it a series of challenging, important cases that cover legal questions as diverse as tax law, freedom of speech, patent law, administrative law, equal protection, and environmental law. In many instances, the Court’s decisions are meaningful not just for the litigants per se, but for society as a whole. Unsurprisingly, predicting the behavior of the Court is one of the great pastimes for legal and political observers. Every year, newspapers, television and radio pundits, academic jour- nals, law reviews, magazines, blogs, and tweets predict how the Court will rule in a particular case. Will the Justices vote based on the political preferences of the President who appointed them or form a coalition along other dimensions? Will the Court counter expectations with an unexpected ruling? PLOS ONE | https://doi.org/10.1371/journal.pone.0174698 April 12, 2017 1 / 18 a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 OPEN ACCESS Citation: Katz DM, Bommarito MJ, II, Blackman J (2017) A general approach for predicting the behavior of the Supreme Court of the United States. PLoS ONE 12(4): e0174698. https://doi. org/10.1371/journal.pone.0174698 Editor: Luı´s A. Nunes Amaral, Northwestern University, UNITED STATES Received: January 17, 2017 Accepted: March 13, 2017 Published: April 12, 2017 Copyright: © 2017 Katz et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: Data and replication code are available on Github at the following URL: https://github.com/mjbommar/scotus-predict-v2/. Funding: The author(s) received no specific funding for this work. Competing interests: All Authors are Members of a LexPredict, LLC which provides consulting services to various legal industry stakeholders. We received no financial contributions from LexPredict or anyone else for this paper. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
  142. 142. From a Pure Machine Learning Perspective — Much of this is not novel EXCEPT the time evolving element of the Random Forest
  143. 143. https://github.com/mjbommar/ scotus-predict-v2/
  144. 144. 243,882 28,009 Case Outcomes JusticeVotes Final Version of #PredictSCOTUS 1816-2015 http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0174698
  145. 145. Final Version of #PredictSCOTUS 1816-2015 case accuracy 70.2% 71.9% justice accuracy http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0174698
  146. 146. Experts, Crowds, Algorithms
  147. 147. http://www.sciencemag.org/news/ 2017/05/artificial-intelligence-prevails- predicting-supreme-court-decisions Professor Katz noted that in the long term …“We believe the blend of experts, crowds, and algorithms is the secret sauce for the whole thing.” May 2nd 2017
  148. 148. For most problems ... ensembles of these streams outperform any single stream
  149. 149. the non-trivial question is how to optimally assemble such streams for particular problems
  150. 150. Humans + Machines
  151. 151. Humans + Machines >
  152. 152. Humans + Machines Humans or Machines >
  153. 153. Here is what we are working on right now …
  154. 154. expert forecast crowd forecast learning problem is to discover how to blend streams of intelligence algorithm forecast ensemble method ensemble model
  155. 155. expert forecast crowd forecast learning problem is to discover how to blend streams of intelligence algorithm forecast ensemble method ensemble model via back testing we can learn the weights to apply for particular problems
  156. 156. By the way, you might ask why does one care about marginal improvements in prediction ? #Fin(Legal)Tech
  157. 157. Given our ability to offer forecasts of judicial outcomes, we wondered if this information could inform an event driven trading strategy ?
  158. 158. Revise + Resubmit @ http://arxiv.org/abs/1508.05751 available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726
  159. 159. The Killer App is Financialization #Fin(Legal)Tech
  160. 160. Once you can engage in #LegalPrediction, that immediately leads to thoughts about financialization #Fin(Legal)Tech
  161. 161. https://computationallegalstudies.com/2016/02/27/fin-legal-tech-laws- future-from-finances-past-an-expanded-version-of-the-deck/ Here is a full presentation of this idea … (it has overlap)
  162. 162. Claim: fin(tech) offers lessons for many areas in law
  163. 163. FinLegalTechConference.comNovember 4, 2016
  164. 164. Fin(Legal)Tech Conference finlegaltechconference.com @Illinois Tech - Chicago Kent College of Law
  165. 165. Fin(Legal)Tech Conference finlegaltechconference.com @Illinois Tech - Chicago Kent College of Law
  166. 166. Fin(Legal)Tech Conference finlegaltechconference.com @Illinois Tech - Chicago Kent College of Law Fin(Legal)Tech Conference finlegaltechconference.com @Illinois Tech - Chicago Kent College of Law Fin(Legal)Tech Conference finlegaltechconference.com @Illinois Tech - Chicago Kent College of Law
  167. 167. With that background 
 I would like to look over the #LegalHorizon
  168. 168. Lots of folks ask me what is next in legal analytics …
  169. 169. A big part of the answer comes from one of the most dominant vectors in tech
  170. 170. both those in positions of leadership and those in technical positions need to take stock
  171. 171. #MLaaS and the Enterprise Open Source Movement Part IV< >
  172. 172. IBM WATSON First major effort at #MLaaS Machine Learning as a Service
  173. 173. IBM Watson is MLaaS and it would have purported to be among the biggest stories in tech over the past few years
  174. 174. Turns out things would layout in a slightly different fashion …
  175. 175. IBM Watson (per se) IBM Watson (as early #MLaaS) vs.
  176. 176. the democratization of machine learning is underway
  177. 177. Emerging Business Model - Machine Learning as a Service #MLaaS
  178. 178. The Cloud Wars
  179. 179. Commercial Examples
  180. 180. Machine Learning as a Service #MLaaS
  181. 181. Machine Learning as a Service #MLaaS
  182. 182. Machine Learning as a Service #MLaaS
  183. 183. Machine Learning as a Service #MLaaS
  184. 184. But wait there is more …
  185. 185. Machine Learning as a Service #MLaaS
  186. 186. Machine Learning as a Service #MLaaS Enterprise Open Source Movement #OpenSource +
  187. 187. Enterprise Open Source Movement #OpenSource
  188. 188. https://techcrunch.com/2016/06/19/the-next-wave-in-software-is-open-adoption-software/
  189. 189. Part V< > The Last Mile Problem and the New Dimension of Competition
  190. 190. historically one needed to build the full stack (i.e end to end) for an application
  191. 191. Standing on 
 the Shoulders of Giants
  192. 192. The (Emerging) Last Mile Problem in (Legal) Analytics
  193. 193. Off the Shelf #MLaaS, etc. (perhaps with some configuration and/or customization) Unique Domain Specific Offering
  194. 194. MLaas + Open Source Decreases Cost of Production Lowers the Cost of Protoyping
  195. 195. The New Ball Game
  196. 196. Workflow Across the Machine Learning Landscape
  197. 197. Piece together the combinations of 
 #MLaaS + open source
  198. 198. to build enterprise applications which are unique combinations drawn from across the #MLaaS / open source spectrum
  199. 199. We are beginning to see the first wave of #MLaaS Implementation Companies …
  200. 200. https://computationallegalstudies.com/2017/05/07/machine-learning- service-mlaas-ecosystem-grows-bonsai-mlaas-implentation-company/ “AI startup Bonsai has raised $7.6 million to grow its platform that simplifies open-source machine learning library TensorFlow to help businesses construct their own artificial intelligence models and incorporate AI into their business.”
  201. 201. Three Implications for #LegalAnalytics #LegalTech #LegalAI Part VI< >
  202. 202. Implication #1< >
  203. 203. every organization in law needs a data strategy
  204. 204. Capture, Clean, Regularize Data to support a range of tasks
  205. 205. Deploy Data for Specific Enterprise Applications Develop a data roadmap
  206. 206. Implication #2< >
  207. 207. every organization in law needs relevant human capital #LegalAnalytics
  208. 208. Opening the Human Capital Bottleneck
  209. 209. Probably going to need homegrow your own talent
  210. 210. http://www.quantitativemethodsclass.com/Professor Daniel Martin Katz Intro Class
  211. 211. http://www.legalanalyticscourse.com/Professor Daniel Martin Katz Professor Michael J. Bommarito II Advanced Class
  212. 212. Implication #3< >
  213. 213. First Wave vs. Second Wave Legal Tech
  214. 214. Second Movers can catch up faster …
  215. 215. Second Movers need less capital …
  216. 216. Second Movers who start now will have lower fixed costs …
  217. 217. probably will not need to go for a series z round of funding
  218. 218. Major Implication The Best Legal Tech is Yet Be Built …
  219. 219. In Conclusion< >
  220. 220. Prediction on the #LegalHorizon
  221. 221. Prediction More Legal Tech More Legal Analytics
  222. 222. Leveraging (in part) …
  223. 223. #MLaaS Machine Learning as a Service
  224. 224. LexPredict.com
  225. 225. thelawlab.com
  226. 226. ComputationalLegalStudies.com BLOG
  227. 227. @ computational
  228. 228. Daniel Martin Katz @ computational computationallegalstudies.com lexpredict.com danielmartinkatz.com illinois tech - chicago kent college of law@ thelawlab.com

×