Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution- NYC

1. daniel martin katz michael j bommarito adjunct professor @ university of michigan associate professor of law @ illinois tech - chicago kent co-founder @ LexPredict co-founder @ LexPredict Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry

2. Forum on Legal Evolution NYC

3. in the legal industry there already is better corn

4. "We always overestimate the change that will occur in the next two years and underestimate the change that will occur in the next ten” - bill gates

5. today’s focus is primarily legal analytics + process engineering © daniel martin katz michael j bommarito

6. before providing some concrete examples - some broad thoughts ... © daniel martin katz michael j bommarito

7. three faces of innovation in legal © daniel martin katz michael j bommarito

8. (1) lawyers for innovators / entrepreneurs © daniel martin katz michael j bommarito

9. © daniel martin katz michael j bommarito what most lawyers and law schools think of as “Law+Entrepreneurship" (1) lawyers for innovators / entrepreneurs

10. (2) lawyers as innovators - substance © daniel martin katz michael j bommarito

11. poison pill - “the most important innovation in corporate law since Samuel Calvin Tate Dodd invented the trust for John D. Rockefeller and Standard Oil in 1879” © daniel martin katz michael j bommarito (2) lawyers as innovators - substance

12. emerging areas - 3D Printing, Driverless Cars, Augmented Reality, Data Breach, Big Data+Privacy, etc. Drones, Internet of Things, CyberSecurity, © daniel martin katz michael j bommarito (2) lawyers as innovators - substance

13. (3) lawyers as innovators - business/process © daniel martin katz michael j bommarito

14. innovation directed toward transforming the practice of law © daniel martin katz michael j bommarito (3) lawyers as innovators - business/process

15. © daniel martin katz michael j bommarito

16. © daniel martin katz michael j bommarito there are different ways that organizations are innovating on the third face

17. {Law Substantive Legal Expertise Analytics Platform AI Computing Process Mapping User Experience Design Thinking Business Models Regulation Marketing + Tech + Design TM + Delivery} © daniel martin katz michael j bommarito

19. © daniel martin katz michael j bommarito some traditional law firms have been very aggressive

21. © daniel martin katz michael j bommarito but most of the innovation is Lex.Startup

22. © daniel martin katz michael j bommarito Lex.Startup is beginning to take hold

23. 15 2009 Lex.Startup

24. 15 2009 Lex.Startup

25. 15 425+ 2009 2014 Law or Legal Related Companies* as highlighted by Josh Kubicki @ ReInventLaw London 2013 Lex.Startup

26. © daniel martin katz michael j bommarito So what are these folks doing?

27. R + D Function in the Legal Industry © daniel martin katz michael j bommarito

28. We Could Imagine a World Where Law Firms Did the R+D for the Industry © daniel martin katz michael j bommarito

29. But That Has (Mostly) Proven Illusive © daniel martin katz michael j bommarito

30. © daniel martin katz michael j bommarito Lex.Startup is undertaking that function

31. © daniel martin katz michael j bommarito Here are the specific approaches that are being undertaken

32. © daniel martin katz michael j bommarito Some organizations are doing more than one

33. © daniel martin katz michael j bommarito labor arbitrage

34. © daniel martin katz michael j bommarito labor arbitrage process/tech arbitrage

35. © daniel martin katz michael j bommarito labor arbitrage process/tech arbitrage regulatory arbitrage

36. © daniel martin katz michael j bommarito labor arbitrage process/tech arbitrage regulatory arbitrage design as the ultimate bespoke

37. © daniel martin katz michael j bommarito labor arbitrage process/tech arbitrage regulatory arbitrage design as the ultimate bespoke predictive analytics

38. © daniel martin katz michael j bommarito could do an individual talk on any of these topics...

42. Predictive Analytics in Law © daniel martin katz michael j bommarito

43. © daniel martin katz michael j bommarito The Data Driven Future of the Legal Industry

44. © daniel martin katz michael j bommarito is already here...

45. © daniel martin katz michael j bommarito 2011

52. 2013

55. © daniel martin katz michael j bommarito Quantitative Legal Prediction - or - How I Learned to Stop Worrying and Start Preparing for the Data Driven Future of the Legal Services Industry Daniel Martin Katz Associate Professor of Law Michigan State University 62 Emory L. J. 909 (2013)

56. © daniel martin katz michael j bommarito Cause and Effect Quantitative Legal Prediction vs.

57. © daniel martin katz michael j bommarito Cause and Effect Quantitative Legal Prediction vs.

58. © daniel martin katz michael j bommarito Machine Learning is the heart of predictive analytics

59. Legal Analytics Professor Daniel Martin Katz Professor Michael J Bommarito II @MSU Law - Winter 2014 © daniel martin katz michael j bommarito

60. Supervised Statistical models Bayesian, e.g., Naïve Bayes Classification Frequentist, e.g., Ordinary Least Squares Neural Networks (NN) Support Vector Machines (SVM) Random Forests (RF) Genetic Algorithms (GA) Semi/Unsupervised Neural Networks (NN) Clustering K-means Hierarchical Radial Basis (RBF) Graph Some Machine Learning Methods © daniel martin katz michael j bommarito

61. http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html © daniel martin katz michael j bommarito

62. classification clustering regression dimension reduction the family of machine learning methods © daniel martin katz michael j bommarito

63. Quick Example of the Methods © daniel martin katz michael j bommarito

64. © daniel martin katz michael j bommarito Adapted from Slides By Victor Lavrenko and Nigel Goddard @ University of Edinburgh Take A LookThese 12

65. © daniel martin katz michael j bommarito 72 Female Human 3 Female Horse 36 Male Human 21 Male Human 67 Male Human 29 Female Human 54 Male Human 44 Male Human 50 Male Human 42 Female Human 6 Male Dog 7 Female Human

66. © daniel martin katz michael j bommarito Classiﬁcation (Supervised Learning) decision boundary female male f( ) Gender?

67. © daniel martin katz michael j bommarito Classiﬁcation (Supervised Learning) decision boundary female male f( ) Gender? Regression (Supervised Learning) #f( ) Age? 723 2 3 67 54 29 42 44 50 7 6 27 44 53 3 68 2 48 10 6 743 4 4

68. © daniel martin katz michael j bommarito Classiﬁcation (Supervised Learning) decision boundary female male f( ) Gender? f( ) Loan Application? Yes Multi Class Classiﬁcation (Supervised Learning) No Maybe Yes Perhaps No Multiclass = Boundary Hyperplane Regression (Supervised Learning) #f( ) Age? 723 2 3 67 54 29 42 44 50 7 6 27 44 53 3 68 2 48 10 6 743 4 4

69. © daniel martin katz michael j bommarito Classiﬁcation (Supervised Learning) decision boundary female male f( ) Gender? f( ) Loan Application? Yes Multi Class Classiﬁcation (Supervised Learning) No Maybe Yes Perhaps No Multiclass = Boundary Hyperplane Regression (Supervised Learning) #f( ) Age? 723 2 3 67 54 29 42 44 50 7 6 27 44 53 3 68 2 48 10 6 743 4 4 Clustering (Unsupervised Learning) Clusterf( ) Group?

70. © daniel martin katz michael j bommarito Regression as a Prediction Tool

71. © daniel martin katz michael j bommarito Regression as a Prediction Tool

72. © daniel martin katz michael j bommarito Standard Linear Regression Can Be Used to Predict a Probability (using LPM, Logit, etc.)

73. © daniel martin katz michael j bommarito Standard Linear Regression Can Be Used to Predict a Quantity

74. © daniel martin katz michael j bommarito Task = Predict the Expected Cost of a Given Legal Service f( ) Cost? # and/or 010 101 001 Regression (Supervised Learning)

75. © daniel martin katz michael j bommarito http://reinventlawchannel.com/ron-gruner-were-on-a-mission/

76. © daniel martin katz michael j bommarito Y = βo +/- β1 ( X1 ) +/- β2 ( X2 ) +/- β3 ( X3 ) +/- β4 ( X3 ) +/- β5 ( X3 ) + ε Y = $151 + $15 ( ) + 161 ( ) + 95 ( ) + 34 ( ) +/- β5 ( ) + ε Per 100 Lawyers If Tier 1 Market is True Partner Status is True Per 10 Years Practice Area

77. © daniel martin katz michael j bommarito Turn Around and Use This Model To Predict Other Lawyers (also Matters, etc.)

78. © daniel martin katz michael j bommarito This Requires a Method to Deal With Changes in Dynamics, etc.

79. © daniel martin katz michael j bommarito This Requires a Method to Update the Model as Time Moves Forward

80. © daniel martin katz michael j bommarito Must Deal With Overﬁtting to the Existing Data

82. Machine Learning and the Future of E-Discovery © daniel martin katz michael j bommarito

83. imagine your client is served with a request for production © daniel martin katz michael j bommarito

84. in random order assume this is the size of the hypothetical document set (emails, memos, etc.)

85. we can sample a subset of the documents

86. we can sample a subset of the documents

87. © daniel martin katz michael j bommarito classification clustering regression dimension reduction

88. © daniel martin katz michael j bommarito classification

90. predictive coding = ~ binary classification © daniel martin katz michael j bommarito

91. © daniel martin katz michael j bommarito LearningTask = Determine Whether a Given Document is Relevant? Relevant Not Relevant f( ) relevance? Binary Classiﬁcation (Supervised Learning) and/or 010 101 001

92. take the sample set as a training set and use human experts © daniel martin katz michael j bommarito

93. the use of the human experts is called “supervised learning” © daniel martin katz michael j bommarito

94. in the simple binary case, ask humans to assign objects to two piles © daniel martin katz michael j bommarito

95. Apply Human Coders © daniel martin katz michael j bommarito

96. yellow = relevant white = non-relevant and return this © daniel martin katz michael j bommarito

97. Non RelevantRelevant © daniel martin katz michael j bommarito

98. Key Insight ... © daniel martin katz michael j bommarito

99. What Allows A Human To Separate These Two Classes of Documents? © daniel martin katz michael j bommarito

100. that precise human process is what “predictive coding” is trying to mimic © daniel martin katz michael j bommarito

101. most vendors are selling a largely undifferentiated product © daniel martin katz michael j bommarito

102. Humans are selecting upon some “features” of the documents © daniel martin katz michael j bommarito

103. to place those documents in their respective bins  (i.e. relevant, non-relevant) © daniel martin katz michael j bommarito

104. features =? text, author, date, other metadata © daniel martin katz michael j bommarito

105. machine learning task is trying to recover (learn) what separates the relevant from the non-relevant documents © daniel martin katz michael j bommarito

106. once we learn the rule / boundary we can apply it to separate the remain documents into the two classes © daniel martin katz michael j bommarito

107. © daniel martin katz michael j bommarito we want to take what we learn here

108. © daniel martin katz michael j bommarito we want to take what we learn here

109. © daniel martin katz michael j bommarito we want to take what we learn here and apply it here

111. the future of e-discovery will follow the arc of machine learning © daniel martin katz michael j bommarito

112. Supervised Unsupervised Predictive Coding (Classification) The Long Term Future Machine Learning Methods 2 x 2 Informed Naive Basic Clustering Algorithm © daniel martin katz michael j bommarito

113. there are different forms of learning by machines ... © daniel martin katz michael j bommarito

114. There Is Learning Within a Matter (i.e. learning from a specific training set) © daniel martin katz michael j bommarito

115. In other words, it is possible for the machine to learn from the experience of having processed documents in the past © daniel martin katz michael j bommarito

119. data aggregation / reusing data help move from the supervised to the semi/unsupervised © daniel martin katz michael j bommarito

120. Supervised Unsupervised Predictive Coding (Classification) The Future Machine Learning Methods 2 x 2 Informed Naive Basic Clustering Algorithm © daniel martin katz michael j bommarito

124. © daniel martin katz michael j bommarito The system comes pre-trained   for provisions including: Title, Parties, Date, Term, Change of Control, Assignment, Indemnity, Confidentiality, Governing Law, License Grant, Bankruptcy, Notice, Amendment, Non-Solicit, and more.

125. Based on testing, we know our system finds 90% or more of the instances of nearly every substantive provision it covers. This 90% number is our system’s recall; its precision differs by provision by provision but is consistently very manageable. © daniel martin katz michael j bommarito

126. We are able to build custom provisions on request. Thanks to our highly customized training algorithms, this process is easy and relatively automated. We are also engaged in adding more provisions. © daniel martin katz michael j bommarito

137. Predicting the Behavior of the United States Supreme Court: A General Approach © daniel martin katz michael j bommarito Black Reed Frankfurter Douglas Jackson Burton Clark Minton Warren Harlan Brennan Whittaker Stewart White Goldberg Fortas Marshall Burger Blackmun Powell Rehnquist Stevens OConnor Scalia Kennedy Souter Thomas Ginsburg Breyer Roberts Alito Sotomayor Kagan 1953 1963 1973 1983 1993 2003 2013 9-0 Reverse 8-1, 7-2, 6-3 19 19 19 19 19 20 20 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 - Reverse 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 - 8-1, 7-2, 6-3 9-0 19 19 19 19 19 20 20

138. feature engineering © daniel martin katz michael j bommarito The real world gives us raw material, at best. Typically, you even have to dig the stuff raw material out of your own unstructured data

145. © daniel martin katz michael j bommarito “John Dragseth, a principal at Fish & Richardson (the most active IP litigation firm in the United States, according to Corporate Counsel magazine), credits Lex Machina’s database with helping him spot meaningful but otherwise hidden trends in IP litigation—and he won’t give details. “If you published it, then people on the other side would know,” he says.

146. © daniel martin katz michael j bommarito Notice there is an offloading of data but it is up to the end user to derive meaning

147. © daniel martin katz michael j bommarito In general, the relevant consumer market is not yet mature when it comes to data science

148. © daniel martin katz michael j bommarito Difficult to sell machine learning technology in instances where the end user does not have the right assets in place

158. © daniel martin katz michael j bommarito before 2009 most of the individuals in the profession have only known the bull market

159. © daniel martin katz michael j bommarito it is a bear market now ... and in a bear market you need a serious strategy

162. So lets be wildcatters

166. © daniel martin katz michael j bommarito Assigning a innovation partner or an innovation committee is probably not enough

168. © daniel martin katz michael j bommarito how many organizations have a full time data scientist (data science team)?

172. daniel martin katz michael j bommarito ii adjunct professor of Law @ michigan state university associate professor of law @ illinois tech - chicago kent co-founder @ LexPredict director of research @ reInventLaw laboratory co-founder @ LexPredict Forum on Legal Evolution NYC

Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution- NYC

You are reading a preview.

Check these out next

Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution- NYC

More Related Content

Slideshows for you (19)

Viewers also liked (7)

Similar to Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution- NYC (20)

More from Daniel Katz (13)