• Save
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC 02.26.14
Upcoming SlideShare
Loading in...5
×
 

Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution- NYC 02.26.14

on

  • 3,838 views

 

Statistics

Views

Total Views
3,838
Slideshare-icon Views on SlideShare
1,358
Embed Views
2,480

Actions

Likes
4
Downloads
2
Comments
0

13 Embeds 2,480

http://computationallegalstudies.com 2352
http://mesaffairesvirtuelles.ca 68
http://feedly.com 35
http://ipv6.computationallegalstudies.com 7
http://digg.com 6
http://www.ipv6.computationallegalstudies.com 4
http://www.computationallegalstudies.com 2
http://www.inoreader.com 1
http://newsblur.com 1
http://ec2-107-21-222-181.compute-1.amazonaws.com 1
https://reader.aol.com 1
http://seoautomated.com 1
https://www.linkedin.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

     Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC 02.26.14 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution- NYC 02.26.14 Presentation Transcript

    • Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry daniel martin katz associate professor of law @ michigan state university co-founder @ reInventLaw laboratory co-founder @ LexPredict michael j bommarito adjunct professor of law @ michigan state university director of research @ reInventLaw laboratory co-founder @ LexPredict
    • Forum on Legal Evolution NYC 02.26.14
    • in the legal industry there already is better corn
    • "We always overestimate the change that will occur in the next two years and underestimate the change that will occur in the next ten” - bill gates
    • today’s focus is primarily legal analytics + process engineering © daniel martin katz michael j bommarito
    • before providing some concrete examples some broad thoughts ... © daniel martin katz michael j bommarito
    • three faces of innovation in legal © daniel martin katz michael j bommarito
    • (1) lawyers for innovators / entrepreneurs © daniel martin katz michael j bommarito
    • (1) lawyers for innovators / entrepreneurs what most lawyers and law schools think of as “Law+Entrepreneurship" © daniel martin katz michael j bommarito
    • (2) lawyers as innovators - substance © daniel martin katz michael j bommarito
    • (2) lawyers as innovators - substance poison pill - “the most important innovation in corporate law since Samuel Calvin Tate Dodd invented the trust for John D. Rockefeller and Standard Oil in 1879” © daniel martin katz michael j bommarito
    • (2) lawyers as innovators - substance emerging areas - 3D Printing, Driverless Cars, Augmented Reality, Drones, Internet of Things, CyberSecurity, Data Breach, Big Data+Privacy, etc. © daniel martin katz michael j bommarito
    • (3) lawyers as innovators - business/process © daniel martin katz michael j bommarito
    • (3) lawyers as innovators - business/process innovation directed toward transforming the practice of law © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • there are different ways that organizations are innovating on the third face © daniel martin katz michael j bommarito
    • {Law + Tech + Design + Delivery} TM Substantive Legal Expertise Analytics Platform AI Computing Process Mapping User Experience Design Thinking Business Models Regulation Marketing © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • some traditional law firms have been very aggressive © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • but most of the innovation is Lex.Startup © daniel martin katz michael j bommarito
    • Lex.Startup is beginning to take hold © daniel martin katz michael j bommarito
    • Lex.Startup 15 2009
    • Lex.Startup 15 2009
    • Lex.Startup 15 2009 425+ 2014 Law or Legal Related Companies * as highlighted by Josh Kubicki @ ReInventLaw London 2013
    • So what are these folks doing? © daniel martin katz michael j bommarito
    • R + D Function in the Legal Industry © daniel martin katz michael j bommarito
    • We Could Imagine a World Where Law Firms Did the R+D for the Industry © daniel martin katz michael j bommarito
    • But That Has (Mostly) Proven Illusive © daniel martin katz michael j bommarito
    • Lex.Startup is undertaking that function © daniel martin katz michael j bommarito
    • Here are the specific approaches that are being undertaken © daniel martin katz michael j bommarito
    • Some organizations are doing more than one © daniel martin katz michael j bommarito
    • labor arbitrage © daniel martin katz michael j bommarito
    • labor arbitrage process/tech arbitrage © daniel martin katz michael j bommarito
    • labor arbitrage process/tech arbitrage regulatory arbitrage © daniel martin katz michael j bommarito
    • labor arbitrage process/tech arbitrage regulatory arbitrage design as the ultimate bespoke © daniel martin katz michael j bommarito
    • labor arbitrage process/tech arbitrage design as the ultimate bespoke regulatory arbitrage predictive analytics © daniel martin katz michael j bommarito
    • could do an individual talk on any of these topics... © daniel martin katz michael j bommarito
    • labor arbitrage process/tech arbitrage design as the ultimate bespoke regulatory arbitrage predictive analytics © daniel martin katz michael j bommarito
    • labor arbitrage process/tech arbitrage design as the ultimate bespoke regulatory arbitrage predictive analytics © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • Predictive Analytics in Law © daniel martin katz michael j bommarito
    • The Data Driven Future of the Legal Industry © daniel martin katz michael j bommarito
    • is already here... © daniel martin katz michael j bommarito
    • 2011 © daniel martin katz michael j bommarito
    • 2011 © daniel martin katz michael j bommarito
    • 2012 © daniel martin katz michael j bommarito
    • 2013 © daniel martin katz michael j bommarito
    • 2013 © daniel martin katz michael j bommarito
    • 2013 © daniel martin katz michael j bommarito
    • 2013 © daniel martin katz michael j bommarito
    • 2013
    • 2013 © daniel martin katz michael j bommarito
    • 2013 © daniel martin katz michael j bommarito
    • Quantitative Legal Prediction - or How I Learned to Stop Worrying and Start Preparing for the Data Driven Future of the Legal Services Industry 62 Emory L. J. 909 (2013) Daniel Martin Katz Associate Professor of Law Michigan State University © daniel martin katz michael j bommarito
    • Cause and Effect vs. Quantitative Legal Prediction © daniel martin katz michael j bommarito
    • Cause and Effect vs. Quantitative Legal Prediction © daniel martin katz michael j bommarito
    • Machine Learning is the heart of predictive analytics © daniel martin katz michael j bommarito
    • Legal Analytics @MSU Law - Winter 2014 Professor Daniel Martin Katz Professor Michael J Bommarito II © daniel martin katz michael j bommarito
    • Some Machine Learning Methods Supervised Semi/Unsupervised Statistical models Neural Networks (NN) Bayesian, e.g., Naïve Bayes Classification Clustering Frequentist, e.g., Ordinary Least Squares K-means Neural Networks (NN) Hierarchical Support Vector Machines (SVM) Radial Basis (RBF) Random Forests (RF) Graph Genetic Algorithms (GA) © daniel martin katz michael j bommarito
    • http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html © daniel martin katz michael j bommarito
    • classification regression clustering dimension reduction the family of machine learning methods © daniel martin katz michael j bommarito
    • Quick Example of the Methods © daniel martin katz michael j bommarito
    • Take A Look These 12 Adapted from Slides By Victor Lavrenko and Nigel Goddard @ University of Edinburgh © daniel martin katz michael j bommarito
    • 72 Female Human 3 Female Horse 36 Male Human 21 Male Human 67 Male Human 6 Male Dog 50 Male Human 42 Female Human 29 Female Human 54 Male Human 44 Male Human 7 Female Human © daniel martin katz michael j bommarito
    • Classification (Supervised Learning) f( ) female Gender? male decision boundary © daniel martin katz michael j bommarito
    • Classification (Supervised Learning) f( ) female Gender? male Regression (Supervised Learning) 44 29 27 decision boundary f( ) Age? 50 53 44 36 37 21 22 42 44 6 10 54 48 67 68 3 3 # 72 74 7 6 © daniel martin katz michael j bommarito
    • Classification (Supervised Learning) f( ) female Gender? male Regression (Supervised Learning) 44 29 27 decision boundary 44 Maybe f( ) Loan Application? 54 48 67 68 3 3 Multi Class Classification (Supervised Learning) 36 37 21 22 6 10 # Age? 50 53 42 44 Yes Perhaps No f( ) 72 74 7 6 Multiclass = Boundary Hyperplane Yes No © daniel martin katz michael j bommarito
    • Classification (Supervised Learning) f( ) female Gender? male Regression (Supervised Learning) 44 29 27 decision boundary 6 10 Maybe f( ) Loan Application? 54 48 67 68 3 3 Multi Class Classification (Supervised Learning) 36 37 21 22 42 44 Yes Perhaps No Age? 50 53 44 # f( ) 72 74 7 6 Clustering (Unsupervised Learning) f( ) Cluster Group? Multiclass = Boundary Hyperplane Yes No © daniel martin katz michael j bommarito
    • Regression as a Prediction Tool © daniel martin katz michael j bommarito
    • Regression as a Prediction Tool © daniel martin katz michael j bommarito
    • Standard Linear Regression Can Be Used to Predict a Probability (using LPM, Logit, etc.) © daniel martin katz michael j bommarito
    • Standard Linear Regression Can Be Used to Predict a Quantity © daniel martin katz michael j bommarito
    • Task = Predict the Expected Cost of a Given Legal Service f( ) and/or 010 101 001 # Cost? Regression (Supervised Learning) © daniel martin katz michael j bommarito
    • http://reinventlawchannel.com/ron-gruner-were-on-a-mission/ © daniel martin katz michael j bommarito
    • Y = βo +/- β1 ( X1 Y = $151 + $15 ) +/- β2 ( X2 ) +/- β3 ( X3 ) +/- Per ( 100 ) Lawyers + 161 ( If Tier 1 Market is True ) + 95 Partner ( Status ) is True β4 ( X3 ) +/- β5 ( X3 ) + ε + 34 Per ( 10 Years ) +/- β5 ( Practice ) + ε Area © daniel martin katz michael j bommarito
    • Turn Around and Use This Model To Predict Other Lawyers (also Matters, etc.) © daniel martin katz michael j bommarito
    • This Requires a Method to Deal With Changes in Dynamics, etc. © daniel martin katz michael j bommarito
    • This Requires a Method to Update the Model as Time Moves Forward © daniel martin katz michael j bommarito
    • Must Deal With Overfitting to the Existing Data © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • Machine Learning and the Future of E-Discovery © daniel martin katz michael j bommarito
    • imagine your client is served with a request for production © daniel martin katz michael j bommarito
    • in random order assume this is the size of the hypothetical document set (emails, memos, etc.)
    • we can sample a subset of the documents
    • we can sample a subset of the documents
    • classification regression clustering dimension reduction © daniel martin katz michael j bommarito
    • classification © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • predictive coding = ~ binary classification © daniel martin katz michael j bommarito
    • Learning Task = Determine Whether a Given Document is Relevant? and/or 010 101 001 f( ) relevance? Relevant Not Relevant Binary Classification (Supervised Learning) © daniel martin katz michael j bommarito
    • take the sample set as a training set and use human experts © daniel martin katz michael j bommarito
    • the use of the human experts is called “supervised learning” © daniel martin katz michael j bommarito
    • in the simple binary case, ask humans to assign objects to two piles © daniel martin katz michael j bommarito
    • Apply Human Coders © daniel martin katz michael j bommarito
    • and return this yellow = relevant white = non-relevant © daniel martin katz michael j bommarito
    • Relevant Non Relevant © daniel martin katz michael j bommarito
    • Key Insight ... © daniel martin katz michael j bommarito
    • What Allows A Human To Separate These Two Classes of Documents? © daniel martin katz michael j bommarito
    • that precise human process is what “predictive coding” is trying to mimic © daniel martin katz michael j bommarito
    • most vendors are selling a largely undifferentiated product © daniel martin katz michael j bommarito
    • Humans are selecting upon some “features” of the documents © daniel martin katz michael j bommarito
    • to place those documents in their respective bins (i.e. relevant, non-relevant) © daniel martin katz michael j bommarito
    • features =? text, author, date, other metadata © daniel martin katz michael j bommarito
    • machine learning task is trying to recover (learn) what separates the relevant from the non-relevant documents © daniel martin katz michael j bommarito
    • once we learn the rule / boundary we can apply it to separate the remain documents into the two classes © daniel martin katz michael j bommarito
    • we want to take what we learn here © daniel martin katz michael j bommarito
    • we want to take what we learn here © daniel martin katz michael j bommarito
    • we want to take what we learn here and apply it here © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • the future of e-discovery will follow the arc of machine learning © daniel martin katz michael j bommarito
    • Machine Learning Methods 2x2 Informed The Long Term Future Supervised Predictive Coding (Classification) Unsupervised Naive Basic Clustering Algorithm © daniel martin katz michael j bommarito
    • there are different forms of learning by machines ... © daniel martin katz michael j bommarito
    • There Is Learning Within a Matter (i.e. learning from a specific training set) © daniel martin katz michael j bommarito
    • In other words, it is possible for the machine to learn from the experience of having processed documents in the past © daniel martin katz michael j bommarito
    • both inside a given company but also across companies ... © daniel martin katz michael j bommarito
    • this is how data aggregation / reusing data becomes very powerful © daniel martin katz michael j bommarito
    • data aggregation / reusing data make the naive into the informed © daniel martin katz michael j bommarito
    • data aggregation / reusing data help move from the supervised to the semi/unsupervised © daniel martin katz michael j bommarito
    • Machine Learning Methods 2x2 Informed The Future Supervised Predictive Coding (Classification) Unsupervised Naive Basic Clustering Algorithm © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • Machine Learning Natural Language Processing and Due Diligence © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • The system comes pre-trained for provisions including: Title, Parties, Date, Term, Change of Control, Assignment, Indemnity, Confidentiality, Gover ning Law, License Grant, Bankruptcy, Notice, Amendment, Non-Solicit, and more. © daniel martin katz michael j bommarito
    • Based on testing, we know our system finds 90% or more of the instances of nearly every substantive provision it covers. This 90% number is our system’s recall; its precision differs by provision by p ro v i s i o n b u t i s c o n s i s t e n t l y v e r y manageable. © daniel martin katz michael j bommarito
    • We are able to build custom provisions on request. Thanks to our highly customized training algorithms, this process is easy and relatively automated. We are also engaged in adding more provisions. © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • Machine Learning and Judicial Behavior © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • 2002 Prediction Tourney and its limits © daniel martin katz michael j bommarito
    • Model Leverages Classification Tree (Tool from Machine Learning) © daniel martin katz michael j bommarito
    • Standard Decision Tree Often Not Generalizable, Often Overfits the Data © daniel martin katz michael j bommarito
    • Need a more complex approach © daniel martin katz michael j bommarito
    • Predicting the Behavior of the United States Supreme Court: A General Approach Daniel Martin Katz Michael J Bommarito Josh Blackman Coming Soon! © daniel martin katz michael j bommarito
    • feature engineering The real world gives us raw material, at best.   Typically, you even have to dig the stuff raw material out of your own unstructured data © daniel martin katz michael j bommarito
    • similar approach can be applied to other problems © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • Case Prediction and Litigation Data © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • “John Dragseth, a principal at Fish & Richardson (the most active IP litigation firm in the United States, according to Corporate Counsel magazine), credits Lex Machina’s database with helping him spot meaningful but otherwise hidden trends in IP litigation—and he won’t give details. “If you published it, then people on the other side would know,” he says. © daniel martin katz michael j bommarito
    • Notice there is an offloading of data but it is up to the end user to derive meaning © daniel martin katz michael j bommarito
    • In general, the relevant consumer market is not yet mature when it comes to data science © daniel martin katz michael j bommarito
    • Difficult to sell machine learning technology in instances where the end user does not have the right assets in place © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • Many other examples ... just starting to come online © daniel martin katz michael j bommarito
    • Attorney Quality and Performance © daniel martin katz michael j bommarito
    • Leveraging Public Data for Legal Insight © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • Change Management is the Hardest Innovation of All © daniel martin katz michael j bommarito
    • Bulls and Bears ~1984 - 2009 ~2009 - 2014 © daniel martin katz michael j bommarito
    • If you were 28 in 1984 than you were 53 in 2009 58 in 2014 © daniel martin katz michael j bommarito
    • before 2009 most of the individuals in the profession have only known the bull market © daniel martin katz michael j bommarito
    • it is a bear market now ... and in a bear market you need a serious strategy © daniel martin katz michael j bommarito
    • analytics/data should be part of that strategy © daniel martin katz michael j bommarito
    • “data is the oil of the 21st Century” © daniel martin katz michael j bommarito
    • So lets be wildcatters
    • law < > finance many elements in law look like finance did 25 years ago © daniel martin katz michael j bommarito
    • © daniel martin katz michael j bommarito
    • When it comes to innovation at the level that is going to be needed ... © daniel martin katz michael j bommarito
    • Assigning a innovation partner or an innovation committee is probably not enough © daniel martin katz michael j bommarito
    • Shunk Works © daniel martin katz michael j bommarito
    • how many organizations have a full time data scientist (data science team)? © daniel martin katz michael j bommarito
    • need a full scale and empowered R+D team (data science) © daniel martin katz michael j bommarito
    • Final Thought © daniel martin katz michael j bommarito
    • Exit, Voice & Loyalty © daniel martin katz michael j bommarito
    • Forum on Legal Evolution NYC 02.26.14 daniel martin katz associate professor of law @ michigan state university co-founder @ reInventLaw laboratory co-founder @ LexPredict michael j bommarito ii adjunct professor of Law @ michigan state university director of research @ reInventLaw laboratory co-founder @ LexPredict