SlideShare a Scribd company logo
1 of 13
Copyright ©2015 Ron Dolin1
Why Measure Quality?
Chicago Legal Innovation
Ron Dolin
Stanford Law School
Notre Dame Law School
April 16, 2015
Copyright ©2015 Ron Dolin2
Quality Control (QC) at Google

Purpose
− Improving core service (ads, search results)
− Efficiency gains w/o harm
− Preventing/discovering hidden problems

Method
− Manual setup
− Automated testing
− Scoring Result Quality (subjective/objective)
− Internal (absolute)
− External (relative)
Copyright ©2015 Ron Dolin3
Legal Services QC Goals

Clients
− compare (component) services
− value-driven purchasing
− ROI

New Entrants
− validation
− market segmentation

Incumbants
− high standards (not via proxy)
− efficiency (w/o loss of quality)
− staff performance metrics
Copyright ©2015 Ron Dolin4
Current QC Inadequate

Subjective

Coarse grain
− outcome aggregates (e.g., annual pay-out)

Proxy
− reputation
− pedigree

Manual
Copyright ©2015 Ron Dolin5
Necessary Methodology

Manual
− hierarchical, direct, per-item
− peer
− sampling
− fuzzy

Automated
− training data
− scoring function (e.g. bar exam essay scores)
− tuning and testing procedures
Copyright ©2015 Ron Dolin6
ODR Evaluation – New Approach

Online divorce mediation: # MSA's

eBay:
− 80% (of 60M!) resolved via automation
− closed ecosystem amenable to optimization
− implementing feedback is within eBay's
control
− lower # disputes per transaction (via
feedback)

DSD – Dispute Systems Design – prevention!
− negotiating interests, rights, or power
Copyright ©2015 Ron Dolin7
Benchmarks

Value = Quality / Cost

Measurable quality metrics

Can't compare without the same quality metrics

Who has the money and power to drive this?

Empirical legal studies: benchmarks

Human evaluation → metrics → tuning

All components of the legal system
Copyright ©2015 Ron Dolin8
Question

When is doing nothing better than doing
something?

Examples:
− bad wills worse than probate
− bad contracts worse than UCC defaults
− wrong party pays worse than no one pays
Copyright ©2015 Ron Dolin9
Legal Technology Framework

Factor 1: When/where to use

Factor 2: Legal and user requirements, design

Factor 3: How to evaluate, compare to non-
tech method

Factor 4: Impact on the legal system
Copyright ©2015 Ron Dolin13
Evaluation – Beyond Coin Tosses

Goals:
− efficiency
− quality (e.g. fairness – process, outcome)
− accuracy
− accessibility

How much quality is efficiency worth?
Copyright ©2015 Ron Dolin14
ODR Evaluation – The Literature

Literature:
− Outcome, process
− Alternative systems
− % settling
− Cost (time, $) to participants and system

Problems
− Comparing online with offline

measured offline? (legal tech generally)

we can't send same case through both
− Subjective vs. objective criteria
Copyright ©2015 Ron Dolin15
What Is Success?
Rebecca Sandefur: “The problem is those people
who think they understand what to do about their
problem... but don't."
Is procedural fairness sufficient?
In order to apply machine learning, we must pick
an objective function.
Copyright ©2015 Ron Dolin16
Factor 4: Impact On The Legal System
Overly Simplistic Approach

How to apply feedback to the legal system?

Which metrics to optimize – reduce # disputes?
− less incarceration? (death penalty for tickets)
− less lawsuits? (shut down the judiciary)
− less divorces? (outlaw marriage – the cause
of the problem is outside the institution that
has to deal with it)

More Related Content

Similar to Why Measure Quality?

An Engineer's Journey Through the Legal System
An Engineer's Journey Through the Legal SystemAn Engineer's Journey Through the Legal System
An Engineer's Journey Through the Legal SystemRon Dolin
 
ODR and Legal Tech
ODR and Legal TechODR and Legal Tech
ODR and Legal TechRon Dolin
 
Using Technology in Your Law Practice - MO SASF
Using Technology in Your Law Practice - MO SASFUsing Technology in Your Law Practice - MO SASF
Using Technology in Your Law Practice - MO SASFDowney Law Group LLC
 
GDPR project board deck (example)
GDPR project board deck (example)GDPR project board deck (example)
GDPR project board deck (example)Tommy Vandepitte
 
CDO Vision: The Value of Data
CDO Vision: The Value of DataCDO Vision: The Value of Data
CDO Vision: The Value of DataDATAVERSITY
 
The Future of Auditing and Fraud Detection
The Future of Auditing and Fraud Detection The Future of Auditing and Fraud Detection
The Future of Auditing and Fraud Detection Jim Kaplan CIA CFE
 
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn UCICove
 
Legal ROI: Quality & Innovation, Incorporating Efficiency
Legal ROI: Quality & Innovation, Incorporating EfficiencyLegal ROI: Quality & Innovation, Incorporating Efficiency
Legal ROI: Quality & Innovation, Incorporating EfficiencyRon Dolin
 
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn UCICove
 
Touchstone Research for Internal Audit 2020 – A Look at the Now and Tomorrow ...
Touchstone Research for Internal Audit 2020 – A Look at the Now and Tomorrow ...Touchstone Research for Internal Audit 2020 – A Look at the Now and Tomorrow ...
Touchstone Research for Internal Audit 2020 – A Look at the Now and Tomorrow ...Jim Kaplan CIA CFE
 
Exception analytics - Balancing Risk & Control
Exception analytics - Balancing Risk & ControlException analytics - Balancing Risk & Control
Exception analytics - Balancing Risk & ControlDan French
 
Lecture 8 Resources and Costs.pptx
Lecture 8 Resources and Costs.pptxLecture 8 Resources and Costs.pptx
Lecture 8 Resources and Costs.pptxGelmelinaLeaLepitenH
 
Developing an Opportunity Pipeline
Developing an Opportunity PipelineDeveloping an Opportunity Pipeline
Developing an Opportunity PipelineTan Wilson, PMP
 
Metrics and Analytics, Guest Lecture, UCLA
Metrics and Analytics, Guest Lecture, UCLAMetrics and Analytics, Guest Lecture, UCLA
Metrics and Analytics, Guest Lecture, UCLADarren Levy
 
Compliance Officer update: What you should know about your Business Partner -...
Compliance Officer update: What you should know about your Business Partner -...Compliance Officer update: What you should know about your Business Partner -...
Compliance Officer update: What you should know about your Business Partner -...vivacidade
 
PAS: The Planning Quality Framework
PAS: The Planning Quality FrameworkPAS: The Planning Quality Framework
PAS: The Planning Quality FrameworkPAS_Team
 

Similar to Why Measure Quality? (20)

An Engineer's Journey Through the Legal System
An Engineer's Journey Through the Legal SystemAn Engineer's Journey Through the Legal System
An Engineer's Journey Through the Legal System
 
ODR and Legal Tech
ODR and Legal TechODR and Legal Tech
ODR and Legal Tech
 
Using Technology in Your Law Practice - MO SASF
Using Technology in Your Law Practice - MO SASFUsing Technology in Your Law Practice - MO SASF
Using Technology in Your Law Practice - MO SASF
 
GDPR project board deck (example)
GDPR project board deck (example)GDPR project board deck (example)
GDPR project board deck (example)
 
CDO Vision: The Value of Data
CDO Vision: The Value of DataCDO Vision: The Value of Data
CDO Vision: The Value of Data
 
The Future of Auditing and Fraud Detection
The Future of Auditing and Fraud Detection The Future of Auditing and Fraud Detection
The Future of Auditing and Fraud Detection
 
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn
 
PQF Overview
PQF OverviewPQF Overview
PQF Overview
 
Legal ROI: Quality & Innovation, Incorporating Efficiency
Legal ROI: Quality & Innovation, Incorporating EfficiencyLegal ROI: Quality & Innovation, Incorporating Efficiency
Legal ROI: Quality & Innovation, Incorporating Efficiency
 
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn
 
Touchstone Research for Internal Audit 2020 – A Look at the Now and Tomorrow ...
Touchstone Research for Internal Audit 2020 – A Look at the Now and Tomorrow ...Touchstone Research for Internal Audit 2020 – A Look at the Now and Tomorrow ...
Touchstone Research for Internal Audit 2020 – A Look at the Now and Tomorrow ...
 
Exception analytics - Balancing Risk & Control
Exception analytics - Balancing Risk & ControlException analytics - Balancing Risk & Control
Exception analytics - Balancing Risk & Control
 
Lecture 8 Resources and Costs.pptx
Lecture 8 Resources and Costs.pptxLecture 8 Resources and Costs.pptx
Lecture 8 Resources and Costs.pptx
 
Recent Outcomes Evaluations of Legal Aid Tech Projects
Recent Outcomes Evaluations of Legal Aid Tech ProjectsRecent Outcomes Evaluations of Legal Aid Tech Projects
Recent Outcomes Evaluations of Legal Aid Tech Projects
 
Presentation it testing outsourcing 2013
Presentation   it testing outsourcing 2013Presentation   it testing outsourcing 2013
Presentation it testing outsourcing 2013
 
Developing an Opportunity Pipeline
Developing an Opportunity PipelineDeveloping an Opportunity Pipeline
Developing an Opportunity Pipeline
 
Metrics and Analytics, Guest Lecture, UCLA
Metrics and Analytics, Guest Lecture, UCLAMetrics and Analytics, Guest Lecture, UCLA
Metrics and Analytics, Guest Lecture, UCLA
 
Compliance Officer update: What you should know about your Business Partner -...
Compliance Officer update: What you should know about your Business Partner -...Compliance Officer update: What you should know about your Business Partner -...
Compliance Officer update: What you should know about your Business Partner -...
 
Future audit analytics
Future audit analyticsFuture audit analytics
Future audit analytics
 
PAS: The Planning Quality Framework
PAS: The Planning Quality FrameworkPAS: The Planning Quality Framework
PAS: The Planning Quality Framework
 

More from Ron Dolin

Legal Innovation: An Engineering Perspective of Law
Legal Innovation: An Engineering Perspective of LawLegal Innovation: An Engineering Perspective of Law
Legal Innovation: An Engineering Perspective of LawRon Dolin
 
Innovation and Emerging Technology
Innovation and Emerging TechnologyInnovation and Emerging Technology
Innovation and Emerging TechnologyRon Dolin
 
A Legal Magic Carpet Ride: Slides from the College of Law Practice Management...
A Legal Magic Carpet Ride: Slides from the College of Law Practice Management...A Legal Magic Carpet Ride: Slides from the College of Law Practice Management...
A Legal Magic Carpet Ride: Slides from the College of Law Practice Management...Ron Dolin
 
Legal tech startup weekend 2014
Legal tech startup weekend 2014Legal tech startup weekend 2014
Legal tech startup weekend 2014Ron Dolin
 
Inconsistency Robustness and the Law: A Random Walk
Inconsistency Robustness and the Law: A Random WalkInconsistency Robustness and the Law: A Random Walk
Inconsistency Robustness and the Law: A Random WalkRon Dolin
 
Teaching Legal Technology
Teaching Legal TechnologyTeaching Legal Technology
Teaching Legal TechnologyRon Dolin
 
Legal Technology and Design
Legal Technology and DesignLegal Technology and Design
Legal Technology and DesignRon Dolin
 
Legal Technology Pipeline
Legal Technology PipelineLegal Technology Pipeline
Legal Technology PipelineRon Dolin
 
Legal Tech Startups
Legal Tech StartupsLegal Tech Startups
Legal Tech StartupsRon Dolin
 
Legal Technology - State Bar of CA - Solo/Small Firm Summit
Legal Technology - State Bar of CA - Solo/Small Firm SummitLegal Technology - State Bar of CA - Solo/Small Firm Summit
Legal Technology - State Bar of CA - Solo/Small Firm SummitRon Dolin
 

More from Ron Dolin (10)

Legal Innovation: An Engineering Perspective of Law
Legal Innovation: An Engineering Perspective of LawLegal Innovation: An Engineering Perspective of Law
Legal Innovation: An Engineering Perspective of Law
 
Innovation and Emerging Technology
Innovation and Emerging TechnologyInnovation and Emerging Technology
Innovation and Emerging Technology
 
A Legal Magic Carpet Ride: Slides from the College of Law Practice Management...
A Legal Magic Carpet Ride: Slides from the College of Law Practice Management...A Legal Magic Carpet Ride: Slides from the College of Law Practice Management...
A Legal Magic Carpet Ride: Slides from the College of Law Practice Management...
 
Legal tech startup weekend 2014
Legal tech startup weekend 2014Legal tech startup weekend 2014
Legal tech startup weekend 2014
 
Inconsistency Robustness and the Law: A Random Walk
Inconsistency Robustness and the Law: A Random WalkInconsistency Robustness and the Law: A Random Walk
Inconsistency Robustness and the Law: A Random Walk
 
Teaching Legal Technology
Teaching Legal TechnologyTeaching Legal Technology
Teaching Legal Technology
 
Legal Technology and Design
Legal Technology and DesignLegal Technology and Design
Legal Technology and Design
 
Legal Technology Pipeline
Legal Technology PipelineLegal Technology Pipeline
Legal Technology Pipeline
 
Legal Tech Startups
Legal Tech StartupsLegal Tech Startups
Legal Tech Startups
 
Legal Technology - State Bar of CA - Solo/Small Firm Summit
Legal Technology - State Bar of CA - Solo/Small Firm SummitLegal Technology - State Bar of CA - Solo/Small Firm Summit
Legal Technology - State Bar of CA - Solo/Small Firm Summit
 

Recently uploaded

Attestation presentation under Transfer of property Act
Attestation presentation under Transfer of property ActAttestation presentation under Transfer of property Act
Attestation presentation under Transfer of property Act2020000445musaib
 
1990-2004 Bar Questions and Answers in Sales
1990-2004 Bar Questions and Answers in Sales1990-2004 Bar Questions and Answers in Sales
1990-2004 Bar Questions and Answers in SalesMelvinPernez2
 
Analysis on Law of Domicile under Private International laws.
Analysis on Law of Domicile under Private International laws.Analysis on Law of Domicile under Private International laws.
Analysis on Law of Domicile under Private International laws.2020000445musaib
 
Hungarian legislation made by Robert Miklos
Hungarian legislation made by Robert MiklosHungarian legislation made by Robert Miklos
Hungarian legislation made by Robert Miklosbeduinpower135
 
Sarvesh Raj IPS - A Journey of Dedication and Leadership.pptx
Sarvesh Raj IPS - A Journey of Dedication and Leadership.pptxSarvesh Raj IPS - A Journey of Dedication and Leadership.pptx
Sarvesh Raj IPS - A Journey of Dedication and Leadership.pptxAnto Jebin
 
citizenship in the Philippines as to the laws applicable
citizenship in the Philippines as to the laws applicablecitizenship in the Philippines as to the laws applicable
citizenship in the Philippines as to the laws applicableSaraSantiago44
 
Wurz Financial - Wealth Counsel to Law Firm Owners Services Guide.pdf
Wurz Financial - Wealth Counsel to Law Firm Owners Services Guide.pdfWurz Financial - Wealth Counsel to Law Firm Owners Services Guide.pdf
Wurz Financial - Wealth Counsel to Law Firm Owners Services Guide.pdfssuser3e15612
 
Conditions Restricting Transfer Under TPA,1882
Conditions Restricting Transfer Under TPA,1882Conditions Restricting Transfer Under TPA,1882
Conditions Restricting Transfer Under TPA,18822020000445musaib
 
Understanding Cyber Crime Litigation: Key Concepts and Legal Frameworks
Understanding Cyber Crime Litigation: Key Concepts and Legal FrameworksUnderstanding Cyber Crime Litigation: Key Concepts and Legal Frameworks
Understanding Cyber Crime Litigation: Key Concepts and Legal FrameworksFinlaw Associates
 
The Punjab Land Reforms AcT 1972 HIRDEBIR.pptx
The Punjab Land Reforms AcT 1972 HIRDEBIR.pptxThe Punjab Land Reforms AcT 1972 HIRDEBIR.pptx
The Punjab Land Reforms AcT 1972 HIRDEBIR.pptxgurcharnsinghlecengl
 
PPT Template - Federal Law Enforcement Training Center
PPT Template - Federal Law Enforcement Training CenterPPT Template - Federal Law Enforcement Training Center
PPT Template - Federal Law Enforcement Training Centerejlfernandez22
 
Labour legislations in India and its history
Labour legislations in India and its historyLabour legislations in India and its history
Labour legislations in India and its historyprasannamurthy6
 
Guide for Drug Education and Vice Control.docx
Guide for Drug Education and Vice Control.docxGuide for Drug Education and Vice Control.docx
Guide for Drug Education and Vice Control.docxjennysansano2
 
RA. 7432 and RA 9994 Senior Citizen .pptx
RA. 7432 and RA 9994 Senior Citizen .pptxRA. 7432 and RA 9994 Senior Citizen .pptx
RA. 7432 and RA 9994 Senior Citizen .pptxJFSB1
 
Alexis O'Connell Arrest Records Houston Texas lexileeyogi
Alexis O'Connell Arrest Records Houston Texas lexileeyogiAlexis O'Connell Arrest Records Houston Texas lexileeyogi
Alexis O'Connell Arrest Records Houston Texas lexileeyogiBlayneRush1
 
Are There Any Alternatives To Jail Time For Sex Crime Convictions in Los Angeles
Are There Any Alternatives To Jail Time For Sex Crime Convictions in Los AngelesAre There Any Alternatives To Jail Time For Sex Crime Convictions in Los Angeles
Are There Any Alternatives To Jail Time For Sex Crime Convictions in Los AngelesChesley Lawyer
 
Illinois Department Of Corrections reentry guide
Illinois Department Of Corrections reentry guideIllinois Department Of Corrections reentry guide
Illinois Department Of Corrections reentry guideillinoisworknet11
 
Department order DAO 97-32. powerpoint presentation
Department order DAO 97-32. powerpoint presentationDepartment order DAO 97-32. powerpoint presentation
Department order DAO 97-32. powerpoint presentationPhillisYvonMarshBagu
 
THE INDIAN CONTRACT ACT 1872 NOTES FOR STUDENTS
THE INDIAN CONTRACT ACT 1872 NOTES FOR STUDENTSTHE INDIAN CONTRACT ACT 1872 NOTES FOR STUDENTS
THE INDIAN CONTRACT ACT 1872 NOTES FOR STUDENTSRoshniSingh312153
 
Grey Area of the Information Technology Act, 2000.pptx
Grey Area of the Information Technology Act, 2000.pptxGrey Area of the Information Technology Act, 2000.pptx
Grey Area of the Information Technology Act, 2000.pptxBharatMunjal4
 

Recently uploaded (20)

Attestation presentation under Transfer of property Act
Attestation presentation under Transfer of property ActAttestation presentation under Transfer of property Act
Attestation presentation under Transfer of property Act
 
1990-2004 Bar Questions and Answers in Sales
1990-2004 Bar Questions and Answers in Sales1990-2004 Bar Questions and Answers in Sales
1990-2004 Bar Questions and Answers in Sales
 
Analysis on Law of Domicile under Private International laws.
Analysis on Law of Domicile under Private International laws.Analysis on Law of Domicile under Private International laws.
Analysis on Law of Domicile under Private International laws.
 
Hungarian legislation made by Robert Miklos
Hungarian legislation made by Robert MiklosHungarian legislation made by Robert Miklos
Hungarian legislation made by Robert Miklos
 
Sarvesh Raj IPS - A Journey of Dedication and Leadership.pptx
Sarvesh Raj IPS - A Journey of Dedication and Leadership.pptxSarvesh Raj IPS - A Journey of Dedication and Leadership.pptx
Sarvesh Raj IPS - A Journey of Dedication and Leadership.pptx
 
citizenship in the Philippines as to the laws applicable
citizenship in the Philippines as to the laws applicablecitizenship in the Philippines as to the laws applicable
citizenship in the Philippines as to the laws applicable
 
Wurz Financial - Wealth Counsel to Law Firm Owners Services Guide.pdf
Wurz Financial - Wealth Counsel to Law Firm Owners Services Guide.pdfWurz Financial - Wealth Counsel to Law Firm Owners Services Guide.pdf
Wurz Financial - Wealth Counsel to Law Firm Owners Services Guide.pdf
 
Conditions Restricting Transfer Under TPA,1882
Conditions Restricting Transfer Under TPA,1882Conditions Restricting Transfer Under TPA,1882
Conditions Restricting Transfer Under TPA,1882
 
Understanding Cyber Crime Litigation: Key Concepts and Legal Frameworks
Understanding Cyber Crime Litigation: Key Concepts and Legal FrameworksUnderstanding Cyber Crime Litigation: Key Concepts and Legal Frameworks
Understanding Cyber Crime Litigation: Key Concepts and Legal Frameworks
 
The Punjab Land Reforms AcT 1972 HIRDEBIR.pptx
The Punjab Land Reforms AcT 1972 HIRDEBIR.pptxThe Punjab Land Reforms AcT 1972 HIRDEBIR.pptx
The Punjab Land Reforms AcT 1972 HIRDEBIR.pptx
 
PPT Template - Federal Law Enforcement Training Center
PPT Template - Federal Law Enforcement Training CenterPPT Template - Federal Law Enforcement Training Center
PPT Template - Federal Law Enforcement Training Center
 
Labour legislations in India and its history
Labour legislations in India and its historyLabour legislations in India and its history
Labour legislations in India and its history
 
Guide for Drug Education and Vice Control.docx
Guide for Drug Education and Vice Control.docxGuide for Drug Education and Vice Control.docx
Guide for Drug Education and Vice Control.docx
 
RA. 7432 and RA 9994 Senior Citizen .pptx
RA. 7432 and RA 9994 Senior Citizen .pptxRA. 7432 and RA 9994 Senior Citizen .pptx
RA. 7432 and RA 9994 Senior Citizen .pptx
 
Alexis O'Connell Arrest Records Houston Texas lexileeyogi
Alexis O'Connell Arrest Records Houston Texas lexileeyogiAlexis O'Connell Arrest Records Houston Texas lexileeyogi
Alexis O'Connell Arrest Records Houston Texas lexileeyogi
 
Are There Any Alternatives To Jail Time For Sex Crime Convictions in Los Angeles
Are There Any Alternatives To Jail Time For Sex Crime Convictions in Los AngelesAre There Any Alternatives To Jail Time For Sex Crime Convictions in Los Angeles
Are There Any Alternatives To Jail Time For Sex Crime Convictions in Los Angeles
 
Illinois Department Of Corrections reentry guide
Illinois Department Of Corrections reentry guideIllinois Department Of Corrections reentry guide
Illinois Department Of Corrections reentry guide
 
Department order DAO 97-32. powerpoint presentation
Department order DAO 97-32. powerpoint presentationDepartment order DAO 97-32. powerpoint presentation
Department order DAO 97-32. powerpoint presentation
 
THE INDIAN CONTRACT ACT 1872 NOTES FOR STUDENTS
THE INDIAN CONTRACT ACT 1872 NOTES FOR STUDENTSTHE INDIAN CONTRACT ACT 1872 NOTES FOR STUDENTS
THE INDIAN CONTRACT ACT 1872 NOTES FOR STUDENTS
 
Grey Area of the Information Technology Act, 2000.pptx
Grey Area of the Information Technology Act, 2000.pptxGrey Area of the Information Technology Act, 2000.pptx
Grey Area of the Information Technology Act, 2000.pptx
 

Why Measure Quality?

  • 1. Copyright ©2015 Ron Dolin1 Why Measure Quality? Chicago Legal Innovation Ron Dolin Stanford Law School Notre Dame Law School April 16, 2015
  • 2. Copyright ©2015 Ron Dolin2 Quality Control (QC) at Google  Purpose − Improving core service (ads, search results) − Efficiency gains w/o harm − Preventing/discovering hidden problems  Method − Manual setup − Automated testing − Scoring Result Quality (subjective/objective) − Internal (absolute) − External (relative)
  • 3. Copyright ©2015 Ron Dolin3 Legal Services QC Goals  Clients − compare (component) services − value-driven purchasing − ROI  New Entrants − validation − market segmentation  Incumbants − high standards (not via proxy) − efficiency (w/o loss of quality) − staff performance metrics
  • 4. Copyright ©2015 Ron Dolin4 Current QC Inadequate  Subjective  Coarse grain − outcome aggregates (e.g., annual pay-out)  Proxy − reputation − pedigree  Manual
  • 5. Copyright ©2015 Ron Dolin5 Necessary Methodology  Manual − hierarchical, direct, per-item − peer − sampling − fuzzy  Automated − training data − scoring function (e.g. bar exam essay scores) − tuning and testing procedures
  • 6. Copyright ©2015 Ron Dolin6 ODR Evaluation – New Approach  Online divorce mediation: # MSA's  eBay: − 80% (of 60M!) resolved via automation − closed ecosystem amenable to optimization − implementing feedback is within eBay's control − lower # disputes per transaction (via feedback)  DSD – Dispute Systems Design – prevention! − negotiating interests, rights, or power
  • 7. Copyright ©2015 Ron Dolin7 Benchmarks  Value = Quality / Cost  Measurable quality metrics  Can't compare without the same quality metrics  Who has the money and power to drive this?  Empirical legal studies: benchmarks  Human evaluation → metrics → tuning  All components of the legal system
  • 8. Copyright ©2015 Ron Dolin8 Question  When is doing nothing better than doing something?  Examples: − bad wills worse than probate − bad contracts worse than UCC defaults − wrong party pays worse than no one pays
  • 9. Copyright ©2015 Ron Dolin9 Legal Technology Framework  Factor 1: When/where to use  Factor 2: Legal and user requirements, design  Factor 3: How to evaluate, compare to non- tech method  Factor 4: Impact on the legal system
  • 10. Copyright ©2015 Ron Dolin13 Evaluation – Beyond Coin Tosses  Goals: − efficiency − quality (e.g. fairness – process, outcome) − accuracy − accessibility  How much quality is efficiency worth?
  • 11. Copyright ©2015 Ron Dolin14 ODR Evaluation – The Literature  Literature: − Outcome, process − Alternative systems − % settling − Cost (time, $) to participants and system  Problems − Comparing online with offline  measured offline? (legal tech generally)  we can't send same case through both − Subjective vs. objective criteria
  • 12. Copyright ©2015 Ron Dolin15 What Is Success? Rebecca Sandefur: “The problem is those people who think they understand what to do about their problem... but don't." Is procedural fairness sufficient? In order to apply machine learning, we must pick an objective function.
  • 13. Copyright ©2015 Ron Dolin16 Factor 4: Impact On The Legal System Overly Simplistic Approach  How to apply feedback to the legal system?  Which metrics to optimize – reduce # disputes? − less incarceration? (death penalty for tickets) − less lawsuits? (shut down the judiciary) − less divorces? (outlaw marriage – the cause of the problem is outside the institution that has to deal with it)