SlideShare a Scribd company logo
1 of 13
Copyright ©2015 Ron Dolin1
Why Measure Quality?
Chicago Legal Innovation
Ron Dolin
Stanford Law School
Notre Dame Law School
April 16, 2015
Copyright ©2015 Ron Dolin2
Quality Control (QC) at Google

Purpose
− Improving core service (ads, search results)
− Efficiency gains w/o harm
− Preventing/discovering hidden problems

Method
− Manual setup
− Automated testing
− Scoring Result Quality (subjective/objective)
− Internal (absolute)
− External (relative)
Copyright ©2015 Ron Dolin3
Legal Services QC Goals

Clients
− compare (component) services
− value-driven purchasing
− ROI

New Entrants
− validation
− market segmentation

Incumbants
− high standards (not via proxy)
− efficiency (w/o loss of quality)
− staff performance metrics
Copyright ©2015 Ron Dolin4
Current QC Inadequate

Subjective

Coarse grain
− outcome aggregates (e.g., annual pay-out)

Proxy
− reputation
− pedigree

Manual
Copyright ©2015 Ron Dolin5
Necessary Methodology

Manual
− hierarchical, direct, per-item
− peer
− sampling
− fuzzy

Automated
− training data
− scoring function (e.g. bar exam essay scores)
− tuning and testing procedures
Copyright ©2015 Ron Dolin6
ODR Evaluation – New Approach

Online divorce mediation: # MSA's

eBay:
− 80% (of 60M!) resolved via automation
− closed ecosystem amenable to optimization
− implementing feedback is within eBay's
control
− lower # disputes per transaction (via
feedback)

DSD – Dispute Systems Design – prevention!
− negotiating interests, rights, or power
Copyright ©2015 Ron Dolin7
Benchmarks

Value = Quality / Cost

Measurable quality metrics

Can't compare without the same quality metrics

Who has the money and power to drive this?

Empirical legal studies: benchmarks

Human evaluation → metrics → tuning

All components of the legal system
Copyright ©2015 Ron Dolin8
Question

When is doing nothing better than doing
something?

Examples:
− bad wills worse than probate
− bad contracts worse than UCC defaults
− wrong party pays worse than no one pays
Copyright ©2015 Ron Dolin9
Legal Technology Framework

Factor 1: When/where to use

Factor 2: Legal and user requirements, design

Factor 3: How to evaluate, compare to non-
tech method

Factor 4: Impact on the legal system
Copyright ©2015 Ron Dolin13
Evaluation – Beyond Coin Tosses

Goals:
− efficiency
− quality (e.g. fairness – process, outcome)
− accuracy
− accessibility

How much quality is efficiency worth?
Copyright ©2015 Ron Dolin14
ODR Evaluation – The Literature

Literature:
− Outcome, process
− Alternative systems
− % settling
− Cost (time, $) to participants and system

Problems
− Comparing online with offline

measured offline? (legal tech generally)

we can't send same case through both
− Subjective vs. objective criteria
Copyright ©2015 Ron Dolin15
What Is Success?
Rebecca Sandefur: “The problem is those people
who think they understand what to do about their
problem... but don't."
Is procedural fairness sufficient?
In order to apply machine learning, we must pick
an objective function.
Copyright ©2015 Ron Dolin16
Factor 4: Impact On The Legal System
Overly Simplistic Approach

How to apply feedback to the legal system?

Which metrics to optimize – reduce # disputes?
− less incarceration? (death penalty for tickets)
− less lawsuits? (shut down the judiciary)
− less divorces? (outlaw marriage – the cause
of the problem is outside the institution that
has to deal with it)

More Related Content

Similar to Why Measure Quality?

Similar to Why Measure Quality? (20)

An Engineer's Journey Through the Legal System
An Engineer's Journey Through the Legal SystemAn Engineer's Journey Through the Legal System
An Engineer's Journey Through the Legal System
 
ODR and Legal Tech
ODR and Legal TechODR and Legal Tech
ODR and Legal Tech
 
Using Technology in Your Law Practice - MO SASF
Using Technology in Your Law Practice - MO SASFUsing Technology in Your Law Practice - MO SASF
Using Technology in Your Law Practice - MO SASF
 
GDPR project board deck (example)
GDPR project board deck (example)GDPR project board deck (example)
GDPR project board deck (example)
 
CDO Vision: The Value of Data
CDO Vision: The Value of DataCDO Vision: The Value of Data
CDO Vision: The Value of Data
 
The Future of Auditing and Fraud Detection
The Future of Auditing and Fraud Detection The Future of Auditing and Fraud Detection
The Future of Auditing and Fraud Detection
 
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn
 
PQF Overview
PQF OverviewPQF Overview
PQF Overview
 
Legal ROI: Quality & Innovation, Incorporating Efficiency
Legal ROI: Quality & Innovation, Incorporating EfficiencyLegal ROI: Quality & Innovation, Incorporating Efficiency
Legal ROI: Quality & Innovation, Incorporating Efficiency
 
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn
Benchmarking: Underutilized Measuring Tool | Dick Dadamo | Lunch & Learn
 
Touchstone Research for Internal Audit 2020 – A Look at the Now and Tomorrow ...
Touchstone Research for Internal Audit 2020 – A Look at the Now and Tomorrow ...Touchstone Research for Internal Audit 2020 – A Look at the Now and Tomorrow ...
Touchstone Research for Internal Audit 2020 – A Look at the Now and Tomorrow ...
 
Exception analytics - Balancing Risk & Control
Exception analytics - Balancing Risk & ControlException analytics - Balancing Risk & Control
Exception analytics - Balancing Risk & Control
 
Lecture 8 Resources and Costs.pptx
Lecture 8 Resources and Costs.pptxLecture 8 Resources and Costs.pptx
Lecture 8 Resources and Costs.pptx
 
Recent Outcomes Evaluations of Legal Aid Tech Projects
Recent Outcomes Evaluations of Legal Aid Tech ProjectsRecent Outcomes Evaluations of Legal Aid Tech Projects
Recent Outcomes Evaluations of Legal Aid Tech Projects
 
Presentation it testing outsourcing 2013
Presentation   it testing outsourcing 2013Presentation   it testing outsourcing 2013
Presentation it testing outsourcing 2013
 
Developing an Opportunity Pipeline
Developing an Opportunity PipelineDeveloping an Opportunity Pipeline
Developing an Opportunity Pipeline
 
Metrics and Analytics, Guest Lecture, UCLA
Metrics and Analytics, Guest Lecture, UCLAMetrics and Analytics, Guest Lecture, UCLA
Metrics and Analytics, Guest Lecture, UCLA
 
Compliance Officer update: What you should know about your Business Partner -...
Compliance Officer update: What you should know about your Business Partner -...Compliance Officer update: What you should know about your Business Partner -...
Compliance Officer update: What you should know about your Business Partner -...
 
Future audit analytics
Future audit analyticsFuture audit analytics
Future audit analytics
 
PAS: The Planning Quality Framework
PAS: The Planning Quality FrameworkPAS: The Planning Quality Framework
PAS: The Planning Quality Framework
 

More from Ron Dolin

More from Ron Dolin (10)

Legal Innovation: An Engineering Perspective of Law
Legal Innovation: An Engineering Perspective of LawLegal Innovation: An Engineering Perspective of Law
Legal Innovation: An Engineering Perspective of Law
 
Innovation and Emerging Technology
Innovation and Emerging TechnologyInnovation and Emerging Technology
Innovation and Emerging Technology
 
A Legal Magic Carpet Ride: Slides from the College of Law Practice Management...
A Legal Magic Carpet Ride: Slides from the College of Law Practice Management...A Legal Magic Carpet Ride: Slides from the College of Law Practice Management...
A Legal Magic Carpet Ride: Slides from the College of Law Practice Management...
 
Legal tech startup weekend 2014
Legal tech startup weekend 2014Legal tech startup weekend 2014
Legal tech startup weekend 2014
 
Inconsistency Robustness and the Law: A Random Walk
Inconsistency Robustness and the Law: A Random WalkInconsistency Robustness and the Law: A Random Walk
Inconsistency Robustness and the Law: A Random Walk
 
Teaching Legal Technology
Teaching Legal TechnologyTeaching Legal Technology
Teaching Legal Technology
 
Legal Technology and Design
Legal Technology and DesignLegal Technology and Design
Legal Technology and Design
 
Legal Technology Pipeline
Legal Technology PipelineLegal Technology Pipeline
Legal Technology Pipeline
 
Legal Tech Startups
Legal Tech StartupsLegal Tech Startups
Legal Tech Startups
 
Legal Technology - State Bar of CA - Solo/Small Firm Summit
Legal Technology - State Bar of CA - Solo/Small Firm SummitLegal Technology - State Bar of CA - Solo/Small Firm Summit
Legal Technology - State Bar of CA - Solo/Small Firm Summit
 

Recently uploaded

Recently uploaded (20)

Casa Tradicion v. Casa Azul Spirits (S.D. Tex. 2024)
Casa Tradicion v. Casa Azul Spirits (S.D. Tex. 2024)Casa Tradicion v. Casa Azul Spirits (S.D. Tex. 2024)
Casa Tradicion v. Casa Azul Spirits (S.D. Tex. 2024)
 
Justice Advocates Legal Defence Firm
Justice Advocates Legal Defence FirmJustice Advocates Legal Defence Firm
Justice Advocates Legal Defence Firm
 
Embed-1-4.pdf Decision of the High Court
Embed-1-4.pdf Decision of the High CourtEmbed-1-4.pdf Decision of the High Court
Embed-1-4.pdf Decision of the High Court
 
Does Apple Neurotechnology Patents Go To Far?
Does Apple  Neurotechnology Patents Go To Far?Does Apple  Neurotechnology Patents Go To Far?
Does Apple Neurotechnology Patents Go To Far?
 
Bail, Disposal, Remand, Registration JD Sir.pptx
Bail, Disposal, Remand, Registration JD Sir.pptxBail, Disposal, Remand, Registration JD Sir.pptx
Bail, Disposal, Remand, Registration JD Sir.pptx
 
CHP 5 OF OFFENCES AGAINST WOMEN AND CHILDREN.pptx
CHP 5 OF OFFENCES AGAINST WOMEN AND CHILDREN.pptxCHP 5 OF OFFENCES AGAINST WOMEN AND CHILDREN.pptx
CHP 5 OF OFFENCES AGAINST WOMEN AND CHILDREN.pptx
 
Application of Doctrine of Renvoi by foreign courts under conflict of laws
Application of Doctrine of Renvoi by foreign courts under conflict of lawsApplication of Doctrine of Renvoi by foreign courts under conflict of laws
Application of Doctrine of Renvoi by foreign courts under conflict of laws
 
Grounds for Quashing 482 CrPC High Court
Grounds for Quashing 482 CrPC High CourtGrounds for Quashing 482 CrPC High Court
Grounds for Quashing 482 CrPC High Court
 
Solidarity and Taxation: the Ubuntu approach in South Africa
Solidarity and Taxation: the Ubuntu approach in South AfricaSolidarity and Taxation: the Ubuntu approach in South Africa
Solidarity and Taxation: the Ubuntu approach in South Africa
 
dandan liu need to rot when she dies..pdf
dandan liu need to rot when she dies..pdfdandan liu need to rot when she dies..pdf
dandan liu need to rot when she dies..pdf
 
Rights of Consumers under Consumer Protection Act, 1986.
Rights of Consumers under Consumer Protection Act, 1986.Rights of Consumers under Consumer Protection Act, 1986.
Rights of Consumers under Consumer Protection Act, 1986.
 
Supreme Court Regulation No. 3 of 2023 on Procedure for Appointment of Arbitr...
Supreme Court Regulation No. 3 of 2023 on Procedure for Appointment of Arbitr...Supreme Court Regulation No. 3 of 2023 on Procedure for Appointment of Arbitr...
Supreme Court Regulation No. 3 of 2023 on Procedure for Appointment of Arbitr...
 
Protection Against Arrest and Detention art 2223 and 24.pptx
Protection Against Arrest and Detention art 2223 and 24.pptxProtection Against Arrest and Detention art 2223 and 24.pptx
Protection Against Arrest and Detention art 2223 and 24.pptx
 
Starbucks Corp. v. Sardarbuksh Coffee Co.
Starbucks Corp. v. Sardarbuksh Coffee Co.Starbucks Corp. v. Sardarbuksh Coffee Co.
Starbucks Corp. v. Sardarbuksh Coffee Co.
 
Everything You Should Know About Child Custody and Parenting While Living in ...
Everything You Should Know About Child Custody and Parenting While Living in ...Everything You Should Know About Child Custody and Parenting While Living in ...
Everything You Should Know About Child Custody and Parenting While Living in ...
 
Indian Partnership Act 1932, Rights and Duties of Partners
Indian Partnership Act 1932, Rights and Duties of PartnersIndian Partnership Act 1932, Rights and Duties of Partners
Indian Partnership Act 1932, Rights and Duties of Partners
 
Mergers and Acquisitions in Kenya - An explanation
Mergers and Acquisitions in Kenya - An explanationMergers and Acquisitions in Kenya - An explanation
Mergers and Acquisitions in Kenya - An explanation
 
A Brief Introduction About Katelyn Prost
A Brief Introduction About Katelyn ProstA Brief Introduction About Katelyn Prost
A Brief Introduction About Katelyn Prost
 
REVIVING OUR STAR GOD IMAGES FROM MARRYING OUR 4 HOLY LAWS OF STAR GODS
REVIVING OUR STAR GOD IMAGES FROM MARRYING OUR 4 HOLY LAWS OF STAR GODSREVIVING OUR STAR GOD IMAGES FROM MARRYING OUR 4 HOLY LAWS OF STAR GODS
REVIVING OUR STAR GOD IMAGES FROM MARRYING OUR 4 HOLY LAWS OF STAR GODS
 
7 Basic Steps of Trust Administration.pdf
7 Basic Steps of Trust Administration.pdf7 Basic Steps of Trust Administration.pdf
7 Basic Steps of Trust Administration.pdf
 

Why Measure Quality?

  • 1. Copyright ©2015 Ron Dolin1 Why Measure Quality? Chicago Legal Innovation Ron Dolin Stanford Law School Notre Dame Law School April 16, 2015
  • 2. Copyright ©2015 Ron Dolin2 Quality Control (QC) at Google  Purpose − Improving core service (ads, search results) − Efficiency gains w/o harm − Preventing/discovering hidden problems  Method − Manual setup − Automated testing − Scoring Result Quality (subjective/objective) − Internal (absolute) − External (relative)
  • 3. Copyright ©2015 Ron Dolin3 Legal Services QC Goals  Clients − compare (component) services − value-driven purchasing − ROI  New Entrants − validation − market segmentation  Incumbants − high standards (not via proxy) − efficiency (w/o loss of quality) − staff performance metrics
  • 4. Copyright ©2015 Ron Dolin4 Current QC Inadequate  Subjective  Coarse grain − outcome aggregates (e.g., annual pay-out)  Proxy − reputation − pedigree  Manual
  • 5. Copyright ©2015 Ron Dolin5 Necessary Methodology  Manual − hierarchical, direct, per-item − peer − sampling − fuzzy  Automated − training data − scoring function (e.g. bar exam essay scores) − tuning and testing procedures
  • 6. Copyright ©2015 Ron Dolin6 ODR Evaluation – New Approach  Online divorce mediation: # MSA's  eBay: − 80% (of 60M!) resolved via automation − closed ecosystem amenable to optimization − implementing feedback is within eBay's control − lower # disputes per transaction (via feedback)  DSD – Dispute Systems Design – prevention! − negotiating interests, rights, or power
  • 7. Copyright ©2015 Ron Dolin7 Benchmarks  Value = Quality / Cost  Measurable quality metrics  Can't compare without the same quality metrics  Who has the money and power to drive this?  Empirical legal studies: benchmarks  Human evaluation → metrics → tuning  All components of the legal system
  • 8. Copyright ©2015 Ron Dolin8 Question  When is doing nothing better than doing something?  Examples: − bad wills worse than probate − bad contracts worse than UCC defaults − wrong party pays worse than no one pays
  • 9. Copyright ©2015 Ron Dolin9 Legal Technology Framework  Factor 1: When/where to use  Factor 2: Legal and user requirements, design  Factor 3: How to evaluate, compare to non- tech method  Factor 4: Impact on the legal system
  • 10. Copyright ©2015 Ron Dolin13 Evaluation – Beyond Coin Tosses  Goals: − efficiency − quality (e.g. fairness – process, outcome) − accuracy − accessibility  How much quality is efficiency worth?
  • 11. Copyright ©2015 Ron Dolin14 ODR Evaluation – The Literature  Literature: − Outcome, process − Alternative systems − % settling − Cost (time, $) to participants and system  Problems − Comparing online with offline  measured offline? (legal tech generally)  we can't send same case through both − Subjective vs. objective criteria
  • 12. Copyright ©2015 Ron Dolin15 What Is Success? Rebecca Sandefur: “The problem is those people who think they understand what to do about their problem... but don't." Is procedural fairness sufficient? In order to apply machine learning, we must pick an objective function.
  • 13. Copyright ©2015 Ron Dolin16 Factor 4: Impact On The Legal System Overly Simplistic Approach  How to apply feedback to the legal system?  Which metrics to optimize – reduce # disputes? − less incarceration? (death penalty for tickets) − less lawsuits? (shut down the judiciary) − less divorces? (outlaw marriage – the cause of the problem is outside the institution that has to deal with it)