Systematic Review is e-Discovery
in Doctor’s Clothing
Joint work with
Matt Lease
ir.ischool.utexas.edu
slideshare.net/matt...
“The place where people & technology meet”
~ Wobbrock et al., 2009
www.ischools.org
2
• System-Reviews
• Electronic Discovery (e-Discovery)
• Toward a Joint Research Agenda
3Matt Lease <ml@utexas.edu>
Roadmap
• System-Reviews
• Electronic Discovery (e-Discovery)
• Toward a Joint Research Agenda
4Matt Lease <ml@utexas.edu>
Roadmap
Evidence-Based Medicine n.
The conscientious, explicit and judicious
use of current best evidence in making
decisions abou...
Systematic reviews: from biomedical
articles to actionable evidence
6
PubMed
?
2 search database
1 formulate question,
protocol & query
4 extract data
treatment
outcome
ba
c d
3 screen retriev...
On average, 75 articles describing results from
clinical trials are published every day.
Bastian, PLoS Med, 2010
The media...
12
Technologies for semi-automated
citation screening are relatively mature
and slowly gaining acceptance
Research on citation screening
• Methods for handling imbalance with asymmetric costs [ICDM
2011; ICDM 2012; KAIS 2013]
• ...
• System-Reviews
• Electronic Discovery (e-Discovery)
• Toward a Joint Research Agenda
14Matt Lease <ml@utexas.edu>
Roadmap
PubMed
?
2 search database
1 formulate question,
protocol & query
4 extract data
treatment
outcome
ba
c d
3 screen retriev...
Manual Review does not Scale
16
Paul, George L., and Jason R. Baron.
Information inflation: Can the legal
system adapt? Ri...
IR Research in e-Discovery
• NIST TREC Track: 2006-2011
• Oard & Webber, FnTIR Book, 2013
• A variety of published work at...
• System-Reviews
• Electronic Discovery (e-Discovery)
• Toward a Joint Research Agenda
18Matt Lease <ml@utexas.edu>
Roadmap
Commonalities
• Need high-recall with bounded cost
• Follow 3-Stage Pipeline Today
– Boolean query
– Screening (traditiona...
Can we crowdsource screening?
Michael Mortenson, Byron C. Wallace, Gaelen Adam, Tom Trikalinos and Tim Kraska.
Crowdsourci...
21
Total Recall: Applications
22
E-Discovery
Total Recall: Strategies
23
Conclusion
• Systematic Review & e-Discovery have much in common,
but SR has received relatively little attention in IR
– ...
Thank You!
ir.ischool.utexas.eduSlides: www.slideshare.net/mattlease
25
Systematic Review is e-Discovery in Doctor’s Clothing
Systematic Review is e-Discovery in Doctor’s Clothing
Systematic Review is e-Discovery in Doctor’s Clothing
Upcoming SlideShare
Loading in …5
×

Systematic Review is e-Discovery in Doctor’s Clothing

388 views

Published on

Presentation at ACM SIGIR 2016 Medical Information Retrieval (MedIR) Workshop (http://medir2016.imag.fr), July 21, 2016. Joint work with Gordon V. Comack, An T. Nguyen, Thomas A. Trikalinos, Byron C. Wallace.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
388
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Systematic Review is e-Discovery in Doctor’s Clothing

  1. 1. Systematic Review is e-Discovery in Doctor’s Clothing Joint work with Matt Lease ir.ischool.utexas.edu slideshare.net/mattlease @mattlease ml@utexas.edu Gordon V. Cormack (U. Waterloo) An Thanh Nguyen (U. Texas) Thomas A. Trikalinos (Brown U.) Byron C. Wallace (U. Texas)
  2. 2. “The place where people & technology meet” ~ Wobbrock et al., 2009 www.ischools.org 2
  3. 3. • System-Reviews • Electronic Discovery (e-Discovery) • Toward a Joint Research Agenda 3Matt Lease <ml@utexas.edu> Roadmap
  4. 4. • System-Reviews • Electronic Discovery (e-Discovery) • Toward a Joint Research Agenda 4Matt Lease <ml@utexas.edu> Roadmap
  5. 5. Evidence-Based Medicine n. The conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients 5
  6. 6. Systematic reviews: from biomedical articles to actionable evidence 6
  7. 7. PubMed ? 2 search database 1 formulate question, protocol & query 4 extract data treatment outcome ba c d 3 screen retrieved citations Studies AIMS1988 ASSET1988 Aber1976 Amery1969 Anderson1983 Bassand1986 Bett1973 Bossaert1987 Brunelli1988 Buchalter1987 Croydon1987 Dewar1963 Durand1987 ECSG−11979 ECSG−21988 EWP1971 Fletcher1959 GISSI1986 Gormsen1973 Guerci1987 Heikinheim1971 ISAM1986 ISISPilot1987 ISIS−21988 Ikram1986 Julian1987 Khaja1983 Leiboff1984 Maublant1988 Meinertz1988 NHFAustra1988 Olson1986 Raizner1985 Rentrop1984 Sainsous1986 Schreiber1986 Simoons1985 TICO1988 Topol1987 WWICSK1983 WWIVSK1988 White1987 Overall (I^2=19% , P=0.147) 0 0.01 0.02 0.04 0.08 0.190.270.38 0.76 1.91 3.82 7.65 18.26 OddsRatio(logscale) 5 synthesize extracted data 7 Formulate RQ & Boolean Query Boolean Search Document Collection All Tasks but #2 done manually by MDs
  8. 8. On average, 75 articles describing results from clinical trials are published every day. Bastian, PLoS Med, 2010 The median length to complete a single review: 1110 person-hours. Allen & Olkin, JAMA, 1998 8
  9. 9. 12 Technologies for semi-automated citation screening are relatively mature and slowly gaining acceptance
  10. 10. Research on citation screening • Methods for handling imbalance with asymmetric costs [ICDM 2011; ICDM 2012; KAIS 2013] • Active learning strategies [KDD 2010; SDM 2011; KDD 2013;] – Nguyen, Wallace, and Lease. Combining Crowd and Expert Labels using Decision Theoretic Active Learning. HCOMP 2015. • Test Collection: github.com/bwallace/crowd-sourced-ebm • Dually supervised methods [ICML 2011; KDD 2010] 13
  11. 11. • System-Reviews • Electronic Discovery (e-Discovery) • Toward a Joint Research Agenda 14Matt Lease <ml@utexas.edu> Roadmap
  12. 12. PubMed ? 2 search database 1 formulate question, protocol & query 4 extract data treatment outcome ba c d 3 screen retrieved citations Studies AIMS1988 ASSET1988 Aber1976 Amery1969 Anderson1983 Bassand1986 Bett1973 Bossaert1987 Brunelli1988 Buchalter1987 Croydon1987 Dewar1963 Durand1987 ECSG−11979 ECSG−21988 EWP1971 Fletcher1959 GISSI1986 Gormsen1973 Guerci1987 Heikinheim1971 ISAM1986 ISISPilot1987 ISIS−21988 Ikram1986 Julian1987 Khaja1983 Leiboff1984 Maublant1988 Meinertz1988 NHFAustra1988 Olson1986 Raizner1985 Rentrop1984 Sainsous1986 Schreiber1986 Simoons1985 TICO1988 Topol1987 WWICSK1983 WWIVSK1988 White1987 Overall (I^2=19% , P=0.147) 0 0.01 0.02 0.04 0.08 0.190.270.38 0.76 1.91 3.82 7.65 18.26 OddsRatio(logscale) 5 synthesize extracted data 15 Request for Production (RFP): Boolean Query Review Documents for “Responsiveness” Parties use documents Review Responsive Documents for Privilege Boolean Search Document Collection Electronically Stored Information (ESI) e.g., Enron email archive
  13. 13. Manual Review does not Scale 16 Paul, George L., and Jason R. Baron. Information inflation: Can the legal system adapt? Rich. JL & Tech. 13 (2007).
  14. 14. IR Research in e-Discovery • NIST TREC Track: 2006-2011 • Oard & Webber, FnTIR Book, 2013 • A variety of published work at SIGIR++ – e.g., Cormack & Grossman, SIGIR 2016 17
  15. 15. • System-Reviews • Electronic Discovery (e-Discovery) • Toward a Joint Research Agenda 18Matt Lease <ml@utexas.edu> Roadmap
  16. 16. Commonalities • Need high-recall with bounded cost • Follow 3-Stage Pipeline Today – Boolean query – Screening (traditionally manual by experts) – Final review & use • Pipeline approach useful but limits improvement – overall framing & unrecoverable errors • Limiting reliance on experts – Traditionally assumed to be infallible 19
  17. 17. Can we crowdsource screening? Michael Mortenson, Byron C. Wallace, Gaelen Adam, Tom Trikalinos and Tim Kraska. Crowdsourcing Citation Screening for Systematic Reviews. (Under review). 20
  18. 18. 21
  19. 19. Total Recall: Applications 22 E-Discovery
  20. 20. Total Recall: Strategies 23
  21. 21. Conclusion • Systematic Review & e-Discovery have much in common, but SR has received relatively little attention in IR – Open problems & current assumptions give IR researchers fertile opportunities for research beyond other IR tasks – Public test collections available for both • github.com/bwallace/crowd-sourced-ebm • Aaron Cohen’s: http://skynet.ohsu.edu/~cohenaa/systematic-drug- class-review-data.html – Reading list: https://github.com/bwallace/automating-ebm- resources/wiki/Papers • TREC Total Recall Track (trec-total-recall.org) offers a great forum for bringing together those interested 24
  22. 22. Thank You! ir.ischool.utexas.eduSlides: www.slideshare.net/mattlease 25

×