Judge Facciola said that for one to "opine that a certain search term or terms would be more likely to produce information than [another] is truly to go where angels fear to tread. This presentation provides the tools needed to feel secure in conducting and comparing searches in electronic discovery. It covers: Why you would want to search What search does, How concept search can help, Dispels myths about concept search, Provides tools for measuring the efficacy of search and review.

1. Search Angels: Have no fear (well, not much any way) Herbert L Roitblat, Ph.D. OrcaTec LLC © 2009 OrcaTec LLC

2. Search Angels

3. Whether search terms or “keywords” will yield the information sought is a complicated question involving the interplay, at least, of the sciences of computer technology, statistics and linguistics. … Given this complexity, for lawyers and judges to dare opine that a certain search term or terms would be more likely to produce information than the terms that were used is truly to go where angels fear to tread. Judge John Facciola, United States v. O’Keefe , No. 06-249 (D.D.C. Feb. 18, 2008).

4. Why Search?

5. Responsive documents come to you

6. Searching and Filtering ?

7. Protect Privilege Consistency Quality

8. Organize

9. Focus

10. Defend

11. Sedona Best Practice Commentary on Search Courts have not only accepted, but in some cases have ordered, the use of keyword searching to define discovery parameters and resolve discovery disputes.

12. What is there to fear in search?

14. Unicode When is ２ ≠ 2? ２ 2 U+ff12 ≠ U+0032 Characters are not always what they seem When Ｕｎｉｃｏｄｅ ≠ U n i c o d e

15. The Defendants are regrettably vague in their description of the seventy keywords used for the text-searchable ESI privilege review, how they were developed, how the search was conducted, and what quality controls were employed to assess their reliability and accuracy. -- Judge Paul Grimm Victor Stanley v. Creative Pipe, Inc.

16. Blair and Maron Blair & Maron Study: 20% recall Lawyers thought that they had virtually all relevant documents, They had only 20% Failed to anticipate all wordings Wording depends on point of view Defense: “Unfortunate incident” Plaintiff: “Disaster” Blair and Maron, Communications of the ACM , 28, 1985, 289-299 “ It is impossibly difficult for users to predict the exact words, word combinations, and phrases that are used by all (or most) relevant documents and only (or primarily) by those documents.”

17. Strike Bank Words are ambiguous

18. How ambiguous? Look it up! The companies have agreed to a brief delay in implementing their agreement. 37 14 39 17 54 62 20 8 84 8 7 9 7,788,584,618,680,320 possible interpretations Each word disambiguates the others # definitions

19. Overcoming that fear So, first of all, let me assert my firm belief that the only thing we have to fear is fear itself -- nameless, unreasoning, unjustified terror which paralyzes needed efforts to convert retreat into advance. Franklin D. Roosevelt

20. Judge Facciola I bring to the parties’ attention recent scholarship that argues that concept searching, as opposed to keyword searching, is more efficient and more likely to produce the most comprehensive results. Disability Rights Council of Greater Wash. v. Wash. Metro. Area Transit Auth., 2007 WL 1585452 (D.D.C. June 1, 2007) See George L. Paul & Jason R. Baron, Information Inflation: Can the Legal System Adapt? 13 Rich. J.L. & Tech. 10 (2007).

21. Black Box? Difficult to understand? Difficult to use? Difficult to defend? Concept Search How does it work?

22. inspect forage looking look for investigate explore SEARCH lookup seek examine hunt hunting Thesaurus

23. Droids Humanoid droids Nonhumanoid droids R-Series droids Droid Taxonomy http://starwars.wikia.com R5-D4 Astromech IG-88 Assassin R2-D2 Astromech C-3PO Protocol 2-1B Surgical T3-M4 Utility

24. Ontology Attorney Lawyer Esq Paralegal Legal assistant Synonym Synonym Associated Associated Law firm Associated Associated Associated

25. Machine learning Neural networks Language modeling Statistical pattern learning Bayesian classifiers Latent semantic indexing Probabilistic classifiers Cluster analysis

26. Concept Search Example Search for: 化粧 Makeup

27. Context for 化粧 化粧 Makeup 肌 Skin 美 Beauty 美容 Beauty of figure or form ケア Care dhc Brand of skin care products スキン Skin キレイ Pretty 始め Origin 口コミ Word of mouth 通販 Mail order

28. Expand query to give context 化粧 化粧 , 美容 , 肌 , 美 , ケア , dhc , スキン , キレイ , 始め , 口コミ , 通販 Query as entered Query as searched Terms are weighted by language model

29. Design Execute Evaluate Sound methodology

30. Evaluation: Human review (TREC08) 71.3% overall agreement http://trec.nist.gov/pubs/trec17/papers/LEGAL.OVERVIEW08.pdf 82% 18% Previously judged nonresponsive 42% 58% Previously judged responsive Currently judged nonresponsive Currently judged responsive

31. Measuring accuracy Precision: The proportion of gold vs. rock in the pan. The proportion of responsive vs. nonresponsive documents that were retrieved. Recall: The proportion of the gold in the river that was found. The proportion of the responsive documents that have been found.

32. Very low recall, except for one service provider’s concept system

33. Common sense dictates that sampling and other quality assurance techniques must be employed to meet requirements of completeness. -- Judge David A. Baker The only prudent way to test the reliability of the keyword search is to perform some appropriate sampling of the documents determined to be privileged and those determined not to be. -- Judge Paul Grimm

34. Elusion: The proportion of gold among the rocks discarded from the pan. The proportion of nonretrieved documents that were responsive. Simple evaluation: Sample & accept on zero

35. Accept on zero Collect sample. Accept batch only if there are 0 defects in the sample. Sample size depends on the maximum acceptable probability of a defect and confidence level.

36. Search Angel: Fear not You’ve got friends Concept Search Sound methodology Empirical evidence Sampling Quality assurance

37. Herbert L Roitblat, Ph.D. OrcaTec LLC www.orcatec.com [email_address] 805-918-4612