Successfully reported this slideshow.

Michigan Information Retrieval Enthusiasts Group Meetup - August 19, 2010


Published on

Two presentation from the Michigan Information Retrieval Enthusiasts Group Meetup on August 19 by Cengage Learning search platform development team.

Scaling Performance Tuning With Lucene by John Nader discusses primary performance hot spots related to scaling to a multi-million document collection. This includes the team's experiences with memory consumption, GC tuning, query expansion, and filter performance. Discusses both the tools used to identify issues and the techniques used to address them.

Relevance Tuning Using TREC Dataset by Rohit Laungani and Ivan Provalov describes the TREC dataset used by the team to improve the relevance of the Lucene-based search platform. Goes over IBM paper and describe the approaches tried: Lexical Affinities, Stemming, Pivot Length Normalization, Sweet Spot Similarity, Term Frequency Average Normalization. Talks about Pseudo Relevance Feedback.

Published in: Technology
  • Be the first to comment

Michigan Information Retrieval Enthusiasts Group Meetup - August 19, 2010

  1. 1. Michigan Information Retrieval Enthusiasts Group Meetup Sponsored by Cengage Learning August 19, 2010
  2. 2. Meetup Agenda <ul><li>6:30-7:00pm – Introductions </li></ul><ul><ul><ul><li>Cengage Learning, Duane May </li></ul></ul></ul><ul><li>7:00-8:30pm – Main Presentations: </li></ul><ul><ul><ul><li>Scaling Performance Tuning With Lucene, John Nader </li></ul></ul></ul><ul><ul><ul><li>Relevancy Tuning Using TREC Data, Rohit Laungani, Ivan Provalov </li></ul></ul></ul><ul><li>8:30-9:00pm – Open Discussion </li></ul>
  3. 3. Scaling Lucene John Nader Software Developer
  4. 4. Agenda <ul><li>Platform Profile </li></ul><ul><li>Architecture Overview </li></ul><ul><li>Adventures in Scalability </li></ul><ul><li>Partitioning </li></ul><ul><li>Memory Usage </li></ul><ul><li>Performance Optimization </li></ul><ul><li>Wrap-up and Discussion </li></ul>
  5. 5. Platform Profile <ul><li>179 Million Documents (4.5 TB) </li></ul><ul><ul><li>Books, Periodicals, Multi-Media, Archives </li></ul></ul><ul><li>60M Terms across 150 Fields per Partition </li></ul><ul><li>6000+ Content Sets - Filters </li></ul><ul><li>Content Rights Management – More Filters </li></ul><ul><li>Accurate Hit Counts </li></ul><ul><li>One Back-End for Multiple Products </li></ul><ul><li>Ranging from chatty portal products to pure library search </li></ul>
  6. 8. Architecture
  7. 9. Runtime Architecture Partition 1 Basic Services Partition 18 Partition 2 Entity Services Web Products EMC Ice Ice NFS NFS NFS
  8. 10. Partition Architecture Search & Retrieve Services Lucene (v3.0.2) Doc Vault JRE 1.6 CPU: AMD 64 Bit 8 Core RAM: 32GB SuSe Linux Ice IIOP (ZeroC)
  9. 11. Not Using SOLR….Yet <ul><li>Implementation Pre-dates SOLR </li></ul><ul><ul><li>Coupling between search and doc retrieval </li></ul></ul><ul><ul><li>Many features embedded in implementation </li></ul></ul><ul><ul><li>Currently Looking at how we could migrate to SOLR </li></ul></ul>
  10. 12. Adventures in Scaling
  11. 13. Impact of Partitioning <ul><li>Over response time much larger than average partition response time. </li></ul><ul><li>Why? </li></ul><ul><ul><li>Clue #1: It’s not the time to combine the results… </li></ul></ul><ul><ul><li>Clue #2: It’s not the network overhead… </li></ul></ul><ul><ul><li>Clue #3: Statistics catch up with you! </li></ul></ul><ul><li>Broker must wait for slowest Partition </li></ul><ul><ul><li>Given a 50% chance a partition responds within 200ms </li></ul></ul><ul><ul><li>Chance all partitions respond within 200ms is: </li></ul></ul><ul><ul><li>(0.5) number of partitions </li></ul></ul>
  12. 14. Partition vs. Overall
  13. 15. Partition vs. Overall
  14. 16. Partition Conclusions <ul><li>If you are considering a partition/shard strategy… </li></ul><ul><ul><li>Plan for the overall response </li></ul></ul><ul><ul><li>Balance between gain of smaller partitions and the loss of waiting for slowest response </li></ul></ul><ul><ul><li>Also consider distribution and aggregation overhead. </li></ul></ul>
  15. 17. Memory Usage <ul><li>Java Heap Shared Between Document Vault and Search </li></ul><ul><ul><li>~3GB for doc vault (entries list, meta-data) </li></ul></ul><ul><ul><li>~3GB for content set filters (1.3M per bit set) </li></ul></ul><ul><ul><li>~2GB Lucene (Terms, Field Cache) </li></ul></ul><ul><ul><li>~1GB Custom Facet Browse Support ( future presentation? ) </li></ul></ul><ul><li>Lucene Memory Mapped Files </li></ul><ul><ul><li>~4GB </li></ul></ul><ul><li>Memory Related Issues: </li></ul><ul><ul><li>Long running GCs </li></ul></ul><ul><ul><li>No room for new features (content rights, custom products) </li></ul></ul>
  16. 18. Lucene Mapped Files
  17. 19. Lucene Mapped Files <ul><li>~15% in memory </li></ul><ul><li>May grow with new products/features </li></ul><ul><li>Still have problems with restarts </li></ul><ul><ul><li>Lucene 2.4.0 lazy-loads terms </li></ul></ul><ul><ul><li>Lucene 3.0.2 improved loading of terms </li></ul></ul><ul><ul><li>Still have issues with first result retrieval </li></ul></ul><ul><ul><li>Plan: Implement ‘Priming Queries’ </li></ul></ul>
  18. 20. Java GC Issues <ul><li>Measure First </li></ul><ul><ul><li>Tools: Eclipse Heap Analysis Tool, jmap, jstat </li></ul></ul><ul><ul><li>Full scale load tests </li></ul></ul><ul><ul><li>Monitor production </li></ul></ul><ul><li>Observations </li></ul><ul><ul><li>Random Heap Spikes </li></ul></ul><ul><ul><li>Heap jump by GBytes in a few seconds </li></ul></ul><ul><ul><li>Continuous GC 5 - 30 minutes! </li></ul></ul><ul><ul><li>Node becomes unresponsive </li></ul></ul>
  19. 21. GC: Cause and Solution <ul><li>Wildcard Query Expansion </li></ul><ul><ul><li>e.g. KE:* </li></ul></ul><ul><ul><li>Iterates over all matching terms to construct query </li></ul></ul><ul><ul><li>Creates lots of objects ( at least in Lucene 2.4 ) </li></ul></ul><ul><ul><li>Lucene documentation warns against using wildcards </li></ul></ul><ul><li>Mitigation Strategies </li></ul><ul><ul><li>Block wildcards on < 3 characters for large fields </li></ul></ul><ul><ul><li>Interrupt ‘runaway’ threads </li></ul></ul><ul><ul><li>Consider the TimedHitCollector ( would like a TimedWildCardExpander ) </li></ul></ul><ul><li>Result: Significantly Reduced Heap Spikes </li></ul>
  20. 22. Additional GC Tuning <ul><li>Java Heap & Memory Mapped Files </li></ul><ul><ul><li>20GB Heap + 4GB MMap keeps us out of swap </li></ul></ul><ul><ul><li>~10GB hard usage (post-GC) </li></ul></ul><ul><li>GC Still an issue </li></ul><ul><ul><li>Full GC once or twice a day </li></ul></ul><ul><ul><li>1 - 2 minute pauses </li></ul></ul><ul><li>Actions: </li></ul><ul><ul><li>Implemented Concurrent Mark Sweep GC </li></ul></ul><ul><ul><li>Lower GC threshold to 67% </li></ul></ul><ul><li>Result: </li></ul><ul><ul><li>Full GC more frequently, but in much less time (15s) </li></ul></ul><ul><ul><li>Less than 1s pause during GC </li></ul></ul>
  21. 23. Performance Challenges <ul><li>Some legacy code performs at 1M docs, but not 10M </li></ul><ul><li>Some new features created bottlenecks </li></ul><ul><li>Content Sets force excessive filtering </li></ul><ul><li>Hit counts force extra processing </li></ul>
  22. 24. Example: Index Browser <ul><li>Requirement: Retrieve a page of terms starting at a given term… </li></ul><ul><ul><li>Get TermEnum - IndexReader.terms(term) </li></ul></ul><ul><ul><li>Get next 50 terms – </li></ul></ul><ul><li>… now add limiting by Content Set </li></ul><ul><ul><li>Iterate over each Term’s Documents </li></ul></ul><ul><ul><ul><li>indexReader.termDocs(term) </li></ul></ul></ul><ul><ul><ul><li> </li></ul></ul></ul><ul><ul><li>Check each doc until at least one is found in Content Set </li></ul></ul><ul><ul><li>Repeat with next Term until you find 50 Terms </li></ul></ul><ul><li>… now add counts by Term </li></ul><ul><ul><li>Must iterate over all docs in Term </li></ul></ul><ul><ul><li>Test if in content set and count </li></ul></ul>
  23. 25. Index Browser Problems <ul><li>Bigger fields (e.g. Keywords) and Smaller Content Sets </li></ul><ul><li>Worst Case Scenario </li></ul><ul><ul><li>Less than 50 docs in Content Set </li></ul></ul><ul><ul><li>Traverse all docs in all terms </li></ul></ul><ul><ul><li>10M docs x 6M terms </li></ul></ul><ul><li>Actually Slower on Lucene 3.0.2 </li></ul><ul><li>Put the Lucene upgrade in jeopardy </li></ul>
  24. 26. Index Browsing Solutions <ul><li>Investigated Root Cause </li></ul><ul><ul><li>Profiler, Stack Dumps, Lucene Source Code, and the Lucene Forum </li></ul></ul><ul><li>Found two different issues </li></ul><ul><ul><li>Lucene 3.0.2 added synchronization </li></ul></ul><ul><ul><li>Java 1.6.0_12 had JIT issues </li></ul></ul><ul><li>Results </li></ul><ul><ul><li>Forum suggested AllTermDocs and seek </li></ul></ul><ul><ul><li>Upgraded to Java 1.6.0_21 </li></ul></ul><ul><ul><li>Performance improved by 10x </li></ul></ul><ul><ul><li>Looking at FilterIndexReader, FilterTermDocs </li></ul></ul>
  25. 27. Other Enhancements <ul><li>Custom implementation for small filters </li></ul><ul><li>Improved filter combination/traversal </li></ul><ul><li>Faceted Search Optimizations </li></ul><ul><li>Caching term docs for smaller content sets </li></ul>
  26. 28. Scaling Lucene: Take-aways <ul><li>Scaling Horizontally is good, but not Linear </li></ul><ul><li>Consider decoupling Lucene from other services (may scale differently) </li></ul><ul><li>Legacy algorithms may not scale </li></ul><ul><li>Beware of Hit Counts </li></ul><ul><li>Full scale load testing is a must! </li></ul>
  27. 29. Discussion
  28. 30. Relevancy Tuning Using TREC Data Rohit Laungani – Senior Systems Analyst Ivan Provalov – Information Architect / Developer
  29. 31. <ul><li>“ For Google, the quality of search has always been about getting you the exact, most relevant answer you were looking for in the shortest amount of time.. These notions of relevance and speed have been baked into our product development and are always a top priority for us... In fact, at any given time we're conducting between 50 and 200 search experiments , all of which are focused on getting you the exact result you're looking for -- faster ” </li></ul><ul><li>Jack Menzel, Group Product Manager, Search @ Google </li></ul>Background - What are others doing?
  30. 32. <ul><li>“ Early this year, we saw a lot of evidence that people are getting much more sophisticated in their searching, asking Google to solve harder problems (for example, by making longer and more complex queries). For this reason, in 2009 alone we have released many improvements: nearly 500 ranking changes” </li></ul><ul><li>Jack Menzel, Group Product Manager, Search @ Google </li></ul>Background - What are others doing?
  31. 33. IR Background – Precision/Recall <ul><li>Recall - fraction of relevant documents which has been retrieved. </li></ul><ul><li>Precision - fraction of retrieved documents which is relevant. </li></ul>Collection Answer Set A Relevant Docs in Result Set Ra Relevant Docs R Entire Collection
  32. 34. IR Background - Relevance
  33. 35. IR Background – Recall-Precision Graph
  34. 36. Mean Average Precision (MAP) Average of precision values at the points at which each relevant document is retrieved. R N N N R N N R R R Calculate precision at each point in the ranking where we find a relevant document, then average these values: 1/5 * (1/1 + 2/5 + 3/8 + 4/9 + 5/10) = 0.544 To get a single-number measure across N , say 50, queries, simply average N MAP scores.
  35. 37. Background -TREC <ul><li>TREC: Text REtrieval Conference ( </li></ul><ul><li>Annual conference since 1992, co-sponsored by the National Institute of Standards and Technology (NIST) and DARPA. </li></ul><ul><li>Aim to improve evaluation methods and measures in IR by increasing the research in IR using relatively large test collections on a variety of datasets </li></ul><ul><li>TREC workshops consists of a set tracks, areas of focus in which particular retrieval tasks are define </li></ul><ul><li>Track creates necessary infrastructure ( test collections, evaluation methodology etc) to support research on its tasks </li></ul><ul><li>Participants submit the P/R values for the final document and query corpus and present their results at the conference </li></ul>
  36. 38. Sample TREC query <top> <num> Number: 305 <title> Most Dangerous Vehicles <desc> Description: Which are the most crashworthy, and least crashworthy, passenger vehicles? <narr> Narrative: A relevant document will contain information on the crashworthiness of a given vehicle or vehicles that can be used to draw a comparison with other vehicles. The document will have to describe/compare vehicles, not drivers. For instance, it should be expected that vehicles preferred by 16-25 year-olds would be involved in more crashes, because that age group is involved in more crashes. I would view number of fatalities per 100 crashes to be more revealing of a vehicle's crashworthiness than the number of crashes per 100,000 miles, for example. </top> LA031689-0177 FT922-1008 LA090190-0126 LA101190-0218 LA082690-0158 LA112590-0109 FT944-136 LA020590-0119 FT944-5300 LA052190-0048 LA051689-0139 FT944-9371 LA032390-0172 LA042790-0172 LA021790-0136 LA092289-0167 LA111189-0013 LA120189-0179 LA020490-0021 LA122989-0063 LA091389-0119 LA072189-0048 FT944-15615 LA091589-0101 LA021289-0208
  37. 39. <DOCNO> LA031689-0177 </DOCNO> <DOCID> 31701 </DOCID> <DATE><P>March 16, 1989, Thursday, Home Edition </P></DATE> <SECTION><P>Business; Part 4; Page 1; Column 5; Financial Desk </P></SECTION> <LENGTH><P>586 words </P></LENGTH> <HEADLINE><P>AGENCY TO LAUNCH STUDY OF FORD BRONCO II AFTER HIGH RATE OF ROLL-OVER ACCIDENTS </P></HEADLINE> <BYLINE><P>By LINDA WILLIAMS, Times Staff Writer </P></BYLINE> <TEXT> <P>The federal government's highway safety watchdog said Wednesday that the Ford Bronco II appears to be involved in more fatal roll-over accidents than other vehicles in its class and that it will seek to determine if the vehicle itself contributes to the accidents. </P> <P>The decision to do an engineering analysis of the Ford Motor Co. utility-sport vehicle grew out of a federal accident study of the Suzuki Samurai, said Tim Hurd, a spokesman for the National Highway Traffic Safety Administration. NHTSA looked at Samurai accidents after Consumer Reports magazine charged that the vehicle had basic design flaws. </P> <P>Several Fatalities </P> <P>However, the accident study showed that the &quot;Ford Bronco II appears to have a higher number of single-vehicle, first event roll-overs, particularly those involving fatalities,&quot; Hurd said. The engineering analysis of the Bronco, the second of three levels of investigation conducted by NHTSA, will cover the 1984-1989 Bronco II models, the agency said. </P> <P>According to a Fatal Accident Reporting System study included in the September report on the Samurai, 43 Bronco II single-vehicle roll-overs caused fatalities, or 19 of every 100,000 vehicles. There were eight Samurai fatal roll-overs, or 6 per 100,000; 13 involving the Chevrolet S10 Blazers or GMC Jimmy, or 6 per 100,000, and six fatal Jeep Cherokee roll-overs, for 2.5 per 100,000. After the accident report, NHTSA declined to investigate the Samurai. </P> ... </TEXT> <GRAPHIC><P> Photo, The Ford Bronco II &quot;appears to have a higher number of single-vehicle, first event roll-overs,&quot; a federal official said. </P></GRAPHIC> <SUBJECT> <P>TRAFFIC ACCIDENTS; FORD MOTOR CORP; NATIONAL HIGHWAY TRAFFIC SAFETY ADMINISTRATION; VEHICLE INSPECTIONS; RECREATIONAL VEHICLES; SUZUKI MOTOR CO; AUTOMOBILE SAFETY </P> </SUBJECT> </DOC> Sample TREC Document
  38. 40. Project Goals <ul><li>Baseline Current Search Platform’s Quality of Retrieval </li></ul><ul><li>Create Integration Tests for the Platform and Run These Regularly </li></ul><ul><li>Evaluate Chinese-based analyzers </li></ul>
  39. 41. Methodology <ul><li>TREC Datasets (English and Chinese) </li></ul><ul><li>IBM Paper 2007 TREC </li></ul><ul><ul><li>Lexical Affinities </li></ul></ul><ul><ul><li>Stemming </li></ul></ul><ul><ul><li>Pivot Length Normalization </li></ul></ul><ul><ul><li>Sweet Spot Similarity </li></ul></ul><ul><ul><li>Term Frequency Average Normalization </li></ul></ul><ul><li>BM25 </li></ul><ul><li>Pseudo Relevance Feedback </li></ul>
  40. 42. Pivoted Length Normalization Relevance Document Length P L p
  41. 43. Pivoted Length Normalization <ul><li>Boosts shorter documents, “punishes” longer ones </li></ul><ul><li>U is the number of unique words in the document </li></ul><ul><li>Pivot is the average of U over all documents </li></ul><ul><li>Length normalization: </li></ul><ul><li>1/((1 - slope) + slope * (U) / (pivot)) </li></ul>
  42. 44. Example of Pivoted Length Normalization
  43. 45. Pseudo Relevance Feedback <ul><li>Retrieve top 5-10 documents for a user query </li></ul><ul><li>Make the assumption these are relevant documents </li></ul><ul><li>Retrieve a few terms from these documents and expand the original query: </li></ul><ul><ul><li>Terms with high term frequencies </li></ul></ul><ul><ul><li>Terms with low document frequencies </li></ul></ul>
  44. 46. Runs Examples
  45. 47. Results - English <ul><li>Default Lucene – 0.149 on TREC-3 Collection (comparable with IBM findings) </li></ul><ul><li>Stemmer – 0.202 </li></ul><ul><li>LA & Stemmer & Phrase – 0.21 </li></ul><ul><li>BM25 – 0.168 </li></ul><ul><li>Sweet Spot Similarity – 0.173 </li></ul><ul><li>Pivoted Length Normalization – 0.184 </li></ul><ul><li>Pivoted Length & Term Frequency Normalization – 0.186 </li></ul><ul><li>Lucene With Porter Stemmer, Pivot Point Document Length Normalization and Pseudo Relevance Feedback (PRF) – 0.30 (100% improvement) </li></ul>
  46. 48. Results - Chinese <ul><li>Chinese Paoding Analyzer – 0.444 (x10 improvement over default after applying Position Filter fix as well as the PRF) </li></ul><ul><li>Rosette Chinese Analyzer – 0.393 (x10 improvement after applying the above techniques) </li></ul>
  47. 49. Conclusions <ul><li>Lucene default relevance ranking performs well, but could be tuned further </li></ul><ul><li>Simple techniques sometimes work best for relevance improvements (stemming, PRF) </li></ul><ul><li>Open source analyzers perform as well as commercial for Chinese corpus (Paoding) </li></ul>
  48. 50. References <ul><li>TREC http:// </li></ul><ul><li>Lucene http:// /java/docs/ </li></ul><ul><li>Introduction to IR </li></ul><ul><li>IBM TREC 2007 </li></ul><ul><li>Relevance http:// ) </li></ul>
  49. 51. Discussion
  50. 52. Collections <ul><li>English </li></ul><ul><ul><li>TREC 3 Ad Hoc Topics </li></ul></ul><ul><ul><li>TIPSTER Disk 1 and Disk 2 </li></ul></ul><ul><ul><li>ClueWeb09 “B” </li></ul></ul><ul><li>Chinese </li></ul><ul><ul><li>TREC5, TREC6 Topics </li></ul></ul><ul><ul><li>Peoples Daily newspaper </li></ul></ul><ul><ul><li>Xinhua newswire </li></ul></ul>