Faceted Search – the 120 Million Documents Story

1. Faceted Search – the 120 Million Documents Story

2. Who am I? <ul><li>My (Buddhist) name is Upayavira </li></ul><ul><li>Consultant with Sourcesense, specialising in search and operational technologies </li></ul><ul><li>A member of the Apache Software Foundation </li></ul>

3. Who are Sourcesense? <ul><li>Open Source integrator, specialising in: </li></ul><ul><ul><ul><li>Search </li></ul></ul></ul><ul><ul><ul><li>Business Intelligence </li></ul></ul></ul><ul><ul><ul><li>Content Management </li></ul></ul></ul><ul><ul><ul><li>Application Lifecycle Management </li></ul></ul></ul><ul><li>Offices in London, Amsterdam, Milan and Rome </li></ul>

4. Committers and Contributors <ul><li>Search: </li></ul><ul><ul><ul><li>Lucene/Solr – contributor </li></ul></ul></ul><ul><ul><ul><li>Hibernate Search – committer </li></ul></ul></ul><ul><ul><ul><li>Lucene Infinispan integration – lead developer </li></ul></ul></ul><ul><ul><ul><li>Apache UIMA – committer </li></ul></ul></ul><ul><li>CMS: </li></ul><ul><ul><ul><li>Apache Chemistry – contributor </li></ul></ul></ul><ul><ul><ul><li>Apache Jackrabbit – contributor </li></ul></ul></ul><ul><ul><ul><li>JBoss GateIn Portal – committer </li></ul></ul></ul><ul><ul><ul><li>OpenSSO-Alfresco - contributor </li></ul></ul></ul>

5. Who is the customer? <ul><li>News search provider </li></ul><ul><li>Industry leader </li></ul><ul><li>Has 100s of servers crawling 1.7m sites each day </li></ul><ul><li>2.5m documents (news and social media) each day </li></ul><ul><li>Keeping 2m/day, 1 month = 60m, 2 months = 120m </li></ul><ul><li>Existing tech old and fragile </li></ul>

6. Their story? <ul><li>Aim: fast and timely search across broad range of content </li></ul><ul><li>Refreshing of their infrastructure: maintainable </li></ul><ul><li>Features: </li></ul><ul><ul><li>Integrated search across news/social media content </li></ul></ul><ul><ul><li>Faceting </li></ul></ul><ul><ul><li>Geospacial search </li></ul></ul><ul><ul><li>Deduplication </li></ul></ul><ul><ul><li>Clustering </li></ul></ul>

7. The Solution:

8. The Solution: Apache Solr <ul><li>Fast inverted index </li></ul><ul><li>Flexible configurable schema </li></ul><ul><li>Faceted search </li></ul><ul><li>Sharding for scalability </li></ul><ul><li>Missing: </li></ul><ul><ul><li>Clustering (Google news style) </li></ul></ul><ul><ul><li>Geolocation </li></ul></ul><ul><ul><li>Deduplication </li></ul></ul><ul><ul><li>Relevance </li></ul></ul>

9. How Solr Works <ul><li>A RESTful web service </li></ul><ul><ul><li>http://solr:8983/solr/select?q=notebook </li></ul></ul><ul><li> </li></ul>

10. How Solr Works <ul><li>A RESTful web service </li></ul><ul><ul><li>http://solr:8983/solr/select?q=notebook </li></ul></ul><ul><li><result name="response" numFound="649" start="0"> <doc> <str name="name">Thinkpad</str> <str name="make">Lenovo</str> </doc> … </result> </li></ul>

11. How Solr Works <ul><li>A RESTful web service </li></ul><ul><ul><li>http://solr:8983/solr/select?q=notebook </li></ul></ul><ul><li><result name="response" numFound="649" start="0"> <doc> <str name="name">Thinkpad</str> <str name="make">Lenovo</str> </doc> … </result> </li></ul><ul><ul><li>http://solr:8983/solr/select?q=notebook&facet=true&facet.field=make </li></ul></ul>

12. How Solr Works <ul><li>A RESTful web service </li></ul><ul><ul><li>http://solr:8983/solr/select?q=notebook </li></ul></ul><ul><li><result name="response" numFound="649" start="0"> <doc> <str name="name">Thinkpad</str> <str name="make">Lenovo</str> </doc> … </result> </li></ul><ul><ul><li>http://solr:8983/solr/select?q=notebook&facet=true&facet.field=make </li></ul></ul><ul><li><lst name="facet_counts"> <lst name="facet_fields"> <lst name="make"> <int name="Lenovo">24</int> <int name="Apple">12</int> … </lst> </lst> </lst> </li></ul>

13. How Solr Works Index

14. How Solr Works Index Index Snapshot Active Index Reader Searches

15. How Solr Works Index Index Snapshot Active Index Reader Searches New Content Active Index Writer

16. How Solr Works Index Index Snapshot Active Index Reader Searches New Content Active Index Writer commit

17. How Solr Works Index Index Snapshot Index Snapshot Index Reader Active Index Reader Searches New Content Active Index Writer

18. How Solr Works Index Index Snapshot Index Snapshot Index Reader Active Index Reader Searches New Content Active Index Writer

19. How Solr Works Index Index Snapshot Index Reader Searches New Content Active Index Writer

20. How Solr Distributes <ul><li>Too many: </li></ul><ul><ul><li>Documents for one index </li></ul></ul><ul><ul><li>Requests for one server </li></ul></ul><ul><ul><li>Chances of failure </li></ul></ul><ul><li>Shards: splitting each index into parts </li></ul><ul><li>Rows: duplicating each index </li></ul>

21. Solr Host Configuration shard 1 shard 2 shard 3 searches

22. Solr Host Configuration shard 1 shard 2 shard 3 co-ordinator

23. Solr Host Configuration shard 1 shard 2 shard 3 co-ordinator load balancer

24. Solr Host Configuration shard 1 shard 2 shard 3 co-ordinator load balancer shard 1 shard 2 shard 3 co-ordinator

26. Solr at The Customer <ul><li>10 shards </li></ul><ul><li>1 co-ordinator </li></ul><ul><li>60m documents </li></ul><ul><li>350Gb indexes (35Gb each shard) </li></ul><ul><li>Indexer code written </li></ul><ul><li>Custom components implemented </li></ul><ul><li>Initial query time 20s </li></ul><ul><li>But soon: </li></ul>

27. Oops. OutOfMemoryError

28. Solr: a Java web application <ul><li>Runs in a Java VM </li></ul><ul><li>JVM manages memory: garbage collection </li></ul><ul><li>JVM allocates memory into buckets </li></ul><ul><li>On JVM startup: specify memory allocations </li></ul>

29. How Solr Works Index Index Snapshot Searches New Content Active Index Writer Active Index Reader

30. How Solr Works Index Index Snapshot Searches New Content Active Index Writer cache Active Index Reader

31. How Solr Works Index Index Snapshot Index Snapshot Index Reader Searches New Content cache Active Index Reader cache commit Active Index Writer

32. How Solr Works Index Index Snapshot Index Snapshot Index Reader Searches New Content Active Index Writer cache Active Index Reader cache

33. How Solr Works Index Index Snapshot Index Reader Searches New Content Active Index Writer cache

34. Optimisation #1: autowarm < listener event = "newSearcher" class = "solr.QuerySenderListener" > < arr name = "queries" > < lst > < str name = "q" > solr </ str > < str name = "relf" > 4 </ str > < str name = "facet.field" > sourceCountryCS </ str > < str name = "facet.field" > entityCSPerson </ str > < str name = "facet.field" > entityCSCompany </ str > < str name = "facet.field" > entityCSProduct </ str > < str name = "facet.field" > sourceCS </ str > < str name = "facet.field" > authorCS </ str > < str name = "facet.field" > stockTickerCS </ str > < str name = "facet.field" > feedClassCS </ str > < str name = "facet.field" > entityCSOrganization </ str > < str name = "facet.field" > platformCS </ str > < str name = "facet.field" > eventOrFactCS </ str > < str name = "facet.field" > sourceRank </ str > < str name = "facet" > true </ str > < str name = "facet.date" > harvestDate </ str > < str name = "facet.date.start" > NOW-1MONTH </ str > < str name = "facet.date.end" > NOW </ str > < str name = "facet.date.gap" > +24HOURS </ str > < str name = "qt" > /duplicate </ str > < str name = "duplicateOrder" > latest </ str > < str name = "collapseFields" > duplicateGroup titleForDuplicates </ str > </ lst > </ arr > </ listener >

35. #2: Garbage collection <ul><li>jstat -gcutil </li></ul><ul><li>S0 S1 E O P YGC YGCT FGC FGCT GCT 0.00 91.16 8.77 86.49 59.59 13075 3832.216 2598 1594.978 5427.193 0.00 91.16 9.05 86.49 59.59 13075 3832.216 2598 1594.978 5427.193 0.00 91.16 9.29 86.49 59.59 13075 3832.216 2598 1594.978 5427.193 0.00 91.16 9.29 86.49 59.59 13075 3832.216 2598 1594.978 5427.193 0.00 91.16 9.29 86.49 59.59 13075 3832.216 2598 1594.978 5427.193 0.00 91.16 9.29 86.49 59.59 13075 3832.216 2598 1594.978 5427.193 0.00 91.16 9.29 86.49 59.59 13075 3832.216 2598 1594.978 5427.193 0.00 91.16 9.61 86.49 59.59 13075 3832.216 2598 1594.978 5427.193 0.00 91.16 9.61 86.49 59.59 13075 3832.216 2598 1594.978 5427.193 </li></ul><ul><li>MaxSearchers: </li></ul><ul><li><maxWarmingSearchers>1</maxWarmingSearchers> </li></ul><ul><li>Rebalanced the memory allocation for JVM: </li></ul><ul><li>-XX:SurvivorRatio=5 -XX:NewRatio=2 -Xmx5300m -Xms5300m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m </li></ul>

36. #3: Profiling <ul><li>Only needed because of custom components </li></ul><ul><li>Discovered 1Gb of cache reallocated every commit </li></ul><ul><li>Mostly a key, stored as a string </li></ul><ul><li>Converted to a 'long' number </li></ul><ul><li>Reduced to 100Mb </li></ul>

37. Managing So Many Hosts <ul><li>With eleven (or 22) hosts, manual building prohibitive </li></ul><ul><li>Hosted at Amazon EC2 </li></ul><ul><li>Scripted instantiation and configuration: </li></ul><ul><ul><li>installing java, creating/mounting partition, etc </li></ul></ul><ul><li>Run concurrently </li></ul><ul><ul><li>Result: 11 hosts available in six minutes </li></ul></ul>

38. Solr Host Configuration shard 1 shard 2 shard 3 co-ordinator load balancer

40. Solr Host Configuration shard 1 shard 2 shard 3 co-ordinator load balancer shard 1 shard 2 shard 3 co-ordinator 35Gb 35Gb 35Gb

42. Solr Host Configuration shard 1 shard 2 shard 3 co-ordinator load balancer shard 1 shard 2 shard 3 co-ordinator Entire row: 40 minutes

43. Content Archiving <ul><li>Important to have the ability to re-index </li></ul><ul><li>Kept a copy of all content pre-ingestion </li></ul><ul><li>Built a tool to ingest this archive </li></ul><ul><li>30 minutes for 1 day, 30 hours for 2 months </li></ul>

44. Being Dynamic <ul><li>Schema changes require re-indexing </li></ul><ul><li>Resharding requires re-indexing </li></ul><ul><li>Automation gave major benefit: </li></ul><ul><ul><li>Can deploy an additional row and reindex </li></ul></ul>

46. Solr Host Configuration shard 1 shard 2 shard 3 co-ordinator load balancer shard 1 shard 2 shard 3 co-ordinator shard 1 shard 2 shard 3 co-ordinator

47. Solr Host Configuration shard 1 shard 2 shard 3 co-ordinator load balancer shard 1 shard 2 shard 3 co-ordinator shard 1 shard 2 shard 3 co-ordinator archive ingestion

48. Solr Host Configuration shard 1 shard 2 shard 3 co-ordinator load balancer shard 1 shard 2 shard 3 co-ordinator shard 1 shard 2 shard 3 co-ordinator

50. Conclusion <ul><li>Service live, and increasing towards 120m docs </li></ul><ul><li>Faceted queries between 1s and 2s. </li></ul><ul><li>Term queries 500ms. </li></ul><ul><li>GC time down from >5000s to 700s/day </li></ul><ul><li>To do still: </li></ul><ul><ul><li>Clustering </li></ul></ul>

51. thank you [email_address]