Optimizing Solr for Performance & User Experience

9,104 views
8,976 views

Published on

Using analytics to understand customer search behavior, we used a strategy pattern to return the right search strategy for each on-site search for a big boost in sales.

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
9,104
On SlideShare
0
From Embeds
0
Number of Embeds
43
Actions
Shares
0
Downloads
64
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • Optimizing Solr for Performance & User Experience

    1. 1. Optimizing Solr for Performance <br />& User Experience<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
    2. 2. <ul><li>Largest selection of sheet music
    3. 3. 12 year old online business
    4. 4. 600k+ SKUs
    5. 5. 1.5M Songs
    6. 6. Faceted navigation with Solr
    7. 7. On-site search with Solr
    8. 8. Lots of needles in a huge haystack</li></ul>SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll : @briandoll<br />
    9. 9. <ul><li>15 VMWare app server instances
    10. 10. 2k app requests per minute
    11. 11. MySQL database
    12. 12. 5 memcache instances</li></ul>SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
    13. 13. <ul><li>1k Solr requests per minute
    14. 14. ~100ms average response time
    15. 15. 3 Solr hosts on VMWare
    16. 16. 50% CPU load, 2GB memory each
    17. 17. 1.00 Apdex Score</li></ul>SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
    18. 18. We love that the structure of your data is part of your Solr environment (schema.xml)<br />And the way you query against that data is part of your app<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
    19. 19. on-site search is not as varied as we had thought<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
    20. 20. Surprisingly, lots of people search<br />like this:<br />Piano<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
    21. 21. What if we applied a strategy pattern to our search queries?<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
    22. 22. “Instrument” searches could get their own strategy.<br />“Artist” searches could get their own.<br />…<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
    23. 23. Make sure your default search is working well first!<br />Subtle changes to our algorithm provided a 32% increase in per-search value, with some phrases seeing an increase of over 3000%!<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
    24. 24. We construct various strategies:<br />...<br />DEFAULT_SEARCH_HASH = {<br /> :id_codes =&gt; { :search_type =&gt; :phrase, :boost =&gt; 1024, :slop =&gt; 2 },<br /> :bucket_proper_names =&gt; { :search_type =&gt; :phrase, :boost =&gt; 500, :slop =&gt; 2 },<br /> :bucket_titles =&gt; { :search_type =&gt; :phrase, :boost =&gt; 450, :slop =&gt; 100 },<br /> :bucket_songs =&gt; { :search_type =&gt; :phrase, :boost =&gt; 64, :slop =&gt; 5 },<br /> :bucket_primary_cats =&gt; { :search_type =&gt; :phrase, :boost =&gt; 32, :slop =&gt; 100},<br /> :bucket_secondary_cats =&gt; { :search_type =&gt; :phrase, :boost =&gt; 16, :slop =&gt; 100 },<br /> :bucket_binding =&gt; { :search_type =&gt; :phrase, :boost =&gt; 2, :slop =&gt; 100},<br /> :bucket_lead_time =&gt; { :search_type =&gt; :phrase, :boost =&gt; 1, :slop =&gt; 100},<br /> :bucket_blurbs =&gt; { :search_type =&gt; :phrase, :boost =&gt; 0.05, :slop =&gt; 7 },<br /> :text =&gt; { :search_type =&gt; :term, :boost =&gt; 0.005 }<br />}<br />...<br />def self.popular_instrument_strategy(search_phrase)<br />params = {:instrument_text =&gt; {:search_type =&gt; :phrase, :boost =&gt; 600, :slop =&gt; 2 } }<br />Strategy.new(&quot;default-instrument&quot;, search_phrase, params , :top_selling)<br />end<br />...<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
    25. 25. And then we have an algorithm to help determine which strategy we should apply to an incoming search request:<br />[redacted]<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
    26. 26. Implementing these strategies provided an 86% increase in per-search value.<br />We can now fine-tune individual categories of searches, as well as specific phrases.<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
    27. 27. Rule of thumb:<br />Don&apos;t try to please everybody. Please the people who make you money.<br />Tuning your search algorithm is an endless game, but if you focus on pleasing customer X, it&apos;ll cost you.<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
    28. 28. In order to make all this happen, we relied heavily on two tools:<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />

    ×