Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Optimizing Solr for Performance & User Experience

9,516 views

Published on

Using analytics to understand customer search behavior, we used a strategy pattern to return the right search strategy for each on-site search for a big boost in sales.

  • Be the first to comment

Optimizing Solr for Performance & User Experience

  1. 1. Optimizing Solr for Performance <br />& User Experience<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
  2. 2. <ul><li>Largest selection of sheet music
  3. 3. 12 year old online business
  4. 4. 600k+ SKUs
  5. 5. 1.5M Songs
  6. 6. Faceted navigation with Solr
  7. 7. On-site search with Solr
  8. 8. Lots of needles in a huge haystack</li></ul>SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll : @briandoll<br />
  9. 9. <ul><li>15 VMWare app server instances
  10. 10. 2k app requests per minute
  11. 11. MySQL database
  12. 12. 5 memcache instances</li></ul>SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
  13. 13. <ul><li>1k Solr requests per minute
  14. 14. ~100ms average response time
  15. 15. 3 Solr hosts on VMWare
  16. 16. 50% CPU load, 2GB memory each
  17. 17. 1.00 Apdex Score</li></ul>SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
  18. 18. We love that the structure of your data is part of your Solr environment (schema.xml)<br />And the way you query against that data is part of your app<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
  19. 19. on-site search is not as varied as we had thought<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
  20. 20. Surprisingly, lots of people search<br />like this:<br />Piano<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
  21. 21. What if we applied a strategy pattern to our search queries?<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
  22. 22. “Instrument” searches could get their own strategy.<br />“Artist” searches could get their own.<br />…<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
  23. 23. Make sure your default search is working well first!<br />Subtle changes to our algorithm provided a 32% increase in per-search value, with some phrases seeing an increase of over 3000%!<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
  24. 24. We construct various strategies:<br />...<br />DEFAULT_SEARCH_HASH = {<br /> :id_codes =&gt; { :search_type =&gt; :phrase, :boost =&gt; 1024, :slop =&gt; 2 },<br /> :bucket_proper_names =&gt; { :search_type =&gt; :phrase, :boost =&gt; 500, :slop =&gt; 2 },<br /> :bucket_titles =&gt; { :search_type =&gt; :phrase, :boost =&gt; 450, :slop =&gt; 100 },<br /> :bucket_songs =&gt; { :search_type =&gt; :phrase, :boost =&gt; 64, :slop =&gt; 5 },<br /> :bucket_primary_cats =&gt; { :search_type =&gt; :phrase, :boost =&gt; 32, :slop =&gt; 100},<br /> :bucket_secondary_cats =&gt; { :search_type =&gt; :phrase, :boost =&gt; 16, :slop =&gt; 100 },<br /> :bucket_binding =&gt; { :search_type =&gt; :phrase, :boost =&gt; 2, :slop =&gt; 100},<br /> :bucket_lead_time =&gt; { :search_type =&gt; :phrase, :boost =&gt; 1, :slop =&gt; 100},<br /> :bucket_blurbs =&gt; { :search_type =&gt; :phrase, :boost =&gt; 0.05, :slop =&gt; 7 },<br /> :text =&gt; { :search_type =&gt; :term, :boost =&gt; 0.005 }<br />}<br />...<br />def self.popular_instrument_strategy(search_phrase)<br />params = {:instrument_text =&gt; {:search_type =&gt; :phrase, :boost =&gt; 600, :slop =&gt; 2 } }<br />Strategy.new(&quot;default-instrument&quot;, search_phrase, params , :top_selling)<br />end<br />...<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
  25. 25. And then we have an algorithm to help determine which strategy we should apply to an incoming search request:<br />[redacted]<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
  26. 26. Implementing these strategies provided an 86% increase in per-search value.<br />We can now fine-tune individual categories of searches, as well as specific phrases.<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
  27. 27. Rule of thumb:<br />Don&apos;t try to please everybody. Please the people who make you money.<br />Tuning your search algorithm is an endless game, but if you focus on pleasing customer X, it&apos;ll cost you.<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />
  28. 28. In order to make all this happen, we relied heavily on two tools:<br />SFBay Apache Lucene/Solr Meetup - Jan 2010 Brian Doll :@briandoll<br />

×