Successfully reported this slideshow.

Our Tale from the Trail of Shadows at REI Co-op - Chris Phillips & Dale Smith, REI Co-op

1

Share

1 of 41
1 of 41

More Related Content

More from Lucidworks

Related Books

Free with a 30 day trial from Scribd

See all

Our Tale from the Trail of Shadows at REI Co-op - Chris Phillips & Dale Smith, REI Co-op

  1. 1. Our Tale From the Trail of Shadows Chris Phillips and Dale Smith REI Co-op
  2. 2. Who Are We? Dale Smith ●  REI Co-Op for 5 years ○  Lead Engineer: Search Platform ●  40 years of software engineering ●  Boeing manufacturing automation ●  Seattle tech, including ○  Go2Net: Dogpile & Metacrawler @dasmith836
  3. 3. Who Are We? Chris Phillips ●  REI Co-op for 7 years 5 Years on Search ●  20 years in the industry ○  Half Life ○  Exchange Indexing ○  Launch of Xbox Kinect @realchrispdad
  4. 4. 01 Who is REI?
  5. 5. Agenda •  What We Did •  Where We Were •  The Big Challenges •  Solutions – The Cool Stuff •  Questions
  6. 6. What We did We replaced our SaaS search engine with an on-premise Solr stack
  7. 7. Where We Were Stuck in the mud!! ●  Not meeting customer or internal expectations ●  Slow searches: 750-900 ms ●  Random outages ●  Not flexible for our business partners ●  Limited visibility into the servers
  8. 8. 01 The Goal REI Co-Op: Quality Comes First
  9. 9. 01 The Goal Customer Expectations ●  Find the most relevant content quickly ●  Go outside and be happy
  10. 10. The Goal Internal Expectations ●  Fast and stable search platform ●  Rapid development cycle ●  Save budget on licensing and support
  11. 11. The Big Challenges No changes to the existing API’s ●  Website ●  Mobile apps ●  Point of Sale apps ●  Internal tools Limit SEO impact
  12. 12. The Big Challenges We were on our previous search engine for 4 years. ●  Had to tune relevance and merchandising rules to match in a matter of months ●  Internal users had to learn new tools ●  Biggest Challenge: How do we verify we haven’t broken something in search?
  13. 13. 01 The Big Challenges The Solr Stack Is Awesome, But Scary ●  Lucidworks Fusion ●  Innovent Solutions Findtuner ●  Multi-term synonyms ●  “47 knobs that all go to Volume 11!”
  14. 14. The Trail of Success Build a Lean Dream Team ●  2 Senior Back-End Developers ●  1 Senior QA Analyst ●  1 SDET ●  PM created a “bubble of safety” ●  Product Owner is the search expert at REI
  15. 15. The Trail of Success Nurture the Team Dynamic ●  Trusted by leadership ●  We trusted each other ●  Accountability ●  Committed to quality ●  We knew what success looked like
  16. 16. The Trail of Success Embrace The Team Philosophy
  17. 17. The Trail of Success Quality: The measure of the excellence of your product. ● “Quality is not an act. It is a habit.” -- Aristotle
  18. 18. The Trail of Success Be Serious About Who Owns Quality ● Everyone owns quality ● Build good habits
  19. 19. The Trail of Success Good Habits ● Co-locate ● Keep the business owner involved ● Engineers are cool! ● Find bugs before code is written ○ Architect together ○ Mental lifting ○ Risks ○ Unit tests ○ Test scenarios
  20. 20. The Trail of Success Good Habits ● Avoid legacy code with 3 questions ○ Testable ○ Supportable ○ Maintainable
  21. 21. Solutions - Use Great Tools ●  Innovent Solutions FindTuner ●  Lucidworks Fusion ●  Solr Cloud ●  These tools were designed to work together
  22. 22. Solutions - Innovent Solutions FindTuner ●  Manages search rules ○ Product promotion ○ Banners, redirects, etc. ○ Query adjustments ●  UI For Business Users ●  Eliminated the need to implement this in- house to replicate SaaS solution
  23. 23. Solutions - Lucidworks Fusion ●  Manages Solr Cloud configuration ●  Provides configurable search pipelines ●  Nice UI for business users ●  Many more features!
  24. 24. Solutions - A Great Software Ecosystem ●  Continuous Integration Model ●  Docker Platform ●  Microservices Architecture ●  High-functioning Monitoring & Metrics
  25. 25. Solutions – Dynamic Facets The Problem ●  ~200,000 product variants ●  ~250 “facet-worthy” attributes ●  Asking Solr to calculate facets for all of these attributes on each request would lead to significant performance issues
  26. 26. Because Two Queries Are Better Than One! ●  Indexed products with list of important attributes ●  First Query: read and calculate 10-15 key attributes for facet request 01 Solutions – Dynamic Facets ●  Second Query: full request with selected facets ●  Solr query only ever includes the 10-15 most important facets
  27. 27. The Problem: ● How do we verify product results from our new search stack? ● How do we verify performance capabilities under production load? Solutions – The Shadow Query The Solution: ● “Shadow Query”: all production searches are sent to both search engines, the results compared using special comparison functions, and metrics recorded on everything. ● All while not affecting existing search performance! ● Basically, we let our customers help test the system
  28. 28. Solutions – The Shadow Query Model
  29. 29. Solutions – The Shadow Query Model
  30. 30. Solutions – The Shadow Query Model
  31. 31. Solutions – Search API Normalization REI search API is search-engine agnostic ●  Avoid exposure of search engine-specific parameters or attributes ●  Preserve existing API contract ●  Preserve existing SEO patterns ●  Enable queries to multiple search engines
  32. 32. Solutions – Search API Normalization An Example Normalized Refinement on REI.com: r=features:Insulated;size:Large;color:Purple fq=features:"Insulated"&fq={!tag=SIZE}size:"Large"&fq={! tag=COLORFAMILY}color_family:"Purple" Translated to Solr: q1=insulated&x1=tab.ftrs&q2=large&x2=tab.size&q3=purple&x3= colorfam Translated to SaaS:
  33. 33. Solutions – Result ComparisonsComparing the Shadow Query Results ●  Record query result times and deltas ●  Record total number of results and deltas ●  Record count of products in result A and not in result B, and vice- versa ●  Record Damerau-Levenshtein Distance, using product IDs instead of letters ●  Use graph dashboards to “visualize quality” as the search metrics converged
  34. 34. Calculated based on operations transforming a list of products from one sequence to another ● Insertion ● Deletion ● Substitution ● Transposition ● Each operation has a cost, weighted higher at the start of the sequence Solutions – Modified Damerau-Levenshtein Distance
  35. 35. Preparing for ‘Solr Day’ ● Load Test using recorded production queries replayed through the shadow query at high load ● Monitored CPU and Memory - It’s all about that swap space ● Diagnose crashes from good bots doing bad things
  36. 36. It’s On!!! We turned it on for our customers and….. ●  Nobody noticed anything had changed ●  They just enjoyed it being 600ms faster!
  37. 37. After The Party... We continued to use the Shadow Query to verify the new Solr stack ●  Kept making calls to the SAAS search engine in the background ●  Provided a safety net for an important sale period ●  Continued to use metrics to validate rule refinements
  38. 38. Wisdom From The Trail Pack It Out! ● Start with a solid team ● Trust matters in all directions ● Use the best tools ● Use data to validate everything ● Stop relying on hope and heroics and instead bake QA into the process and the system
  39. 39. Awesome People Innovent Solutions Jason Hellman, Anria Billavara, & Team Lucidworks Greg Harris & Support Team
  40. 40. Contact Us Dale Smith ● Twitter: @dasmith836 ● dasmith836@yahoo.com ● LinkedIn: dasmith836 ● Facebook: dasmith836 (hm, I’m sensing a pattern...) Chris Phillips ● Twitter: @realchrispdad ● LinkedIn: realchrispdad
  41. 41. Thank You

×