Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How Humans & Machines Can Improve Site Search Results - Search Y: Paris

524 views

Published on

When it comes to improving your site search results, it takes both human ingenuity and some machine learning. Learn how to improve your site search results with hands-on tactics & then learn how to structure a machine learning element to augment it. This presentation was delivered live and in person at the 2020 Paris Search Y Conference

Published in: Marketing
  • Be the first to comment

  • Be the first to like this

How Humans & Machines Can Improve Site Search Results - Search Y: Paris

  1. 1. 7 February 2020 | Paris - Bercy L’Événement Search Marketing PARIS 2020
  2. 2. 2 TEACHING MACHINES & HUMANS TO IMPROVE SITE SEARCH RESULTS JP SHERMAN MANAGER OF SEARCH & FINDABILITY
  3. 3. STATE OF SITE SEARCH Search is much larger than search engines. @JPSHERMAN
  4. 4. BIGGER THAN GOOGLE? Now, what about those non-search engine searches? Amazon, Facebook, Sohu, Weibo, Reddit, Instagram, Twitter, Ebay…. Web Search App Search GOOGLE’S 2 TRILLION PER YEAR SEARCH VOLUME 4 Maintaining Site search will ● Increase Conversions ● Reduce Abandonment ● Reinforce Expertise ● Deliver a Good User & Brand Experience @JPSHERMAN
  5. 5. SEARCH AS A BEHAVIOR IS FRACTURED THERE ARE MORE WAYS TO SEARCH THAN EVER. 5 Search isn’t just a search engine. It’s in an application, in IoT, in smart devices Findability Is: ● Understanding “How” ● Understanding Selection ● Understanding Behavior ● Understanding Intent @JPSHERMAN
  6. 6. IF THEY’RE SEARCHING ON YOUR SITE... IF THEY DON’T FIND IT, THEY WILL LEAVE YOU. THEY THINK YOU HAVE WHAT THEY’RE LOOKING FOR. 6 If a user cannot find what they’re looking for, they know that Google is less than a second away. ● They think you have what they want ● They’re probably right ● If it’s not findable ● They’re gone. @JPSHERMAN
  7. 7. IF THEY FIND IT, DO BALLOONS DROP? THAT’S THE EXPECTATION. NO. 7 @JPSHERMAN
  8. 8. USERS REMEMBER THEIR SITE SEARCH EXPERIENCE USERS ARE NOT KIND. 8 Clever girl... A poor search experience is remembered. ● Some trust is lost ● They’ll go to Google ● They may find what they’re looking for. ● Let's hope your competitor doesn’t rank. @JPSHERMAN
  9. 9. SEARCH BEHAVIOR: HOW … NOT WHAT... USERS SCAN WITH PURPOSE AND INTENT 9 Passive Search Active Search Users apply criteria as they scan through your results ● They have acceptance and rejection criteria ● They spend less than a second scanning a snippet ● Perception of Value is Critical @JPSHERMAN
  10. 10. SITE SEARCH BEHAVIORAL SCIENCE INFORMATION SCENT TRAILS USERS LOOK FOR “INFORMATION SCENT TRAILS” 10 USERS SCAN FOR PATTERNS ● They include elements of or related to their intent ● They look at textual, image proximities ● Active vs. Passive Scanning ● Value Signals. @JPSHERMAN
  11. 11. INFORMATION SCENT TRAILS A QUICK EXAMPLE 11 An intent based word- cloud. ● Users scan ● When words match intent ● Acceptance & Rejection Criteria. ● One will lead to an information trail. TYPES PROPERTIES @JPSHERMAN
  12. 12. USER PERCEPTION OF VALUE WITH INTENT, USERS LOOK FOR VALUE 12 Results for “Road Bikes” ● sigh. ● They all look alike ● Which one is good? @JPSHERMAN
  13. 13. USER PERCEPTION OF VALUE WITH INTENT, USERS LOOK FOR VALUE 13 Results for “Road Bikes” ● Value applied as metadata. ● Triggers for behavior ● Which one is better? @JPSHERMAN
  14. 14. THINGS HUMANS CAN DO TO IMPROVE RESULTS SPOILER ALERT: IT’S A LOT OF THE STUFF WE ALREADY DO 14 Actionable Tasks to Improve Site Search Results: ● Keyword Metadata ● Synonym Lists ● Boosted Results ● SERP Features ● Clickstream Data ● Personalization @JPSHERMAN
  15. 15. THINGS HUMANS CAN DO TO IMPROVE RESULTS IMPROVE THE SERP DESIGN 15 Actionable Tasks to Improve Site Search Results: ● SERP Features AUTOCOMPLETE/ AUTOSUGGEST FACETS KEYMATCH KNOWLEDGE GRAPH NATURAL RESULTS @JPSHERMAN
  16. 16. THINGS HUMANS CAN DO TO IMPROVE RESULTS LOCATION CAN BE A STRONG SIGNAL OF INTENT 16 Actionable Tasks to Improve Site Search Results: ● Personalization Keyword: Bike Tires Saint-Brieuc Bay Portes du Soleil Location Bias Can Deliver Intent Road Bike Tires Mountain Bike Tires @JPSHERMAN
  17. 17. THINGS HUMANS CAN DO TO IMPROVE RESULTS MEASURE HUMAN BEHAVIOR 17 Users apply criteria as they scan through your results ● Measure consumption & conversion ● Measure dwell time ● Measure time from query to conversion @JPSHERMAN
  18. 18. THINGS HUMANS CAN DO TO IMPROVE RESULTS SPOILER ALERT: IT’S A LOT OF THE STUFF WE ALREADY DO 18 Design your SERP for the user. ● SERP Design ● Accessibility for people with visual impairments ● Snippet Design ● Features ● Disambiguation @JPSHERMAN
  19. 19. SO.. UH… WHAT’S THE POINT? Site Search is a massive behavior across the web. 1. Simple changes to the search platform & content will pay off 2. Users who search your site think you have what they want 3. Metadata and what is displayed in the SERP influences CTR 4. Use boosting of content to quickly rank on your site-search 5. Consider Design 6. Consider Accessibility Don’t Be Google. Google has to figure out “everything”. You don't. Be Better Than Google. YOU CAN DO A LOT TO MAKE SITE SEARCH BETTER, BUT THERE’S MORE 19 @JPSHERMAN
  20. 20. UNDERSTANDING CONTEXT AT RED HAT, WE SELL FREE SOFTWARE. THIS IS WHO WE ARE 20 @JPSHERMAN
  21. 21. WE SUPPORT PEOPLE FIRST WHICH MEANS THAT WE ARE A SUBSCRIPTION & SUPPORT COMPANY. THIS IS MY CONTEXT 21 @JPSHERMAN
  22. 22. HAPPY PEOPLE RECOGNIZE VALUE THE FASTER PEOPLE FIND ANSWERS TO THEIR SUPPORT NEEDS, THE HAPPIER THEY ARE. THIS IS HOW WE SUCCEED. 22 @JPSHERMAN
  23. 23. SEARCH INTENT IS REALLY HARD PEOPLE LOOK FOR INFORMATION…. UNIQUELY. PEOPLE CAN BE WEIRD 23 my linux is broken @JPSHERMAN
  24. 24. SETTING UP THE MACHINE TO LEARN WOULDN’T IT BE GREAT IF WE COULD PREDICT INTENT? MAGIC ISN’T REQUIRED. 24 @JPSHERMAN
  25. 25. RISE OF THE MACHINES - GOALS - REDUCE ZERO RESULTS - IMPROVE MATCHING - IMPROVE CTR SOME THINGS TO REMEMBER 25 @JPSHERMAN
  26. 26. FIRST, LET’S LOOK AT THE DATA YOU HAVE. - UNSTRUCTURED DATA STRUCTURE IS VERY IMPORTANT 26 - STRUCTURED DATA @JPSHERMAN
  27. 27. NEXT, LET’S LOOK AT YOUR INFORMATION ARCHITECTURE - UNSTRUCTURED IA STRUCTURE IS STILL IMPORTANT 27 - STRUCTURED IA @JPSHERMAN
  28. 28. NOW, LET’S LOOK AT THE PLAN START SIMPLE, INCREASE COMPLEXITY 28 keywordsearch Taxonomies EntityExtraction Ontologies Queryintent Queryclassification Semantic parsing Clustering RelevancyTuning Signals A/BTesting LTR Self LearningGoal Reference: https://www.slideshare.net/treygrainger/intent-algorithms Trey Grainger, SVP Engineering Lucidworks @JPSHERMAN
  29. 29. THE PLATFORM OF THE ENGINE SEARCH PLATFORM FREE, OPEN-SOURCE & POWERFUL - LUCENE IS POWERFUL. 29 Lucene is a Java-based, free and open-source search engine software platform. Lucene has several different flavors. ● Apache Nutch ● Apache Solr ● Compass ● CrateDB ● DocFetcher ● ElasticSearch ● Kinosearch ● Swiftype @JPSHERMAN
  30. 30. THE MACHINE LEARNING APPLICATION DATA SET THESE ARE YOUR QUERIES OR CONTENT YOU WANT TO LEARN FROM… FOR EXAMPLE. “ARE THESE CATS OR DOGS?” DATA SETS, TRAINING SETS AND HOW TO MEASURE. 30 @JPSHERMAN
  31. 31. THE MACHINE LEARNING APPLICATION TRAINING SET PARTITION YOUR DATA SET TO 10% EVALUATION, 90% TRAINING. DATA SETS, TRAINING SETS AND HOW TO MEASURE. 31 DATA SET: TRAINING SET: EVALUATION @JPSHERMAN
  32. 32. THE MACHINE LEARNING APPLICATION TO QUANTIFY “DOG-NESS” OR “CAT-NESS” A POWERFUL FORMULA IS THE OKAPI BM25 FORMULA. DATA SETS, TRAINING SETS AND HOW TO MEASURE. 32 Good resources on data science formulation. https://www.datasciencecentral.com/profiles/blogs/140-machine-learning-formulas Reference: https://www.amazon.com/Solr-Action-Trey-Grainger/dp/1617291021 Reference: https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables @JPSHERMAN
  33. 33. THE MACHINE LEARNING APPLICATION DATA OUTPUT AND FRACTIONAL SCORES. HOW CLOSE TO “CAT-NESS” IS THIS? DATA SETS, TRAINING SETS AND HOW TO MEASURE. 33 0.99732134 0.87569821 0.62587471 0.0000111 @JPSHERMAN
  34. 34. THE MACHINE LEARNING APPLICATION WHAT IF THE QUERY ISN’T “CATS” BUT “FUZZY ANIMALS”? THIS IS A CORE VALUE OF STRUCTURED MARKUP AS AN ATTRIBUTE SIGNAL DATA SETS, TRAINING SETS AND HOW TO MEASURE. 34 0.9632115 0.9178585 0.9244844 0.9371025 @dawnieandois an inspirationto all of us. @JPSHERMAN
  35. 35. THE RECIPE FOR MACHINE LEARNING LEARN TO RANK PLUGIN (LTR) IT’S NOT AS HARD AS YOU MAY THINK. 35 LTR is a Lucene-compatible plugin that allows the application of machine learning. It uses a wide variety of ranking signals. ● QUERY INDEPENDENT: Looks only at the body of indexed content ● QUERY DEPENDENT: Looks at both the query and the document, most often a TF- IDF score. ● QUERY LEVEL FEATURES: Looks only at the query @JPSHERMAN
  36. 36. HOW MACHINES PREDICT INTENT A SUPPORT ORGANIZATION’S PRIMARY TASK IS TO HELP THIS IS THE CORE OF FINDABILITY - TO DELIVER THE RIGHT INFORMATION - AT THE RIGHT TIME - TO RESOLVE AN ISSUE - QUICKLY - THROUGH ANY CHANNEL THROUGH THE CONTEXT OF SUPPORT 36 @JPSHERMAN
  37. 37. CONFIGURING SYSTEM FOR EXPERIMENTS STARTING SMALL: - WE SELECTED A SINGLE PRODUCT - RED HAT OPENSHIFT - WE SELECTED A SINGLE USE-CASE - TROUBLESHOOTING - WE SELECTED THE CONTENT - SOLUTION CONTENT TYPE MORE DATA SETS, TRAINING SETS AND EVALUATION SETS 37 @JPSHERMAN
  38. 38. CONFIGURING SYSTEM FOR EXPERIMENTS STARTING SMALL: - SOLUTION CONTENT IS 30% OF CONTENT. MORE DATA SETS, TRAINING SETS AND EVALUATION SETS 38 DOCUMENTATION VIDEOS ARTICLES SECURITY PRODUCT SOLUTION @JPSHERMAN
  39. 39. CONFIGURING SYSTEM FOR EXPERIMENTS APPROXIMATELY 100K INDIVIDUAL PIECES OF CONTENT. - 90K WENT TO TRAINING PARTITION - 10K WENT TO EVALUATION MORE DATA SETS, TRAINING SETS AND EVALUATION SETS 39 @JPSHERMAN
  40. 40. RUN THE RELEVANCY ALGORITHM RUN THE EVALUATION - FIRST CHECK: - DOES THIS LOOK RIGHT? - CONFIRM/ CORRECT SAMPLE - DEFINE “SUCCESS” WHAT PERCENT EQUALS “GOOD” LETS ASSUME FAILURE. WHAT CAN BE DONE? THIS REQUIRES SOME HUMAN INTERVENTION 40 @JPSHERMAN
  41. 41. RELEVENCY TUNING: THE MACHINE PARTS FIXING THE MACHINE FIRST 41 - CHECK “WEIGHTS”: - SIGNAL WEIGHT - CTR WEIGHT - IMPRESSION WEIGHT - SYNONYM WEIGHT - INTERNAL LINK WEIGHT - SERP IMPRESSIONS @JPSHERMAN
  42. 42. RELEVENCY TUNING: THE METADATA PARTS TUNING THE UNDERLYING STRUCTURE OF THE DATA 42 - CHECK “WEIGHTS”: - TITLE, DESCRIPTION & KEYWORD METADATA - STRUCTURED MARKUP - SCHEMA - PAGE COMPONENTS ABSTRACT CONCLUSION COMMENTS - ENTITIES/ TAXONOMIES/ ONTOLOGIES @JPSHERMAN
  43. 43. RELEVENCY TUNING: THE CONTENT PARTS PRACTICAL EXAMPLE OF WHY CONTENT IS KING 43 - CHECK CONTENT: - IS YOUR CONTENT DESCRIPTIVE? - DOES IT HAVE DIVERSE LANGUAGE/ WORDS? - IS IT ORGANIZED? - ARE THEIR DIFFERENT CONTENT TYPES? THIS IS “GOOD SEO FOR CONTENT” @JPSHERMAN
  44. 44. TUNING THE MACHINE FOR IMPROVEMENT TRAINING DATA - CONTROL GROUP EVALUATION DATA - EXPERIMENTAL GROUP USE CTR AS A SUCCESS SIGNAL DETERMINE SIGNIFICANCE CONFIRM WITH METRICS. 44 @JPSHERMAN
  45. 45. PUTTING IT ALL TOGETHER CONTENTQUALITY - WRITE GREAT, DESCRIPTIVE CONTENT METADATA & IA - DONTIGNORE YOURMETADATA, SCHEMA & STRUCTURE MACHINE RELEVANCY - THE MACHINE WILL ATTEMPTTO UNDERSTAND, BUT TUNING REQUIRES HUMANS. MEASUREMENT - DEFINE WHAT SUCCESS IS, SEGMENT& MEASURE CTR. MACHINE LEARNING & INTENTION DETECTION IS A BALANCING ACT 45 @JPSHERMAN
  46. 46. THANK YOU SO MUCH 46@JPSHERMAN PLEASE FEEL FREE TO TALK TO ME. BECAUSE NO ONE DOES THIS ALONE… THANK YOU: - JASON BARNARD - MANIKANDAN SIVANESAN - JIM SCARBOROUGH - DAWN ANDERSON - MARIANNE SWEENY - TREY GRAINGER - CHARLIE HULL - JAIMIE ALBERICO - HAMLET BATISTA - BRITNEY MULLER - MARTHA VAN BERKEL - GRANT INGERSOLL - JR OAKES - MICHAEL KING - MARK TRAPHAGEN - BARRY ADAMS - JENN HOFFMAN - JENNY HALASZ

×