Site Search Analytics: Conversations with your customers

Slide 1: Site Search Analytics: Conversations with your customers Louis Rosenfeld lou@louisrosenfeld.com Search Solutions Jumpstart April 17, 2008 ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 3: Anatomy of a search log (from Google Search Appliance) Critical elements in bold: IP address, time/date stamp, query, and # of results: XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystyleshe et=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ie=UTF-8&client=www&q=license+plate &ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XXX.XXX. X.104 HTTP/1.1" 200 8283 146 0.16 XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystyleshe et=www&q=regional+transportation+governance +commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17 ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 6: Querying the queries: Generic questions to help get started 1. What are the most frequent unique queries? 2. Are frequent queries retrieving quality results? 3. Click-through rates per frequent query? 4. Most frequently clicked result per query? 5. Which frequent queries retrieve zero results? 6. What are the referrer pages for frequent queries? 7. Which queries retrieve popular documents? 8. What interesting patterns emerge in general? ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 7: Improving metadata and navigation: How SSA can help 1. Analyze for metadata value tone, granularity, and trends 2. Determine (and invest in) important metadata attributes 3. Study queries that follow navigation failures/changes to create better contextual navigation 4. Look to improve “top-down” navigation (e.g., indices, taxonomy, main page) ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 8: Determining metadata values through clustering One week’s top 50 queries, clustered by topic Represents 20% of all search activity < 1 hour of work Can help suggest preferred terms ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 9: Determining metadata values through similar queries BBC explores similar queries (i.e., collaborative ﬁltering) Helps with misspellings, synonyms, best bets ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 10: Determining metadata attributes (AKA types, ﬁelds) Top queries clustered and prioritized by attributes:  Place  Dept./ Program  Service  Task 1-2 hours work ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 12: Supporting contextual navigation through embedded queries SLI Systems can display other queries related to a particular query… …and queries related to a particular document ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 14: Improving content: How SSA can help 1. Verify content spidering 2. Plug content gaps (placeholders, new content, appropriate types, best bets) 3. Improve content positioning (highlighting, demotion, deletion) 4. Prioritize content development (invest in appropriate topics and types, more eﬃcient migration and inventory) ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 15: Developing new content: Addressing emergent content needs Using SSA to plug gaps 0 results report (from behaviortracking.com)  Failure analysis suggests unmet content needs by topic (e.g., frequent queries for “iPod wristband” and “iPod holder for jogging”)  Content typing (covered later) suggests what kind of content to create (e.g., “product comparison” and “speciﬁcations”) ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 16: Developing new content: Addressing emergent content needs Use SSA to develop editorial agenda; determine relative “demand” for content topics and document types Available content (supply) Content by query (demand) ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 18: Monitoring trends: Tuning content to meet demand How might such trends impact your content? Search engine? Navigation? From behaviortracking.com ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 21: Comparing short head and long tail queries (from Vanguard) Short head: common queries Long tail: common queries Beneficiary form 403(b)(7) account asset transfer 401(k) authorization beneficiary automatic investing career Wire transfer instructions forms adoption agreement amt international wire transfers money market socially responsible investing location Vanguard tax identification number loans IRA Asset Transfer form calculator fdic insured account early withdrawal penalties ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 22: The shape of your content: Determining content types “What kind of page would users want when they searched this term?” Top queries clustered and prioritized by content types:  Application  News/ Announce- ments  Main page  Contact info  Instructions 1-2 hours work ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 23: Content types help expose content models, improve navigation album pages artist descriptions TV listings album reviews discography artist bios ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved. 23

Slide 27: Improving search: How SSA can help 1. Understand how search works (and what can be changed/ﬁxed) 2. Validate “the box” UI design 3. Look for opportunities to “soup up” queries (e.g., syntax, spell-checking) 4. Design better search results (individual and groups) 5. Blow up “advanced search” and contextualize functions 6. Buy a better engine—or improve your own ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 28: Improving “the box”: Initial query entry UI Validate width of query entry ﬁeld  Use query data to calculate “high number” (e.g., 90th percentile) for query length in characters  Does your entry ﬁeld accommodate that length? Amazon: 65 characters Microsoft: 25 characters ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 29: Improving “the box”: Search suggestions Populate query entry with common queries (Ask) Alternative: use controlled vocabulary (Netﬂix movie titles) ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 30: Query enhancement: Handling unique query types Little ﬁxes can add up  Spell-checking  Proper names (people, places)  Unique identiﬁers (e.g., SKUs, ISBNs) ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 32: Search results design: What to show per individual result …more descriptive information …less information for known- for open-ended searchers item searchers ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 33: Search results design: SSA helps determine sorting Financial Times found a high level of dates within queries Solution: support chronological sorting and ﬁltering ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 34: Search results design: SSA helps determine typing Content typing can enable:  Filtering by exposing types (HP)  Faceted classiﬁcation/ navigation (WebMD) ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 35: Search results design: Which results where? #10 result is clicked through more often than #s 6, 7, 8, and 9 (ten results per page) From SLI Systems (www.sli-systems.com) ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 38: How we determine what users want Qualitative research (observation)  Field studies  Focus groups  Card sorting  Usability testing and task analysis Quantitative research (data-driven)  Help/reference desk/switchboard  Web analytics (SEO/SEM, clickstream, SSA) ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 39: SSA as a user research method: How is it diﬀerent? Data that’s:  Real  High volume  Readily available Analysis that’s:  Quantitative  Inexpensive  Scalable  Complementary to other user research methods ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 40: Comparing SSA with other quantitative user research methods Data-driven methods  Clickstream/server log analysis: tells you where more than what users wants (complementary)  Event logging: comparable but often less data and usually at a different point in customer/user lifecycle  SEO/SEM: Different query types (finding where the answer is versus finding the answer); often focused on commerce conversions ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 42: SSA and qualitative user research methods What questions lead to Why questions Examples  Personas: SSA supplements ﬁctitious construct with real data (i.e., likely queries a persona might have)  Task analysis: SSA helps identify common tasks to evaluate  Field studies: SSA suggests issues to monitor in the ﬁeld ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 43: Augmenting personas and audience proﬁles with frequent queries Persona example (from Adaptive Path) Frequent queries added (in green) ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 44: Using common queries to enhance/fuel task analyses Use for tasks that are more what than how (i.e., testing information needs rather than functions) Examples • Can you ﬁnd a map of the campus? • What study abroad options are available to MSU students? • When is the last home football game of the season? ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 45: Identifying trends to monitor during ﬁeld studies Example: catalog clothing retailer SSA showed:  Preponderance of SKUs, even though they were not displayed on the site  What were they doing there? Subsequent ﬁeld study showed:  Customers preferred print catalog to web catalog to identify products  Customers preferred web catalog for ordering Action: add SKUs to web catalog ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 46: Combining quantitative and qualitative Exploring the needs of a particular audience: users of BBC children’s content ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 47: The power of data: Why UX needs SSA 1. Balance: UX methodologies/practitioners are often quantitatively weak 2. Legitimacy: Data-driven analyses make an impression on people hard to impress 3. Cost: The data is available; the analysis scales well with available time 4. Fidelity: The data is behavioral, real, and voluminous 5. Comprehensiveness: Quantitative methods complement and help improve qualitative methods (and vice versa) ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 48: Please share your SSA knowledge: Visit our book in progress site Search Analytics for Your Site: Conversations with your Customers by Louis Rosenfeld and Richard Wiggins (Rosenfeld Media, 2008) Site URL: www.rosenfeldmedia.com/books/searchanalytics/ Feed URL: feeds.rosenfeldmedia.com/searchanalytics/ ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 49: Slides and discounts Slides available here: http://www.slideshare.net/lrosenfeld For a discount, use code… FOEARLEY10 …at the Rosenfeld Media site for 10% oﬀ all of our titles ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slide 50: Contact Information Louis Rosenfeld Rosenfeld Media, LLC 705 Carroll Street, #2L Brooklyn, NY 11215 USA lou@louisrosenfeld.com www.louisrosenfeld.com www.rosenfeldmedia.com ©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

Slideshare.net (beta)