Search Analytics: Powerful diagnostics for your site

Search Analytics: Powerful diagnostics for your site Search Solutions Jumpstart Conference Call Series November 3, 2006 Louis Rosenfeld www.louisrosenfeld.com www.rosenfeldmedia.com/books/searchanalytics

About Me
Information architecture (IA) consultant; formerly president Argus Associates
Publisher and founder, Rosenfeld Media (www.rosenfeldmedia.com)
Background in librarianship/information science; consult for Fortune 500s
Co-author, Information Architecture for the World Wide Web (3rd edition out this fall)
Co-founder, Information Architecture Institute (www.iainstitute.org) and UXnet (www.uxnet.org)

Anatomy of a Search Log (from Google Search Appliance)
Critical elements in bold: IP address , time/date stamp , query , and # of results:
XXX.XXX.X.104 - - [ 10/Jul/2006:10:25:46 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q= lincense+plate &ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02
XXX.XXX.X.104 - - [ 10/Jul/2006:10:25:48 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&client=www&q= license+plate &ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16
XXX.XXX.XX.130 - - [ 10/Jul/2006:10:24:38 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q= regional+transportation+governance+commission &ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17
Full legend and more examples here: http://www.rosenfeldmedia.com/books/searchanalytics/blog/log_sample_google_appliance/

The Head, the Long Tail, and the Interesting Stuff in Between Sorting queries by frequency results in a Zipf Distribution Can we improve performance for the most popular queries?

SA as Diagnostic Tool: What can you fix or improve?
User Research
Interface Design: search entry interface, search results
Retrieval Algorithm Modification
Navigation Design
Metadata Development
Content Development

User Research: What do they want?…
SA is a true expression of users’ information needs (often surprising: e.g., SKU numbers at LL Bean; URLs at IBM)
Provides context by displaying aspects of single search sessions

User Research: …who wants it?…
What can you learn from knowing these things?
What specific segments want; determined by:
Security clearance
IP address
Job function
Account information
Which pages they initiate searches from

User Research: …and when do they want it?
Time-based variation (and clustered queries)
By hour, by day, by season
Helps determine “best bets” and “guide” development

Search Entry Interface Design: “The Box” or something else?
SA identifies “dead end” points (e.g., 0 hits, 2000 hits) where assistance could be added (e.g., revise search, browsing alternative)
Syntax of queries informs selection of search features to expose (e.g., use of Boolean operators, fielded searching)
… OR…

Search Results Interface Design: Which results where?
#10 result is clicked through more often than #s 6, 7, 8, and 9 (ten results per page)
From SLI Systems (www.sli-systems.com)

Search Results Interface Design: How to sort results?
Financial Times has found that users often include dates in their queries
Obvious but effective improvement: Allow users to sort by date

Search System: What to change?
Identify new functionality: Financial Times added spell checking
Retrieval algorithm modifications:
Deloitte, Barnes & Noble use SA to demonstrate that basic improvements (e.g., Best Bets) are insufficient
Financial Times weights company names higher

Navigation: Any improvements?
Michigan State University builds A-Z index automatically based on frequent queries

Navigation: Where does it fail?
Track and study pages (excluding main page) where search is initiated
Are there obvious issues that would cause a “dead end”?
Are there user studies that could test/validate problems on these pages?
Sandia Labs analyzes most requested documents to test content independent of site structure; results used to improve structure

Metadata Development: How do users express their needs?
SA provides a sense of tone: how users’ needs are expressed
Jargon (e.g., “cancer” vs. “oncology,” “lorry” vs. “truck,” acronyms)
Length (e.g., number of terms/query)
Syntax (e.g., Boolean, natural language, keyword)

Metadata Development: Which metadata values?
SA helps in the creation of controlled vocabularies
Terms are fodder for metadata values (e.g., “cell phone,” “JFK” vs. “John Kennedy,” “country music”), especially for determining preferred terms
Works with tools that cluster synonyms (example from www.behaviortracking.com), enabling concept searching and thesaurus development

Metadata Development: Which metadata attributes?
SA helps in the creation of vocabularies
Simple cluster analysis can detect metadata attributes (e.g., “product,” “person,” “topic”)
Look for variations between short head and long tail (Deloitte intranet: “known-item” queries are common; research topics are infrequent)
known-item queries research queries

Content Development: Do we have the right content?
SA identifies content that can’t be found (0 results)
Does the content exist? If so, there are wording, metadata, or spidering problems
If not, why not?
www.behaviortracking.com

Content Development: Are we featuring the right stuff?
Clickthrough tracking helps determine which results should rise to the top (example: SLI Systems)
Also suggests which “best bets” to develop to address common queries

Organizational Impact: Educational opportunities
SA is a way to “reverse engineer” how your site performs in order to:
Sensitize organization to analytics, specifically related to findability
Sensitize content owners/authors to benefits of good practices around content titling, tagging, and navigational placement

Organizational Impact: Rethinking how you do things
Financial Times learns about breaking stories from their logs by monitoring spikes in company names and individuals’ names and comparing with their current coverage
Discrepancy = possible breaking story; reporter is assigned to follow up
Next step? Assign reporters to “beats” that emerge from SA

The Ideal SLA report 1/2 (Avi Rappoport, SearchTools.com)
# searches for each week/month/quarter/year
Top 1% of queries (cluster by stem if possible)
Top 10% of no-matches queries
Top 10% of low-matches queries? (one to 4 hits, or more depending on site size)
# empty searches
Changes in these over the last week/month/quarter/year
Changes’ correlation to changes in the site, search engine, company profile

The Ideal SLA report 2/2 (Avi Rappoport , SearchTools.com)
Queries showing significant increases
Patterns in less-frequent queries -- names? places? web site addresses?
Top pages retrieved in search results and the queries that retrieved them
Queries that retrieved the best/most important pages
For search zones, create reports for each zone (will have significant impact on no-matches data)

SA as User Research Method: Sleeper, but no panacea
Benefits
Non-intrusive
Inexpensive and (usually) accessible
Large volume of “real” data
Represents actual usage patterns
Drawbacks
Provides an incomplete picture of usage: was user satisfied at session’s end?
Difficult to analyze: where are the commercial tools?
Ultimately an excellent complement to qualitative methods (e.g., task analysis, field studies)

SA Headaches: What gets in the way?
Lack of time
Few useful tools for parsing logs, generating reports
Tension between those who want to perform SA and those who “own” the data (chiefly IT)
Ignorance of the method
Hard work and/or boredom of doing analysis
From summer 2006 survey (134 responses) www.rosenfeldmedia.com/books/searchanalytics/blog/search_analytics_survey_result/

Please Share Your SA Knowledge: Visit our “book in progress” site
Search Analytics for Your Site: Conversations with your Customers by Louis Rosenfeld and Richard Wiggins (Rosenfeld Media, 2007)
Site URL: www.rosenfeldmedia.com/books/searchanalytics/
Feed URL: feeds.rosenfeldmedia.com/searchanalytics/
Site contains:
Reading list
Survey results
Perl script for parsing logs
Log samples
… and more

Contact Information
Louis Rosenfeld LLC
902 Miller Avenue
Ann Arbor, Michigan 48103 USA
[email_address]
www.louisrosenfeld.com
+1.734.302.3323 voice
+1.734.661.1655 fax

http://cre8pc.com	17
http://www.linkedin.com	3
http://www.slideshare.net	1
http://www.techgig.com	1

Search Analytics: Powerful diagnostics for your site

by Louis Rosenfeld

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 22

Statistics

Search Analytics: Powerful diagnostics for your site Presentation Transcript