More Related Content Similar to Site Search Analytics Workshop Presentation (20) More from Louis Rosenfeld (18) Site Search Analytics Workshop Presentation1. Site Search Analytics for a
Better User Experience
Louis Rosenfeld
lou@louisrosenfeld.com
Washington, DC • October 22, 2010
1
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
2. About me
2
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
3. Agenda
1. A Quick Demo
2. A Brief Introduction
3. Exercise: Pattern Analysis
4. Improving Metadata and Navigation
5. Exercise: Failure Analysis
6. Improving Content
7. Exercise: Session Analysis
8. Improving Search
9. Vanguard Case Study
10. Improving User Research and Web Analytics
11. “Advanced” Topics/Discussion
12. Technical Stuff
3
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
4. 1 A Quick Demo
4
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
5. 2 A Brief Introduction to
Site Search Analytics
5
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
6. Anatomy of a search log
(from Google Search Appliance)
Critical elements in bold: IP address, time/date stamp, query, and # of results:
XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystyleshe
et=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02
XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ie=UTF-8&client=www&q=license+plate
&ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XXX.XXX.
X.104 HTTP/1.1" 200 8283 146 0.16
XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystyleshe
et=www&q=regional+transportation+governance
+commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17
6
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
7. The Zipf Curve:
Short head, middle torso, long tail
7
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
9. Querying the queries:
Generic questions to help get started
1. What are the most frequent unique queries?
2. Are frequent queries retrieving quality results?
3. Click-through rates per frequent query?
4. Most frequently clicked result per query?
5. Which frequent queries retrieve zero results?
6. What are the referrer pages for frequent queries?
7. Which queries retrieve popular documents?
8. What interesting patterns emerge in general?
9
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
10. Tuning your questions:
From generic to speci c
Net ix asks
1. Which movies most frequently searched?
2. Which of them most frequently clicked through?
3. Which of them least frequently added to queue?
10
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
11. Tuning your questions:
From generic to speci c
Net ix asks
1. Which movies most frequently searched?
2. Which of them most frequently clicked through?
3. Which of them least frequently added to queue?
10
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
12. Tuning your questions:
From generic to speci c
Net ix asks
1. Which movies most frequently searched?
2. Which of them most frequently clicked through?
3. Which of them least frequently added to queue?
10
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
13. Tuning your questions:
From generic to speci c
Net ix asks
1. Which movies most frequently searched?
2. Which of them most frequently clicked through?
3. Which of them least frequently added to queue?
10
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
14. The point of SSA:
Help determine, meet users’ desires
11
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
15. How do we know what users want?
Determine what they’re doing (behavior)
Web analytics (SEO/SEM, clickstream, SSA)
Help/reference desk/switchboard
Eye tracking studies…
Determine why they’re doing it (intent)
Usability testing and task analysis
Field studies
Focus groups
Card sorting…
12
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
16. Squaring analytics and user research:
Top-Down and Bottom-Up
Top-Down Analysis
Align metrics with organizational goals/KPI
(Key Performance Indicators)
Benchmark, measure, monitor, optimize for the
world you know
Bottom-Up Analysis
Look for trends, patterns, and outliers through
exploratory data analysis (“play”)
Identify surprises: the world you don’t know
13
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
17. Bottom-Up analysis generates
patterns, diagnoses, improvements
Pro ling users’ needs: what more than why
Identifying and diagnosing problems with
Content
Navigation and metadata
Search: interface design and con guration
Designing more effectively through
More effective prioritization of your work
Asking better, clearer questions
Complementing and improving your user research
and evaluation
14
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
18. Top-Down Analysis starts with
your organization’s goals
Making money
• Commerce (e.g., Amazon)
• Generating leads (e.g., MetLife)
• Selling advertising (e.g., ESPN.com)
Saving money/time
• Support/self-service (e.g., IRS.gov)
KPI (e.g., “increase sales per visit”) are
composed of speci c metrics
15
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
19. 3 Try and Discuss:
Pattern Analysis
16
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
20. 4 Improving metadata
and navigation
17
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
21. 1) Determining metadata values
through clustering
One week’s top 50
queries, clustered
by topic
Represents 20% of
all search activity
< 1 hour of work
Can help suggest
preferred terms
18
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
22. 1a) Determining metadata values
through similar queries
• BBC explores similar
queries (i.e.,
collaborative
ltering)
• Helps with
misspellings,
synonyms, best bets
19
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
23. 1b) Determining term granularity
through session analysis
Steps
1. Sample a variety of sessions
2. Compare “starting points” (initial queries) with
“end points”
3. Any patterns in terms of speci city?
Example
“HP 1012 printer” -> “HP 1012 printer driver”
20
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
24. 2) Testing and tuning
metadata values
Approaches
Tracking appearances, disappearances, and
trends among terms
Testing terms by querying them
Deriving terms through “reverse lookup”
21
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
25. 2a) Term trend tracking
Which term deserves your attention? Flash in
pan, or something important?
Next steps and implications: add new term,
or replace/delete existing term
22
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
26. 2b) Term trend tracking
(sample report from ISYS)
23
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
27. 2c) Testing your
metadata values: steps
1. Choose a manageable number of common
queries (e.g., top 25)
2. Check for matches among metadata values
(e.g., query “campus map” matches metadata
term “map”)
3. Queries without corresponding metadata
values may indicate gaps in your metadata
4. You might nd new metadata values among
your queries with 0 or few results
24
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
28. 2d) Term derivation from
reverse lookup: steps
1. Identify “important” documents (e.g., popular,
or strategic, or new)
2. Generate list of queries that retrieve those
important documents
3. Determine if there are metadata terms that
correspond to those queries
4. If not, you’ve identi ed metadata gaps or
opportunities to tweak existing terms
25
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
29. 3) Determining metadata attributes
(AKA types, elds)
Top queries
clustered and
prioritized by
attributes:
Place
Dept./
Program
Service
Task
1-2 hours work
26
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
30. 3a) Good metadata attributes
improve navigation
Indirect improvements to any navigation system
driven by metadata:
Site-wide taxonomies
Local taxonomies
Thesauri
Direct improvements to other navigation
systems
Contextual navigation
A-Z indices
Main pages and other major pages
27
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
31. 4) Identifying failures to
address with better navigation
Do this if you have combined query and clickstream
data
Steps
1. Determine top failed queries (“desire paths”) from
critical pages (e.g., product pages)
2. Determine to which documents users navigate
once they give up on search
3. Build those results into these queries’ result pages
OR extrapolate and present those documents’
categories (if available)
28
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
32. 5) Better site-wide navigation
through A-Z indices: 3 approaches
1. Manual: grab top queries and insert in
index (embedding queries in links)
2. Automated: generate index of queries on
the y (embedding queries in links)
3. Hybrid: repurpose individual best bets
results as single alphabetized index (see
Michigan State University example)
29
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
33. 5a) Hybrid A-Z index example:
Michigan State University
Frequent “recycled”
keywords best bets
30
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
34. 6) Making the obvious obvious
on the main page
Making sure “maps” were on the MSU main page
(Though this didn’t entirely solve the problem)
31
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
35. 5 Try and discuss:
Failure Analysis
32
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
36. Sample failure report
from Harvard Business
School intranet
33
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
37. 6 Improving content
34
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
38. 1) Identifying/addressing
content gaps
Using SSA to plug gaps 0 results report
(from behaviortracking.com)
Failure analysis suggests
unmet content needs by
topic (e.g., frequent
queries for “iPod
wristband” and “iPod
holder for jogging”)
Content typing suggests
what kind of content to
create (e.g., “product
comparison” and
“speci cations”) 35
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
39. 1a) Identifying/addressing
content gaps
Simple...
1.Regularly review most frequent null result
queries
2.Try variants of each query to validate that the
documents are simply not there
3.If they’re not: create new content
Opportunity!
Your alternative meds site gets queries for
“Nyquil”
What do you do?
36
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
41. 2) Making relevant content even
more relevant
Three common problems
1. De cient titling
2. De cient or poorly applied metadata
3. Poor writing
38
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
42. 2a) A simple case of using the
right copy
Intuit intranet
• Searches for “get back overpaid wages”
retrieved 0 results
• Term clearly should been included in the
appropriate Intuit page
• Easy x!
39
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
43. 2b) Getting marketing to do
the right thing
Jargon vs. Plain Language at
Washtenaw Community College
• Online courses were marketed using College on
Demand (“COD”) and FlexEd
• Signup rates were lower than expected
• Compare with “online”
• SSA determined
which approach
was more effective
40
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
44. 2c) Getting content owners to do
the right thing
Sandia National Labs
• Regularly record which documents came up at
position #1 for 50 most frequent queries
• If and when that top document falls out of
position #1, document's owner is alerted
• Result: healthy dialogue (often about
following policies and procedures and their
value)
41
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
45. 2d) Getting content owners to do the
right thing: wielding reports
Connecting pages (and their owners) that are
found through search...
42
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
46. 2e) Getting content owners to do the
right thing: wielding reports
...with how those pages were found
43
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
47. 3) Learning how users
understand your content
Learn more about speci c queries by the
terms they co-occur with (see also: Session
Analysis)
BBC: why did
queries for “Farepak”
suddenly spike?
44
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
48. 3a) Learning how users
understand your content
Context revealed
• “Xmas hampers”
(what was stolen)
• “watch dog”
(consumer affairs
program)
• “in adminstration”
(Farepak went
bust)
45
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
49. 4) Learning from trends: seasonality
46
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
50. 4a) Learning from trends:
Tuning content to meet demand
How might such trends impact your content?
Search engine? Navigation?
From behaviortracking.com
47
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
51. 5) The “shape” of your information:
Determining content types
“What kind of page would users want when they searched this term?”
Top queries
clustered and
prioritized by
content types:
Application
News/
Announce-
ments
Main page
Contact info
Instructions
1-2 hours work
48
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
52. 5a) Using content types to expose,
improve contextual navigation
album pages artist descriptions
TV listings
album reviews discography artist bios
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved. 94
53. 5b) Using content types to improve
search results
50
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
54. 7 Try and discuss:
Session Analysis
51
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
55. Session analysis from TFAnet intranet
These queries co-occur within sessions: why?
!
!
!
52
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
56. TFAnet session analysis results
• Searches for “delta ICEG” perform
poorly (way below the fold)
• Users than try an (incorrect)
alternative (“delta learning team”)
53
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
57. 8 Improving search
54
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
58. Search systems: basic anatomy
…and know your own system; its
functionality will impact your query data
55
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
59. 1) Plugging gaps in your
search engine’s index
Search engines often don’t know about all
content (especially in large, distributed
web environments)
Analyze queries with null results--which
actually have no content?
Do you see a pattern? Does it suggest a
single department/sub-site/content type?
56
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
60. 2) Making “The Box” wide enough
Validate width of query entry eld
Use query data to calculate comfortably “high
number” for query length in characters
Does your entry eld accommodate that
length?
Amazon: 65 characters
Microsoft: 25 characters
57
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
61. 2a) Making “The Box” wide enough
Top 500 queries from AIGA.org (37% of all
search activity) • Mean/median=@10
• Get rid of outliers
(e.g., 62, 36)
• Low 20s seems
“comfortable”
• Adjust for longer
long tail queries
!
!
58
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
62. 3) Making query entry easier:
Type ahead/auto-complete
Populate query entry with common queries (Ask)
Better approach: reuse keywords from Best Bets
development (MSU)
59
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
63. 3a) Making query entry easier:
Type ahead/auto-complete
Alternatives to using query data: existing metadata
(e.g., ESPN player names, Net ix movie titles)
60
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
64. 4) Supporting search re nement:
What exactly is “advanced”?
61
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
65. 4) Supporting search re nement:
What exactly is “advanced”?
61
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
66. 4a) “Advanced” search is a
meaningless term
= narrow search at
IRS.gov
= broaden search at
U Alaska-Fairbanks
!
62
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
67. 4b) Moving from advanced search to
search re nement
No more feature ghettoes: contextualize
search “help”
Analyze session data and queries originating
from search results pages to
Narrow results
How do users revise after large retrievals?
Build those functions into a "revise your search" UI
Broaden results
How do users revise after small/0 retrievals?
Build those functions into a "revise your search" UI
63
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
68. 4c) “Revise your Search” in action
64
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
69. 5) Taking advantage of unique query
types: proper nouns, unique IDs
65
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
70. 5a) Taking advantage of unique query
types: glossaries
66
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
71. 5b) Taking advantage of unique
query types: sorting & ltering
!
67
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
72. 6) Improving a
“no results found” page
Most 0 results pages are as unfriendly as an
unloved 404 error page
!
68
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
73. 6a) Improving a “no results found”
page
JellyBelly.com repeats popular searches;
much better than nothing
!
69
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
74. 7) Designing better
individual results
Help searchers choose most appropriate
documents by showing most appropriate
information per result
Dependent on a sense of common
information-seeking behaviors
Known-item: show author, title, date
Open-ended: show keywords, description
SSA can help determine common
information-seeking behaviors
70
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
75. 7a) Designing better
individual results
…more descriptive information for …less information for known-
open-ended searchers item searchers
71
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
76. 7b) Designing better
individual results
Financial Times found a high level of dates
within queries
Solution: support chronological sorting and
ltering
72
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
77. 8) Designing better result sets
Content typing can enable:
Filtering by exposing types
(HP)
Faceted classi cation/
navigation (WebMD)
73
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
78. 8a) Designing better result sets:
Curious stuff
#10 result is clicked through more often than #s 6, 7, 8, and 9
(ten results per page)
From SLI Systems (www.sli-systems.com) 74
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
79. 9) Best bets:
Going beyond meeting basic needs
Ensure useful results for top X (50? 100?)
most frequent queries
Usually determined manually (guided by
documented logic)
Technically straightforward; editorial and
political issues can be difficult
75
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
80. 9a) Best bets: NCI example
(clearly marked)
76
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
81. 9b) Best bets: HP example
(subtle; exposes content model)
77
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
82. 10) Choose functional requirements
for your (next) search engine
1. Basic con guration
New features (e.g., spell-checking)
Weighting (e.g., Financial Times and proper
names)
2. Tool selection/recon guration
Develop functional specs for new tool
Better yet, help tweak existing tool
Justify investment: Deloitte and Vanguard use SSA
to demonstrate that improvements that don’t
involve engine (e.g., Best Bets) are insufficient
78
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
83. 9 A case study from
Vanguard
79
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
84. For more on the
Vanguard case study…
Presentation, tools here: http://bit.ly/D3B8c
80
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
85. 10 Improving user research
and web analytics
81
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
86. Landscape of User Research Methods
(Christian Rohrer, xdstrategy.com)
82
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
87. The power of data:
Why UX needs SSA
1. Balance: UX methodologies/practitioners
are often quantitatively weak
2. Legitimacy: Data-driven analyses make an
impression on people hard to impress
3. Cost: The data is available; the analysis
scales well with available time
4. Fidelity: The data is behavioral, real, and
voluminous
5. Comprehensiveness: Quantitative methods
complement qualitative methods (vice versa)
83
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
88. Where and when does SSA t?
Or, the what and why of user research
Generic What questions lead to speci c What
questions
Use quantitative methods (e.g., SSA) to determine
the contextual diagnostics/KPIs
Example: Net ix
What questions lead to new Why questions
Use SSA to prioritize investments in qualitative
exploration
Example: clothing retailer following SSA with a
eld study
84
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
89. Comparing SSA with other
quantitative user research methods
Some data-driven methods
Clickstream/server log analysis: tells you where
users go rather than what users want
(complementary)
Event logging: comparable but often less data
and usually at a different point in customer/user
lifecycle
SEO/SEM: Different query types ( nding where
the answer is versus nding the answer); often
focused on commerce conversions
85
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
90. SSA and
qualitative user research methods
What questions lead to Why questions
Some examples
Personas: SSA supplements ctitious
construct with real data (i.e., likely queries a
persona might have)
Task analysis: SSA helps identify common
tasks to evaluate
Field studies: SSA suggests issues to monitor
in the eld
86
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
91. Augmenting personas and audience
pro les with frequent queries
Persona example (from
Adaptive Path)
Frequent queries added (in
green)
87
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
92. Using common queries to
enhance/fuel task analyses
Use for tasks that are more what than how
(i.e., testing information needs rather than
functions)
Examples
• Can you nd a map of
the campus?
• What study abroad
options are available to
MSU students?
• When is the last home
football game of the
season?
88
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
93. Identifying trends to monitor
during eld studies
Example: catalog clothing retailer
SSA showed
Preponderance of SKUs, even though they were
not displayed on the site
What were they doing there?
Subsequent eld study showed
Customers preferred print catalog to web catalog
to identify products
Customers preferred web catalog for ordering
Action: add SKUs to web catalog
89
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
95. Web Analytics also are
Top-Down and Bottom-Up
Top-down
• Do your Key Performance Indicators (KPI) address search
(and, more broadly, nding?)
• Example: NPS (Net Promoter Score) addresses whether or
not user would recommend company, but includes few
search-related metrics
Bottom-up
• Are you gathering search-related metrics?
• Example: % of unique queries with results
91
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
96. Mapping KPI and metrics:
A generic “search success” KPI
92
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
97. KPI built around goals and use cases
involving search (and browse)
From main page
• Find a product or service that I know to exist/
know how to describe.
• Find a product or service that I think exists, but
don’t know how to describe.
From a product/service page
• Find help using a product or service.
• Find a different/more appropriate product or
service
93
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
98. Search Metrics: general examples
(Lee Romero, blog.leeromero.org)
• Total searches for a given time period
• Total distinct search terms for a given time period
• Total distinct words for a given time period
• Average words per search
• Top searches for a given time period
• Top Searches over time
• Not found searches
• Error searches
• Ratio of searches performed each reporting period to the
number of visits for that same time period
94
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
99. Search Metrics: search engine tuning
(Jeannine Bartlett, earley.com)
Do users not nd what they want because the search engine
and its ranking and relevance algorithms have not been
adequately tuned?
Example Benchmarks and Metrics
• # of valid queries returning no results / total unique queries
• Relative % search results per data source
• Relative % click throughs per data source
• Pass/fail % for queries using stemming
• Pass/fail % for queries with misspellings
• Precision measures of “seed” documents sent through the tagging
process
95
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
100. Search Metrics: query entry
(Jeannine Bartlett, earley.com)
Do users not nd what they want because the UI for
expressing search terms is inadequate or unintuitive?
Example Benchmarks and Metrics
• % queries in the bottom set of the Zipf Curve ( at vs. hockey-stick
distribution)
• % queries with no click throughs
• % queries using syntactic metadata ltering (date, author, source,
document type, geography, etc.)
• % queries using Boolean search grammar
• % queries using type-ahead against taxonomy terms and synonyms
• % queries using faceted semantic re nement
• % pages from which search is available
96
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
101. Search Metrics: result sets
(Jeannine Bartlett, earley.com)
Do users not nd what they want because the UI for
visualizing result sets is inadequate or unintuitive?
Example Benchmarks and Metrics
• % queries utilizing multiple results views
• % queries with drill-down through clusters
• % queries using iterative syntactic metadata ltering (date range,
sorting, type or source inclusion/exclusion, etc.)
• % queries suggesting broader/narrower terms
• % queries suggesting “Best Bets” or “See Also”
• % queries using iterative semantic term ltering, inclusion or
exclusion
97
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
102. 11 “Advanced” topics/
discussion
98
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
104. Analyzing the long tail:
Why and how?
Why
Synonym mining and term disambiguation
(e.g., if “transfer” is a short head query, do
variants show up in long tail?)
The data is different (more esoteric queries);
therefore query categories are different
Richer source for pattern analysis
When you’ve already nailed the short head
How: regular random sampling
100
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
105. Long tail queries:
Longer, more complex (from Vanguard)
Short head: common queries Long tail: common queries
Bene ciary form 403(b)(7) account asset transfer
401(k) authorization
bene ciary automatic investing
career Wire transfer instructions
forms adoption agreement
amt international wire transfers
money market socially responsible investing
location Vanguard tax identi cation number
loans IRA Asset Transfer form
calculator fdic insured account
early withdrawal penalties
101
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
106. SSA and the organization:
Adoption by UX practitioners
Quantitative, data-driven approach that gets
UX “taken seriously”
Requires getting hands dirty with data,
negotiating with IT
Needs to scale up gradually (“SSA on a
budget”)
102
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
107. SSA and the organization:
Adoption by management
Appeal of numbers/metrics
Viral nature of analytics reports
Appeal of scalable approach and quick,
measurable wins
Tuning, rather than replacing, search engine
(and other enterprise–class applications)
103
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
108. SSA and the organization:
Marrying analytics and UX
1. Beyond diagnostics: SSA as a
predictive tool (example: Financial
Times)
2. The business of not converting:
moving beyond transactions toward
holistic user experience
104
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
109. 12 The technical stuff
105
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
110. Clicks and queries:
Often stored separately
106
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
111. Formats for query capture
Flat le search log: often produced by local
search engines; requires parsing
Local search database: queries automatically
stored in database; greater reporting exibility
Turnkey search appliance: local or remotely
hosted; often provide web based reporting
tools
Analytics application: integrated with
clickstream data
107
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
112. Flat le query storage
108
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
113. Flat le record example
(from Google Appliance)
123.45.67.89 - 25/Mar/2006 10:15:32 –
http://www.ipodstuff.com/search?q=nanotubes - Firefox
1.0.7; Windows NT 5.1 - 740674ce2123e969
• 123.45.67.89 is the Internet Protocol (IP) address for the computer that
originated the search. An IP address may be static – that is, it’s assigned
permanently (more or less) to one computer or it may be dynamic; i.e. the
Internet service provider assigns the address randomly from a pool of
addresses. In any event, this is the address of your customer’s computer.
• 25/Mar/2006 10:15:32 is the date and time of the query.
• http://www.ipodstuff.com/search?q=nanotubes is the request URL,
including the search query. In this example, the user typed “nanotubes” into
the search box.
• Firefox 1.0.7; Windows NT 5.1 is the Web browser and operating system of
the computer used by the customer entering the query
• 740674ce2123e969 is a “cookie.” A cookie is a unique le stored on a
computer accessing a particular function on the Web. Cookies can be used to
track user behavior across time, including search activity.
109
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
115. Local search database:
Bene ts over at le approach
Easier to store, especially over time
Already parsed
Better report generation abilities, including
ad hoc
Can be used in combination with best bets,
automated indices (great example at
msu.edu)
111
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
116. Turnkey search software appliance
Typically include local search database with
built-in reports
Provide minimal access to query data (some
built-in reports, few if any ad hoc reports)
SSA might mean breaking open the “black
box” (or undercutting vendors’ professional
service offerings)
Examples: Verity Ultraseek, Mondosoft,
Google Appliance
112
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
117. Analytics applications
Javascript inserted into pages to intercept
queries and other user activity
Analytics vendors typically not focused on site
SSA, so reports may require custom
development
Closest to delivering “Holy Grail” of integrated
data (search + browse)
Examples: Google Analytics, Omniture
113
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
119. How do you do it?
Pros and cons…
Capturing queries
Analyzing queries
115
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
120. What We’ve Covered
1. A Quick Demo
2. A Brief Introduction
3. Exercise: Pattern Analysis
4. Improving Metadata and Navigation
5. Exercise: Failure Analysis
6. Improving Content
7. Exercise: Session Analysis
8. Improving Search
9. Vanguard Case Study
10. Improving User Research and Web Analytics
11. “Advanced” Topics/Discussion
12. Technical Stuff
116
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
121. Discussion
Did I answer your burning question?
What three things will you try out at work
tomorrow?
117
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
122. Please Share Your SSA Knowledge:
Visit our book in progress site
Search Analytics for Your Site:
Conversations with your
Customers by Louis
Rosenfeld and
Marko Hurst
(Rosenfeld Media, 2011)
Site URL: www.rosenfeldmedia.com/books/searchanalytics/
Feed URL: feeds.rosenfeldmedia.com/searchanalytics/
118
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
123. Contact Information
Louis Rosenfeld
457 Third Street, #4R
Brooklyn, NY 11215 USA
lou@louisrosenfeld.com
www.louisrosenfeld.com
www.rosenfeldmedia.com
Twitter: @louisrosenfeld, @rosenfeldmedia
119
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Editor's Notes AND: About you? We get two major things out of this data: SESSIONS and FREQUENT QUERIES Start with AIGA via Google AnalyticsThen move to MSU via Excel spreadsheetThen show WW Norton session table But first, a little quiz: 1) How many know who their primary audiences are? 2) And, how many know what the main tasks and topics those audiences need to get from your site?If not, then… what’s the point? Deleting term may come back to bite you Augment with SSA Field study example: use clothing retailer that found SKUs in its logs when SKUs didn’t appear on its ecommerce site SEO benefits: Local queries might serve as predictors of future keywords NAMES OF EXAMPLES: ASK RW