Site Search Analytics Workshop Presentation

Site Search Analytics for a
Better User Experience

Louis Rosenfeld
lou@louisrosenfeld.com
Washington, DC • October 22, 2010

1
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.

About me

2

Agenda

1. A Quick Demo
2. A Brief Introduction
3. Exercise: Pattern Analysis
4. Improving Metadata and Navigation
5. Exercise: Failure Analysis
6. Improving Content
7. Exercise: Session Analysis
8. Improving Search
9. Vanguard Case Study
10. Improving User Research and Web Analytics
11. “Advanced” Topics/Discussion
12. Technical Stuﬀ

3

1 A Quick Demo

4

2 A Brief Introduction to
Site Search Analytics

5

Anatomy of a search log
(from Google Search Appliance)
Critical elements in bold: IP address, time/date stamp, query, and # of results:

XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?
access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL
%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystyleshe
et=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02
XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search?
%3Ad1&ie=UTF-8&client=www&q=license+plate
&ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XXX.XXX.
X.104 HTTP/1.1" 200 8283 146 0.16
XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search?
%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystyleshe
et=www&q=regional+transportation+governance
+commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17

6

The Zipf Curve:
Short head, middle torso, long tail

7

Zipf in text:
Diminishing returns

8

Querying the queries:
Generic questions to help get started
1. What are the most frequent unique queries?
2. Are frequent queries retrieving quality results?
3. Click-through rates per frequent query?
4. Most frequently clicked result per query?
5. Which frequent queries retrieve zero results?
6. What are the referrer pages for frequent queries?
7. Which queries retrieve popular documents?
8. What interesting patterns emerge in general?

9

Tuning your questions:
From generic to speci c
Net ix asks
1. Which movies most frequently searched?
2. Which of them most frequently clicked through?
3. Which of them least frequently added to queue?

10

The point of SSA:
Help determine, meet users’ desires

11

How do we know what users want?

Determine what they’re doing (behavior)
 Web analytics (SEO/SEM, clickstream, SSA)
 Help/reference desk/switchboard
 Eye tracking studies…
Determine why they’re doing it (intent)
 Usability testing and task analysis
 Field studies
 Focus groups
 Card sorting…
12

Squaring analytics and user research:
Top-Down and Bottom-Up
Top-Down Analysis
 Align metrics with organizational goals/KPI
(Key Performance Indicators)
 Benchmark, measure, monitor, optimize for the
world you know
Bottom-Up Analysis
 Look for trends, patterns, and outliers through
exploratory data analysis (“play”)
 Identify surprises: the world you don’t know
13

Bottom-Up analysis generates
patterns, diagnoses, improvements
Pro ling users’ needs: what more than why
Identifying and diagnosing problems with
 Content
 Navigation and metadata
 Search: interface design and con guration
Designing more eﬀectively through
 More eﬀective prioritization of your work
 Asking better, clearer questions
 Complementing and improving your user research
and evaluation
14

Top-Down Analysis starts with
your organization’s goals
Making money
• Commerce (e.g., Amazon)
• Generating leads (e.g., MetLife)
• Selling advertising (e.g., ESPN.com)
Saving money/time
• Support/self-service (e.g., IRS.gov)
KPI (e.g., “increase sales per visit”) are
composed of speci c metrics
15

3 Try and Discuss:
Pattern Analysis

16

4 Improving metadata
and navigation

17

1) Determining metadata values
through clustering
One week’s top 50
queries, clustered
by topic
Represents 20% of
all search activity
< 1 hour of work
Can help suggest
preferred terms

18

1a) Determining metadata values
through similar queries
• BBC explores similar
queries (i.e.,
collaborative
ltering)
• Helps with
misspellings,
synonyms, best bets

19

1b) Determining term granularity
through session analysis
Steps
1. Sample a variety of sessions
2. Compare “starting points” (initial queries) with
“end points”
3. Any patterns in terms of speci city?
Example
 “HP 1012 printer” -> “HP 1012 printer driver”

20

2) Testing and tuning
metadata values
Approaches
 Tracking appearances, disappearances, and
trends among terms
 Testing terms by querying them
 Deriving terms through “reverse lookup”

21

2a) Term trend tracking

Which term deserves your attention? Flash in
pan, or something important?
Next steps and implications: add new term,
or replace/delete existing term

22

2b) Term trend tracking
(sample report from ISYS)

23

2c) Testing your
metadata values: steps
1. Choose a manageable number of common
queries (e.g., top 25)
2. Check for matches among metadata values
(e.g., query “campus map” matches metadata
term “map”)
3. Queries without corresponding metadata
values may indicate gaps in your metadata
4. You might nd new metadata values among
your queries with 0 or few results

24

2d) Term derivation from
reverse lookup: steps
1. Identify “important” documents (e.g., popular,
or strategic, or new)
2. Generate list of queries that retrieve those
important documents
3. Determine if there are metadata terms that
correspond to those queries
4. If not, you’ve identi ed metadata gaps or
opportunities to tweak existing terms

25

3) Determining metadata attributes
(AKA types, elds)
Top queries
clustered and
prioritized by
attributes:
 Place
 Dept./
Program
 Service
 Task
1-2 hours work

26

3a) Good metadata attributes
improve navigation
Indirect improvements to any navigation system
driven by metadata:
 Site-wide taxonomies
 Local taxonomies
 Thesauri
Direct improvements to other navigation
systems
 Contextual navigation
 A-Z indices
 Main pages and other major pages
27

4) Identifying failures to
address with better navigation
Do this if you have combined query and clickstream
data
Steps
1. Determine top failed queries (“desire paths”) from
critical pages (e.g., product pages)
2. Determine to which documents users navigate
once they give up on search
3. Build those results into these queries’ result pages
OR extrapolate and present those documents’
categories (if available)

28

5) Better site-wide navigation
through A-Z indices: 3 approaches
1. Manual: grab top queries and insert in
index (embedding queries in links)
2. Automated: generate index of queries on
the y (embedding queries in links)
3. Hybrid: repurpose individual best bets
results as single alphabetized index (see
Michigan State University example)

29

5a) Hybrid A-Z index example:
Michigan State University

Frequent “recycled”
keywords best bets

30

6) Making the obvious obvious
on the main page
Making sure “maps” were on the MSU main page
(Though this didn’t entirely solve the problem)

31

5 Try and discuss:
Failure Analysis

32

Sample failure report
from Harvard Business
School intranet

33

6 Improving content

34

1) Identifying/addressing
content gaps
Using SSA to plug gaps 0 results report
(from behaviortracking.com)
 Failure analysis suggests
unmet content needs by
topic (e.g., frequent
queries for “iPod
wristband” and “iPod
holder for jogging”)
 Content typing suggests
what kind of content to
create (e.g., “product
comparison” and
“speci cations”) 35

1a) Identifying/addressing
content gaps
Simple...
1.Regularly review most frequent null result
queries
2.Try variants of each query to validate that the
documents are simply not there
3.If they’re not: create new content
Opportunity!
 Your alternative meds site gets queries for
“Nyquil”
 What do you do?
36

1b) Identifying/addressing
content gaps
Searching for “freedom of information” at Michigan State U...

...brings up a best bet
placeholder

37

2) Making relevant content even
more relevant
Three common problems
1. De cient titling
2. De cient or poorly applied metadata
3. Poor writing

38

2a) A simple case of using the
right copy
Intuit intranet
• Searches for “get back overpaid wages”
retrieved 0 results
• Term clearly should been included in the
appropriate Intuit page
• Easy x!

39

2b) Getting marketing to do
the right thing
Jargon vs. Plain Language at
Washtenaw Community College
• Online courses were marketed using College on
Demand (“COD”) and FlexEd
• Signup rates were lower than expected
• Compare with “online”
• SSA determined
which approach
was more eﬀective
40

2c) Getting content owners to do
the right thing
Sandia National Labs
• Regularly record which documents came up at
position #1 for 50 most frequent queries
• If and when that top document falls out of
position #1, document's owner is alerted
• Result: healthy dialogue (often about
following policies and procedures and their
value)

41

2d) Getting content owners to do the
right thing: wielding reports
Connecting pages (and their owners) that are
found through search...

42

2e) Getting content owners to do the
right thing: wielding reports
...with how those pages were found

43

3) Learning how users
understand your content
Learn more about speci c queries by the
terms they co-occur with (see also: Session
Analysis)
BBC: why did
queries for “Farepak”
suddenly spike?

44

3a) Learning how users
understand your content
Context revealed
• “Xmas hampers”
(what was stolen)
• “watch dog”
(consumer aﬀairs
program)
• “in adminstration”
(Farepak went
bust)

45

4) Learning from trends: seasonality

46

4a) Learning from trends:
Tuning content to meet demand
How might such trends impact your content?
Search engine? Navigation?

From behaviortracking.com
47

5) The “shape” of your information:
Determining content types
“What kind of page would users want when they searched this term?”
Top queries
clustered and
prioritized by
content types:
 Application
 News/
Announce-
ments
 Main page
 Contact info
 Instructions
1-2 hours work

48

5a) Using content types to expose,
improve contextual navigation

album pages artist descriptions
TV listings

album reviews discography artist bios

©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved. 94

5b) Using content types to improve
search results

50

7 Try and discuss:
Session Analysis

51

Session analysis from TFAnet intranet

These queries co-occur within sessions: why?
!

!

!

52

TFAnet session analysis results

• Searches for “delta ICEG” perform
poorly (way below the fold)
• Users than try an (incorrect)
alternative (“delta learning team”)
53

8 Improving search

54

Search systems: basic anatomy

…and know your own system; its
functionality will impact your query data
55

1) Plugging gaps in your
search engine’s index
Search engines often don’t know about all
content (especially in large, distributed
web environments)
Analyze queries with null results--which
actually have no content?
Do you see a pattern? Does it suggest a
single department/sub-site/content type?

56

2) Making “The Box” wide enough

Validate width of query entry eld
 Use query data to calculate comfortably “high
number” for query length in characters
 Does your entry eld accommodate that
length?

Amazon: 65 characters

Microsoft: 25 characters

57

2a) Making “The Box” wide enough

Top 500 queries from AIGA.org (37% of all
search activity) • Mean/median=@10
• Get rid of outliers
(e.g., 62, 36)
• Low 20s seems
“comfortable”
• Adjust for longer
long tail queries

!
!

58

3) Making query entry easier:
Type ahead/auto-complete
Populate query entry with common queries (Ask)
Better approach: reuse keywords from Best Bets
development (MSU)

59

3a) Making query entry easier:
Type ahead/auto-complete
Alternatives to using query data: existing metadata
(e.g., ESPN player names, Net ix movie titles)

60

4) Supporting search re nement:
What exactly is “advanced”?

61

4a) “Advanced” search is a
meaningless term
= narrow search at
IRS.gov
= broaden search at
U Alaska-Fairbanks

!

62

4b) Moving from advanced search to
search re nement
No more feature ghettoes: contextualize
search “help”
Analyze session data and queries originating
from search results pages to
 Narrow results
 How do users revise after large retrievals?
 Build those functions into a "revise your search" UI
 Broaden results
 How do users revise after small/0 retrievals?
 Build those functions into a "revise your search" UI
63

4c) “Revise your Search” in action

64

5) Taking advantage of unique query
types: proper nouns, unique IDs

65

5a) Taking advantage of unique query
types: glossaries

66

5b) Taking advantage of unique
query types: sorting & ltering

!

67

6) Improving a
“no results found” page
Most 0 results pages are as unfriendly as an
unloved 404 error page

!
68

6a) Improving a “no results found”
page
JellyBelly.com repeats popular searches;
much better than nothing

!
69

7) Designing better
individual results
Help searchers choose most appropriate
documents by showing most appropriate
information per result
Dependent on a sense of common
information-seeking behaviors
 Known-item: show author, title, date
 Open-ended: show keywords, description
SSA can help determine common
information-seeking behaviors
70

7a) Designing better
individual results
…more descriptive information for …less information for known-
open-ended searchers item searchers

71

7b) Designing better
individual results
Financial Times found a high level of dates
within queries
Solution: support chronological sorting and
ltering

72

8) Designing better result sets

Content typing can enable:
 Filtering by exposing types
(HP)
 Faceted classi cation/
navigation (WebMD)

73

8a) Designing better result sets:
Curious stuﬀ
#10 result is clicked through more often than #s 6, 7, 8, and 9
(ten results per page)

From SLI Systems (www.sli-systems.com) 74

9) Best bets:
Going beyond meeting basic needs
Ensure useful results for top X (50? 100?)
most frequent queries
Usually determined manually (guided by
documented logic)
Technically straightforward; editorial and
political issues can be diﬃcult

75

9a) Best bets: NCI example
(clearly marked)

76

9b) Best bets: HP example
(subtle; exposes content model)

77

10) Choose functional requirements
for your (next) search engine
1. Basic con guration
 New features (e.g., spell-checking)
 Weighting (e.g., Financial Times and proper
names)
2. Tool selection/recon guration
 Develop functional specs for new tool
 Better yet, help tweak existing tool
 Justify investment: Deloitte and Vanguard use SSA
to demonstrate that improvements that don’t
involve engine (e.g., Best Bets) are insuﬃcient

78

9 A case study from
Vanguard

79

For more on the
Vanguard case study…
Presentation, tools here: http://bit.ly/D3B8c

80

10 Improving user research
and web analytics

81

Landscape of User Research Methods
(Christian Rohrer, xdstrategy.com)

82

The power of data:
Why UX needs SSA
1. Balance: UX methodologies/practitioners
are often quantitatively weak
2. Legitimacy: Data-driven analyses make an
impression on people hard to impress
3. Cost: The data is available; the analysis
scales well with available time
4. Fidelity: The data is behavioral, real, and
voluminous
5. Comprehensiveness: Quantitative methods
complement qualitative methods (vice versa)

83

Where and when does SSA t?
Or, the what and why of user research
Generic What questions lead to speci c What
questions
 Use quantitative methods (e.g., SSA) to determine
the contextual diagnostics/KPIs
 Example: Net ix
What questions lead to new Why questions
 Use SSA to prioritize investments in qualitative
exploration
 Example: clothing retailer following SSA with a
eld study

84

Comparing SSA with other
quantitative user research methods
Some data-driven methods
 Clickstream/server log analysis: tells you where
users go rather than what users want
(complementary)
 Event logging: comparable but often less data
and usually at a diﬀerent point in customer/user
lifecycle
 SEO/SEM: Diﬀerent query types ( nding where
the answer is versus nding the answer); often
focused on commerce conversions

85

SSA and
qualitative user research methods
What questions lead to Why questions
Some examples
 Personas: SSA supplements ctitious
construct with real data (i.e., likely queries a
persona might have)
 Task analysis: SSA helps identify common
tasks to evaluate
 Field studies: SSA suggests issues to monitor
in the eld

86

Augmenting personas and audience
pro les with frequent queries

Persona example (from
Adaptive Path)

Frequent queries added (in
green)

87

Using common queries to
enhance/fuel task analyses
Use for tasks that are more what than how
(i.e., testing information needs rather than
functions)

Examples
• Can you nd a map of
the campus?
• What study abroad
options are available to
MSU students?
• When is the last home
football game of the
season?

88

Identifying trends to monitor
during eld studies
Example: catalog clothing retailer
SSA showed
 Preponderance of SKUs, even though they were
not displayed on the site
 What were they doing there?
Subsequent eld study showed
 Customers preferred print catalog to web catalog
to identify products
 Customers preferred web catalog for ordering
Action: add SKUs to web catalog
89

Combining quantitative
and qualitative
Exploring the needs of a particular audience:
users of BBC
children’s
content

90

Web Analytics also are
Top-Down and Bottom-Up
Top-down
• Do your Key Performance Indicators (KPI) address search
(and, more broadly, nding?)
• Example: NPS (Net Promoter Score) addresses whether or
not user would recommend company, but includes few
search-related metrics
Bottom-up
• Are you gathering search-related metrics?
• Example: % of unique queries with results

91

Mapping KPI and metrics:
A generic “search success” KPI

92

KPI built around goals and use cases
involving search (and browse)
From main page
• Find a product or service that I know to exist/
know how to describe.
• Find a product or service that I think exists, but
don’t know how to describe.
From a product/service page
• Find help using a product or service.
• Find a diﬀerent/more appropriate product or
service

93

Search Metrics: general examples
(Lee Romero, blog.leeromero.org)
• Total searches for a given time period
• Total distinct search terms for a given time period
• Total distinct words for a given time period
• Average words per search
• Top searches for a given time period
• Top Searches over time
• Not found searches
• Error searches
• Ratio of searches performed each reporting period to the
number of visits for that same time period

94

Search Metrics: search engine tuning
(Jeannine Bartlett, earley.com)
Do users not nd what they want because the search engine
and its ranking and relevance algorithms have not been
adequately tuned?
Example Benchmarks and Metrics
• # of valid queries returning no results / total unique queries
• Relative % search results per data source
• Relative % click throughs per data source
• Pass/fail % for queries using stemming
• Pass/fail % for queries with misspellings
• Precision measures of “seed” documents sent through the tagging
process

95

Search Metrics: query entry
Do users not nd what they want because the UI for
expressing search terms is inadequate or unintuitive?
• % queries in the bottom set of the Zipf Curve ( at vs. hockey-stick
distribution)
• % queries with no click throughs
• % queries using syntactic metadata ltering (date, author, source,
document type, geography, etc.)
• % queries using Boolean search grammar
• % queries using type-ahead against taxonomy terms and synonyms
• % queries using faceted semantic re nement
• % pages from which search is available

96

Search Metrics: result sets
Do users not nd what they want because the UI for
visualizing result sets is inadequate or unintuitive?
• % queries utilizing multiple results views
• % queries with drill-down through clusters
• % queries using iterative syntactic metadata ltering (date range,
sorting, type or source inclusion/exclusion, etc.)
• % queries suggesting broader/narrower terms
• % queries suggesting “Best Bets” or “See Also”
• % queries using iterative semantic term ltering, inclusion or
exclusion

97

11 “Advanced” topics/
discussion

98

Comparing referral queries
with local queries

99

Analyzing the long tail:
Why and how?
Why
 Synonym mining and term disambiguation
(e.g., if “transfer” is a short head query, do
variants show up in long tail?)
 The data is diﬀerent (more esoteric queries);
therefore query categories are diﬀerent
 Richer source for pattern analysis
 When you’ve already nailed the short head
How: regular random sampling
100

Long tail queries:
Longer, more complex (from Vanguard)

Short head: common queries Long tail: common queries
Bene ciary form 403(b)(7) account asset transfer
401(k) authorization
bene ciary automatic investing
career Wire transfer instructions
forms adoption agreement
amt international wire transfers
money market socially responsible investing
location Vanguard tax identi cation number
loans IRA Asset Transfer form
calculator fdic insured account
early withdrawal penalties

101

SSA and the organization:
Adoption by UX practitioners
Quantitative, data-driven approach that gets
UX “taken seriously”
Requires getting hands dirty with data,
negotiating with IT
Needs to scale up gradually (“SSA on a
budget”)

102

Adoption by management
Appeal of numbers/metrics
Viral nature of analytics reports
Appeal of scalable approach and quick,
measurable wins
Tuning, rather than replacing, search engine
(and other enterprise–class applications)

103

Marrying analytics and UX
1. Beyond diagnostics: SSA as a
predictive tool (example: Financial
Times)
2. The business of not converting:
moving beyond transactions toward
holistic user experience

104

12 The technical stuﬀ

105

Clicks and queries:
Often stored separately

106

Formats for query capture

Flat le search log: often produced by local
search engines; requires parsing
Local search database: queries automatically
stored in database; greater reporting exibility
Turnkey search appliance: local or remotely
hosted; often provide web based reporting
tools
Analytics application: integrated with
clickstream data

107

Flat le query storage

108

Flat le record example
(from Google Appliance)

123.45.67.89 - 25/Mar/2006 10:15:32 –
http://www.ipodstuff.com/search?q=nanotubes - Firefox
1.0.7; Windows NT 5.1 - 740674ce2123e969

• 123.45.67.89 is the Internet Protocol (IP) address for the computer that
originated the search. An IP address may be static – that is, it’s assigned
permanently (more or less) to one computer or it may be dynamic; i.e. the
Internet service provider assigns the address randomly from a pool of
addresses. In any event, this is the address of your customer’s computer.
• 25/Mar/2006 10:15:32 is the date and time of the query.
• http://www.ipodstuﬀ.com/search?q=nanotubes is the request URL,
including the search query. In this example, the user typed “nanotubes” into
the search box.
• Firefox 1.0.7; Windows NT 5.1 is the Web browser and operating system of
the computer used by the customer entering the query
• 740674ce2123e969 is a “cookie.” A cookie is a unique le stored on a
computer accessing a particular function on the Web. Cookies can be used to
track user behavior across time, including search activity.

109

Local search database

110

Local search database:
Bene ts over at le approach
Easier to store, especially over time
Already parsed
Better report generation abilities, including
ad hoc
Can be used in combination with best bets,
automated indices (great example at
msu.edu)

111

Turnkey search software appliance

Typically include local search database with
built-in reports
Provide minimal access to query data (some
built-in reports, few if any ad hoc reports)
SSA might mean breaking open the “black
box” (or undercutting vendors’ professional
service oﬀerings)
Examples: Verity Ultraseek, Mondosoft,
Google Appliance

112

Analytics applications

Javascript inserted into pages to intercept
queries and other user activity

Analytics vendors typically not focused on site
SSA, so reports may require custom
development
Closest to delivering “Holy Grail” of integrated
data (search + browse)
Examples: Google Analytics, Omniture
113

Analytics applications:
Sample report from Google Analytics

114

How do you do it?
Pros and cons…
Capturing queries

Analyzing queries

115

What We’ve Covered

1. A Quick Demo
2. A Brief Introduction
3. Exercise: Pattern Analysis
4. Improving Metadata and Navigation
5. Exercise: Failure Analysis
6. Improving Content
7. Exercise: Session Analysis
8. Improving Search
9. Vanguard Case Study
10. Improving User Research and Web Analytics
11. “Advanced” Topics/Discussion
12. Technical Stuﬀ

116

Discussion

Did I answer your burning question?

What three things will you try out at work
tomorrow?

117

Please Share Your SSA Knowledge:
Visit our book in progress site
Search Analytics for Your Site:
Conversations with your
Customers by Louis
Rosenfeld and
Marko Hurst
(Rosenfeld Media, 2011)

Site URL: www.rosenfeldmedia.com/books/searchanalytics/
Feed URL: feeds.rosenfeldmedia.com/searchanalytics/

118

Contact Information

Louis Rosenfeld
457 Third Street, #4R
Brooklyn, NY 11215 USA

lou@louisrosenfeld.com

www.louisrosenfeld.com
www.rosenfeldmedia.com

Twitter: @louisrosenfeld, @rosenfeldmedia
119

Site Search Analytics Workshop Presentation

Recommended

Recommended

More Related Content

What's hot

What's hot (8)

Similar to Site Search Analytics Workshop Presentation

Similar to Site Search Analytics Workshop Presentation (20)

More from Louis Rosenfeld

More from Louis Rosenfeld (18)

Recently uploaded

Recently uploaded (20)

Site Search Analytics Workshop Presentation

Editor's Notes