Workshop:
Search Analytics
forYour Site
Louis Rosenfeld
lou@louisrosenfeld.com • @louisrosenfeld
Webdagene • 9 September 2...
Hello, my name is Lou
www.louisrosenfeld.com | www.rosenfeldmedia.com
Agenda
1.The basics of Site Search Analytics (SSA)
2.Exercise 1 (pattern analysis)
3.Things you can do with SSA
4.Exercise...
Let’s look at the data
No, let’s look at the real data
Critical elements in bold: IP address, time/date stamp, query, and # of
results:
XXX.XXX.X...
No, let’s look at the real data
Critical elements in bold: IP address, time/date stamp, query, and # of
results:
XXX.XXX.X...
No, let’s look at the real data
Critical elements in bold: IP address, time/date stamp, query, and # of
results:
XXX.XXX.X...
SSA is semantically rich data, and...
SSA is semantically rich data, and...
Queries
sorted by
frequency
...what users want--in their own words
A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents
meet the needs of your most import...
A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents
meet the needs of your most import...
A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents
meet the needs of your most import...
A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents
meet the needs of your most import...
A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents
meet the needs of your most import...
A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents
meet the needs of your most import...
(and the tail is quite long)
(and the tail is quite long)
(and the tail is quite long)
(and the tail is quite long)
(and the tail is quite long)
(and the tail is quite long)
The Long Tail is
much longer than
you’d suspect
The Zipf Distribution, textually
Insert Long Tail here
Agenda
1.The basics of Site Search Analytics (SSA)
2.Exercise 1 (pattern analysis)
3.Things you can do with SSA
4.Exercise...
Exercise 1 (pattern analysis)
Work in pairs
• Each pair should have a laptop with
Microsoft Excel
• Laptop platform (Mac, ...
Agenda
1.The basics of Site Search Analytics (SSA)
2.Exercise 1 (pattern analysis)
3.Things you can do with SSA
4.Exercise...
Tune site-wide navigation
Nailing the basics
in top-down navigation
Nailing the basics
in top-down navigation
Tune contextual navigation
Start with basic SSA data:
queries and query frequency
Percent: volume
of search activity
for a unique
query during a
part...
Tease out common content types
Tease out common content types
Tease out common content types
Took an hour to...
• Analyze top 50 queries (20% of all search activity)
• Ask and iterate:...
Clear content types lead to
better contextual navigation
artist descriptions
album reviews
album pages
artist biosdiscogra...
Make search smarter
Clear content types improve
search performance
Clear content types improve
search performance
Clear content types improve
search performance
Content objects
related to products
Clear content types improve
search performance
Content objects
related to products
Raw search results
Enabling filtering/faceted search
Contextualizing “advanced” features
Session data suggest
progression and context
Session data suggest
progression and context
search session patterns
1. solar energy
2. how solar energy works
Session data suggest
progression and context
search session patterns
1. solar energy
2. how solar energy works
search sess...
Session data suggest
progression and context
search session patterns
1. solar energy
2. how solar energy works
search sess...
Session data suggest
progression and context
search session patterns
1. solar energy
2. how solar energy works
search sess...
Session data suggest
progression and context
search session patterns
1. solar energy
2. how solar energy works
search sess...
Recognizing
proper nouns,
dates, and
unique ID#s
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Identifying a need for a glossary
27
Smarter best bets
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
29
Best bets without guessing
Frequent
keywords
“recycled”
best bets
Learn how audiences differ
Who cares about what? (AIGA.org)
Who cares about what? (AIGA.org)
Who cares about what? (Open U)
Who cares about what? (Open U)
Who cares about what? (Open U)
Who cares about what? (Open U)
Why analyze queries by audience?
Fortify your personas with data
Learn about differences between audiences
• Open Universi...
Reduce jargon
Save the brand by killing jargon
Jargon related to online education: FlexEd, COD,
College on Demand
Marketing’s solution: ...
Agenda
1.The basics of Site Search Analytics (SSA)
2.Exercise 1 (pattern analysis)
3.Things you can do with SSA
4.Exercise...
Exercise 2 (longitudinal analysis)
Work in pairs
• Each pair should have a laptop with
Microsoft Excel
• Laptop platform (...
Agenda
1.The basics of Site Search Analytics (SSA)
2.Exercise 1 (pattern analysis)
3.Things you can do with SSA
4.Exercise...
Know when to publish what
Interest in the
football team:
going...
Interest in the
football team:
going...
...going...
Interest in the
football team:
going...
...going...
gone
Interest in the
football team:
going...
...going...
gone
Time to
study!
Before
Tax Day
After
Tax Day
Identify trends
Learn from failure
Failed navigation?
Examining unexpected searching
Look for places
searches happen
beyond main page
What’s going on?
• Navi...
Where navigation is failing
(“Professional Resources” page)
Do users and
AIGA mean
different
things by
“Professional
Resou...
Comparing what users find
and what they want
Comparing what users find
and what they want
Failed business goals?
Developing custom metrics
Netflix asks
1. Which movies most frequently searched? (query count)
2. Wh...
Failed business goals?
Developing custom metrics
Netflix asks
1. Which movies most frequently searched? (query count)
2. Wh...
Failed business goals?
Developing custom metrics
Netflix asks
1. Which movies most frequently searched? (query count)
2. Wh...
Learn from search sessions
Sample search session
(Teach for America intranet)
Session analysis
These queries co-occur within sessions: why?






TFAnet session analysis results
• Searches for “delta ICEG” perform
poorly (way below the fold)
• Users then try an (incor...
Identify content gaps
0 results report
(from behaviortracking.com)
Are we missing
something?
Are we missing a
type of something?
Identifying gaps
helps force an issue
Identify failed content






1.Choose a
content type (e.g.,
events)
2.Ask:“Where
should users go
from here?”
3.Analyze the
frequent queries
from ...












Analyze frequent queries generated from each content sample
Make content owners into
stakeholders
Sandia National Labs
• Regularly record which documents came up
at position #1 for 50 most frequent queries
• If and when ...
Connecting pages (and their owners)
that are found through search...
...with how those pages were found
Predict the future
Shaping the
FinancialTimes’ editorial agenda
FT compares these
• Spiking queries
for proper nouns
(i.e., people and
compan...
Agenda
1.The basics of Site Search Analytics (SSA)
2.Exercise 1 (pattern analysis)
3.Things you can do with SSA
4.Exercise...
Avoiding a disaster atVanguard
Vanguard used SSA to help benchmark
existing search engine’s performance and
help select ne...
What to do?
Test performance of most frequent queries
Measure using original two sets of metrics
1.relevance: how reliably...
Relevance: 5 metrics
(queries tested have “best” result)
Mean: Average distance from the top
Median: Less sensitive to out...
Relevance: 5 metrics
(queries tested have “best” result)
Mean: Average distance from the top
Median: Less sensitive to out...
Relevance: 5 metrics
(queries tested have “best” result)
Mean: Average distance from the top
Median: Less sensitive to out...
Relevance: 5 metrics
(queries tested have “best” result)
Mean: Average distance from the top
Median: Less sensitive to out...
Precision:
rating scale
Evaluate frequent queries’ top search results on this scale
• r / Relevant: Based on the informati...
Precision:
three metrics
Metrics based on degrees of permissiveness
1. strict: only counts completely relevant results
2. ...
Putting it all together:
old engine (target) and new
Note: low relevance and high precision scores are optimal
More on Van...
Agenda
1.The basics of Site Search Analytics (SSA)
2.Exercise 1 (pattern analysis)
3.Things you can do with SSA
4.Exercise...
Mapping KPI and metrics:
A generic “search success” KPI
Search Metrics: general examples
(Lee Romero, blog.leeromero.org)
• Total searches for a given time period
• Total distinc...
Search Metrics: search engine tuning
(Jeannine Bartlett, earley.com)
Do users not find what they want because the search en...
Search Metrics: query entry
(Jeannine Bartlett, earley.com)
Do users not find what they want because the UI for
expressing ...
Search Metrics: result sets
(Jeannine Bartlett, earley.com)
Do users not find what they want because the UI for
visualizing...
Agenda
1.The basics of Site Search Analytics (SSA)
2.Exercise 1 (pattern analysis)
3.Things you can do with SSA
4.Exercise...
Things to do today
1.Set up SSA in Google Analytics
2.Query your queries
3.Start developing a site report card
4.Start inc...
Turn on SSA in Google Analytics
Set up GA for your site if you haven’t already
Then teach it to parse and capture your
sea...
Seed your analysis by
querying your queries
Starter questions
1. What are the most frequent unique queries?
2. Are frequen...
Use SSA to start work
on a site report card
Use SSA to start work
on a site report card
SSA helps
determine common
information needs
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
From Christian Rohrer, xdstrategy.com
©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Augment personas and audience
profiles with frequ...
Agenda
1.The basics of Site Search Analytics (SSA)
2.Exercise 1 (pattern analysis)
3.Things you can do with SSA
4.Exercise...
Comparing referral queries
with local queries
Long tail queries:
Longer, more complex (fromVanguard)
Short head: common queries Long tail: common queries
Beneficiary for...
Now on sale
	

 Search Analytics forYour Site:
Conversations with
Your Customers
by Louis Rosenfeld
(Rosenfeld Media, 2011...
Louis Rosenfeld
lou@louisrosenfeld.com
www.louisrosenfeld.com
www.rosenfeldmedia.com
@louisrosenfeld
@rosenfeldmedia
This ...
Site search analytics workshop presentation
Site search analytics workshop presentation
Site search analytics workshop presentation
Site search analytics workshop presentation
Upcoming SlideShare
Loading in …5
×

Site search analytics workshop presentation

8,070
-1

Published on

Workshop presented at Webdagene 2013 (http://webdagene.no/en/) September 9, 2013; UX Lisbon (http://www.ux-lx.com), May 12, 2011; UX Hong Kong (http://www.uxhongkong.com/), February 17, 2011.

Published in: Design, Technology, Business
1 Comment
28 Likes
Statistics
Notes
No Downloads
Views
Total Views
8,070
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
350
Comments
1
Likes
28
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • We get two major things out of this data: SESSIONS and FREQUENT QUERIES\n
  • Your brain on data: what will it do?\n
  • Your brain on data: what will it do?\n
  • \n
  • \n
  • Amazing drawing by Eva-Lotta Lamm: www.evalotta.net\n
  • Amazing drawing by Eva-Lotta Lamm: www.evalotta.net\n
  • Amazing drawing by Eva-Lotta Lamm: www.evalotta.net\n
  • Amazing drawing by Eva-Lotta Lamm: www.evalotta.net\n
  • Amazing drawing by Eva-Lotta Lamm: www.evalotta.net\n
  • Amazing drawing by Eva-Lotta Lamm: www.evalotta.net\n
  • Amazing drawing by Eva-Lotta Lamm: www.evalotta.net\n
  • Amazing drawing by Eva-Lotta Lamm: www.evalotta.net\n
  • Amazing drawing by Eva-Lotta Lamm: www.evalotta.net\n
  • Amazing drawing by Eva-Lotta Lamm: www.evalotta.net\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Personas: http://www.uie.com/images/blog/YahooExamplePersona.gif\nTable: From Jarrett, Quesenbery, Stirling, and Allen’s report “Search Behaviour at OU;” April 6, 2007.\n
  • Personas: http://www.uie.com/images/blog/YahooExamplePersona.gif\nTable: From Jarrett, Quesenbery, Stirling, and Allen’s report “Search Behaviour at OU;” April 6, 2007.\n
  • Personas: http://www.uie.com/images/blog/YahooExamplePersona.gif\nTable: From Jarrett, Quesenbery, Stirling, and Allen’s report “Search Behaviour at OU;” April 6, 2007.\n
  • Personas: http://www.uie.com/images/blog/YahooExamplePersona.gif\nTable: From Jarrett, Quesenbery, Stirling, and Allen’s report “Search Behaviour at OU;” April 6, 2007.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Examples\n “OO7” versus “007”\n Porn-related (not carried by Netflix)\n “yoga”: not stocking enough? Or not indexing enough record content? Some other problem?\n
  • Examples\n “OO7” versus “007”\n Porn-related (not carried by Netflix)\n “yoga”: not stocking enough? Or not indexing enough record content? Some other problem?\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • More great illustrations by Eva-Lotta Lamm\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Site search analytics workshop presentation

    1. 1. Workshop: Search Analytics forYour Site Louis Rosenfeld lou@louisrosenfeld.com • @louisrosenfeld Webdagene • 9 September 2013
    2. 2. Hello, my name is Lou www.louisrosenfeld.com | www.rosenfeldmedia.com
    3. 3. Agenda 1.The basics of Site Search Analytics (SSA) 2.Exercise 1 (pattern analysis) 3.Things you can do with SSA 4.Exercise 2 (longitudinal analysis 5.More things you can do with SSA 6.A case study 7.More on metrics 8.Things you can do today 9.Discussion
    4. 4. Let’s look at the data
    5. 5. No, let’s look at the real data Critical elements in bold: IP address, time/date stamp, query, and # of results: XXX.XXX.X.104 - - [10/Jul/2011:10:25:46 -0800] "GET /search?access=p&entqr=0 &output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8 &client=www&oe=UTF-8&proxystylesheet=www& q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2011:10:25:48 -0800] "GET /searchaccess=p&entqr=0 &output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ie=UTF-8&client=www& q=license+plate&ud=1&site=AllSites &spell=1&oe=UTF-8&proxystylesheet=www& ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16
    6. 6. No, let’s look at the real data Critical elements in bold: IP address, time/date stamp, query, and # of results: XXX.XXX.X.104 - - [10/Jul/2011:10:25:46 -0800] "GET /search?access=p&entqr=0 &output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8 &client=www&oe=UTF-8&proxystylesheet=www& q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2011:10:25:48 -0800] "GET /searchaccess=p&entqr=0 &output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ie=UTF-8&client=www& q=license+plate&ud=1&site=AllSites &spell=1&oe=UTF-8&proxystylesheet=www& ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16 What are users searching?
    7. 7. No, let’s look at the real data Critical elements in bold: IP address, time/date stamp, query, and # of results: XXX.XXX.X.104 - - [10/Jul/2011:10:25:46 -0800] "GET /search?access=p&entqr=0 &output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8 &client=www&oe=UTF-8&proxystylesheet=www& q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2011:10:25:48 -0800] "GET /searchaccess=p&entqr=0 &output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ie=UTF-8&client=www& q=license+plate&ud=1&site=AllSites &spell=1&oe=UTF-8&proxystylesheet=www& ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16 What are users searching? How often are users failing?
    8. 8. SSA is semantically rich data, and...
    9. 9. SSA is semantically rich data, and... Queries sorted by frequency
    10. 10. ...what users want--in their own words
    11. 11. A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents meet the needs of your most important audiences
    12. 12. A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents meet the needs of your most important audiences Not all queries are distributed equally
    13. 13. A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents meet the needs of your most important audiences
    14. 14. A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents meet the needs of your most important audiences Nor do they diminish gradually
    15. 15. A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents meet the needs of your most important audiences
    16. 16. A little goes a long wayA handful of queries/tasks/ways to navigate/features/ documents meet the needs of your most important audiences 80/20 rule isn’t quite accurate
    17. 17. (and the tail is quite long)
    18. 18. (and the tail is quite long)
    19. 19. (and the tail is quite long)
    20. 20. (and the tail is quite long)
    21. 21. (and the tail is quite long)
    22. 22. (and the tail is quite long) The Long Tail is much longer than you’d suspect
    23. 23. The Zipf Distribution, textually
    24. 24. Insert Long Tail here
    25. 25. Agenda 1.The basics of Site Search Analytics (SSA) 2.Exercise 1 (pattern analysis) 3.Things you can do with SSA 4.Exercise 2 (longitudinal analysis 5.More things you can do with SSA 6.A case study 7.More on metrics 8.Things you can do today 9.Discussion
    26. 26. Exercise 1 (pattern analysis) Work in pairs • Each pair should have a laptop with Microsoft Excel • Laptop platform (Mac, PC) doesn’t matter Download data files: 2005-October.xls Refer to exercise sheet No right answers Have fun!
    27. 27. Agenda 1.The basics of Site Search Analytics (SSA) 2.Exercise 1 (pattern analysis) 3.Things you can do with SSA 4.Exercise 2 (longitudinal analysis 5.More things you can do with SSA 6.A case study 7.More on metrics 8.Things you can do today 9.Discussion
    28. 28. Tune site-wide navigation
    29. 29. Nailing the basics in top-down navigation
    30. 30. Nailing the basics in top-down navigation
    31. 31. Tune contextual navigation
    32. 32. Start with basic SSA data: queries and query frequency Percent: volume of search activity for a unique query during a particular time period Cumulative Percent: running sum of percentages
    33. 33. Tease out common content types
    34. 34. Tease out common content types
    35. 35. Tease out common content types Took an hour to... • Analyze top 50 queries (20% of all search activity) • Ask and iterate: “what kind of content would users be looking for when they searched these terms?” • Add cumulative percentages Result: prioritized list of potential content types #1) application: 11.77% #2) reference: 10.5% #3) instructions: 8.6% #4) main/navigation pages: 5.91% #5) contact info: 5.79% #6) news/announcements: 4.27%
    36. 36. Clear content types lead to better contextual navigation artist descriptions album reviews album pages artist biosdiscography TV listings
    37. 37. Make search smarter
    38. 38. Clear content types improve search performance
    39. 39. Clear content types improve search performance
    40. 40. Clear content types improve search performance Content objects related to products
    41. 41. Clear content types improve search performance Content objects related to products Raw search results
    42. 42. Enabling filtering/faceted search
    43. 43. Contextualizing “advanced” features
    44. 44. Session data suggest progression and context
    45. 45. Session data suggest progression and context search session patterns 1. solar energy 2. how solar energy works
    46. 46. Session data suggest progression and context search session patterns 1. solar energy 2. how solar energy works search session patterns 1. solar energy 2. energy
    47. 47. Session data suggest progression and context search session patterns 1. solar energy 2. how solar energy works search session patterns 1. solar energy 2. energy search session patterns 1. solar energy 2. solar energy charts
    48. 48. Session data suggest progression and context search session patterns 1. solar energy 2. how solar energy works search session patterns 1. solar energy 2. energy search session patterns 1. solar energy 2. solar energy charts search session patterns 1. solar energy 2. explain solar energy
    49. 49. Session data suggest progression and context search session patterns 1. solar energy 2. how solar energy works search session patterns 1. solar energy 2. energy search session patterns 1. solar energy 2. solar energy charts search session patterns 1. solar energy 2. explain solar energy search session patterns 1. solar energy 2. solar energy news
    50. 50. Recognizing proper nouns, dates, and unique ID#s
    51. 51. ©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved. Identifying a need for a glossary 27
    52. 52. Smarter best bets
    53. 53. ©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved. 29 Best bets without guessing
    54. 54. Frequent keywords “recycled” best bets
    55. 55. Learn how audiences differ
    56. 56. Who cares about what? (AIGA.org)
    57. 57. Who cares about what? (AIGA.org)
    58. 58. Who cares about what? (Open U)
    59. 59. Who cares about what? (Open U)
    60. 60. Who cares about what? (Open U)
    61. 61. Who cares about what? (Open U)
    62. 62. Why analyze queries by audience? Fortify your personas with data Learn about differences between audiences • Open University “Enquirers”: 16 of 25 queries are for subjects not taught at OU • Open University Students: search for course codes, topics dealing with completing program Determine what’s commonly important to all audiences (these queries better work well)
    63. 63. Reduce jargon
    64. 64. Save the brand by killing jargon Jargon related to online education: FlexEd, COD, College on Demand Marketing’s solution: expensive campaign to educate public (via posters, brochures) Result: content relabeled, money saved query rank query #22 online* #101 COD #259 College on Demand #389 FlexTrack *“online”part of 213 queries
    65. 65. Agenda 1.The basics of Site Search Analytics (SSA) 2.Exercise 1 (pattern analysis) 3.Things you can do with SSA 4.Exercise 2 (longitudinal analysis 5.More things you can do with SSA 6.A case study 7.More on metrics 8.Things you can do today 9.Discussion
    66. 66. Exercise 2 (longitudinal analysis) Work in pairs • Each pair should have a laptop with Microsoft Excel • Laptop platform (Mac, PC) doesn’t matter Download data files: 2006-February.xls + 2006-June.xls Refer to exercise sheet No right answers Have fun!
    67. 67. Agenda 1.The basics of Site Search Analytics (SSA) 2.Exercise 1 (pattern analysis) 3.Things you can do with SSA 4.Exercise 2 (longitudinal analysis 5.More things you can do with SSA 6.A case study 7.More on metrics 8.Things you can do today 9.Discussion
    68. 68. Know when to publish what
    69. 69. Interest in the football team: going...
    70. 70. Interest in the football team: going... ...going...
    71. 71. Interest in the football team: going... ...going... gone
    72. 72. Interest in the football team: going... ...going... gone Time to study!
    73. 73. Before Tax Day
    74. 74. After Tax Day
    75. 75. Identify trends
    76. 76. Learn from failure
    77. 77. Failed navigation? Examining unexpected searching Look for places searches happen beyond main page What’s going on? • Navigational failure? • Content failure? • Something else?
    78. 78. Where navigation is failing (“Professional Resources” page) Do users and AIGA mean different things by “Professional Resources”?
    79. 79. Comparing what users find and what they want
    80. 80. Comparing what users find and what they want
    81. 81. Failed business goals? Developing custom metrics Netflix asks 1. Which movies most frequently searched? (query count) 2. Which of them most frequently clicked through? (MDP views) 3. Which of them least frequently added to queue? (queue adds)
    82. 82. Failed business goals? Developing custom metrics Netflix asks 1. Which movies most frequently searched? (query count) 2. Which of them most frequently clicked through? (MDP views) 3. Which of them least frequently added to queue? (queue adds)
    83. 83. Failed business goals? Developing custom metrics Netflix asks 1. Which movies most frequently searched? (query count) 2. Which of them most frequently clicked through? (MDP views) 3. Which of them least frequently added to queue? (queue adds)
    84. 84. Learn from search sessions
    85. 85. Sample search session (Teach for America intranet)
    86. 86. Session analysis These queries co-occur within sessions: why? 
 
 

    87. 87. TFAnet session analysis results • Searches for “delta ICEG” perform poorly (way below the fold) • Users then try an (incorrect) alternative (“delta learning team”) 54
    88. 88. Identify content gaps
    89. 89. 0 results report (from behaviortracking.com) Are we missing something? Are we missing a type of something?
    90. 90. Identifying gaps helps force an issue
    91. 91. Identify failed content
    92. 92. 
 
 
 1.Choose a content type (e.g., events) 2.Ask:“Where should users go from here?” 3.Analyze the frequent queries from this content type from aiga.org
    93. 93. 
 
 
 
 
 
 Analyze frequent queries generated from each content sample
    94. 94. Make content owners into stakeholders
    95. 95. Sandia National Labs • Regularly record which documents came up at position #1 for 50 most frequent queries • If and when that top document falls out of position #1, document's owner is alerted • Result: healthy dialogue (often about following policies and procedures and their value)
    96. 96. Connecting pages (and their owners) that are found through search...
    97. 97. ...with how those pages were found
    98. 98. Predict the future
    99. 99. Shaping the FinancialTimes’ editorial agenda FT compares these • Spiking queries for proper nouns (i.e., people and companies) • Recent editorial coverage of people and companies Discrepancy? • Breaking story?! • Let the editors know!
    100. 100. Agenda 1.The basics of Site Search Analytics (SSA) 2.Exercise 1 (pattern analysis) 3.Things you can do with SSA 4.Exercise 2 (longitudinal analysis 5.More things you can do with SSA 6.A case study 7.More on metrics 8.Things you can do today 9.Discussion
    101. 101. Avoiding a disaster atVanguard Vanguard used SSA to help benchmark existing search engine’s performance and help select new engine New search engine “performed” poorly But IT needed convincing to delay launch Information Architect & Dev Team Meeting Search seems to have a few problems… Nah . Where’s the proof? You can’t tell for sure.
    102. 102. What to do? Test performance of most frequent queries Measure using original two sets of metrics 1.relevance: how reliably the search engine returns the best matches first 2.precision: proportion of relevant and irrelevant results clustered at the top of the list
    103. 103. Relevance: 5 metrics (queries tested have “best” result) Mean: Average distance from the top Median: Less sensitive to outliers, but not useful once at least half are ranked #1 Count - Below 1st: How often is the best target something other than 1st? Count – Below 5th: How often is the best target outside the critical area? Count – Below 10th: How often is the best target beyond the first page?
    104. 104. Relevance: 5 metrics (queries tested have “best” result) Mean: Average distance from the top Median: Less sensitive to outliers, but not useful once at least half are ranked #1 Count - Below 1st: How often is the best target something other than 1st? Count – Below 5th: How often is the best target outside the critical area? Count – Below 10th: How often is the best target beyond the first page? OK!
    105. 105. Relevance: 5 metrics (queries tested have “best” result) Mean: Average distance from the top Median: Less sensitive to outliers, but not useful once at least half are ranked #1 Count - Below 1st: How often is the best target something other than 1st? Count – Below 5th: How often is the best target outside the critical area? Count – Below 10th: How often is the best target beyond the first page? OK! Hmmm...
    106. 106. Relevance: 5 metrics (queries tested have “best” result) Mean: Average distance from the top Median: Less sensitive to outliers, but not useful once at least half are ranked #1 Count - Below 1st: How often is the best target something other than 1st? Count – Below 5th: How often is the best target outside the critical area? Count – Below 10th: How often is the best target beyond the first page? OK! Hmmm... Uh oh
    107. 107. Precision: rating scale Evaluate frequent queries’ top search results on this scale • r / Relevant: Based on the information the user provided, the page's ranking is completely relevant • n / Near: The page is not a perfect match, but it’s clearly reasonable for it to be ranked highly • m / Misplaced: You can see why the search engine returned it, but it should not be ranked highly • i / Irrelevant: The result has no apparent relationship to the user’s search
    108. 108. Precision: three metrics Metrics based on degrees of permissiveness 1. strict: only counts completely relevant results 2. loose: counts relevant and near results 3. permissive: counts relevant, near, and misplaced results
    109. 109. Putting it all together: old engine (target) and new Note: low relevance and high precision scores are optimal More on Vanguard case study: http://bit.ly/D3B8c
    110. 110. Agenda 1.The basics of Site Search Analytics (SSA) 2.Exercise 1 (pattern analysis) 3.Things you can do with SSA 4.Exercise 2 (longitudinal analysis 5.More things you can do with SSA 6.A case study 7.More on metrics 8.Things you can do today 9.Discussion
    111. 111. Mapping KPI and metrics: A generic “search success” KPI
    112. 112. Search Metrics: general examples (Lee Romero, blog.leeromero.org) • Total searches for a given time period • Total distinct search terms for a given time period • Total distinct words for a given time period • Average words per search • Top searches for a given time period • Top Searches over time • Not found searches • Error searches • Ratio of searches performed each reporting period to the number of visits for that same time period
    113. 113. Search Metrics: search engine tuning (Jeannine Bartlett, earley.com) Do users not find what they want because the search engine and its ranking and relevance algorithms have not been adequately tuned? Example Benchmarks and Metrics • # of valid queries returning no results / total unique queries • Relative % search results per data source • Relative % click throughs per data source • Pass/fail % for queries using stemming • Pass/fail % for queries with misspellings • Precision measures of“seed”documents sent through the tagging process
    114. 114. Search Metrics: query entry (Jeannine Bartlett, earley.com) Do users not find what they want because the UI for expressing search terms is inadequate or unintuitive? Example Benchmarks and Metrics • % queries in the bottom set of the Zipf Curve (flat vs. hockey-stick distribution) • % queries with no click throughs • % queries using syntactic metadata filtering (date, author, source, document type, geography, etc.) • % queries using Boolean search grammar • % queries using type-ahead against taxonomy terms and synonyms • % queries using faceted semantic refinement • % pages from which search is available
    115. 115. Search Metrics: result sets (Jeannine Bartlett, earley.com) Do users not find what they want because the UI for visualizing result sets is inadequate or unintuitive? Example Benchmarks and Metrics • % queries utilizing multiple results views • % queries with drill-down through clusters • % queries using iterative syntactic metadata filtering (date range, sorting, type or source inclusion/exclusion, etc.) • % queries suggesting broader/narrower terms • % queries suggesting“Best Bets”or“See Also” • % queries using iterative semantic term filtering, inclusion or exclusion
    116. 116. Agenda 1.The basics of Site Search Analytics (SSA) 2.Exercise 1 (pattern analysis) 3.Things you can do with SSA 4.Exercise 2 (longitudinal analysis 5.More things you can do with SSA 6.A case study 7.More on metrics 8.Things you can do today 9.Discussion
    117. 117. Things to do today 1.Set up SSA in Google Analytics 2.Query your queries 3.Start developing a site report card 4.Start incorporating SSA into your user research program
    118. 118. Turn on SSA in Google Analytics Set up GA for your site if you haven’t already Then teach it to parse and capture your search engine’s queries (not set by default) References • http://is.gd/cR0qr • http://is.gd/cR0qP
    119. 119. Seed your analysis by querying your queries Starter questions 1. What are the most frequent unique queries? 2. Are frequent queries retrieving quality results? 3. Click-through rates per frequent query? 4. Most frequently clicked result per query? 5. Which frequent queries retrieve zero results? 6. What are the referrer pages for frequent queries? 7. Which queries retrieve popular documents? 8. What interesting patterns emerge in general?
    120. 120. Use SSA to start work on a site report card
    121. 121. Use SSA to start work on a site report card SSA helps determine common information needs
    122. 122. ©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved. From Christian Rohrer, xdstrategy.com
    123. 123. ©2010 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved. Augment personas and audience profiles with frequent queries Persona example (from Adaptive Path) Frequent queries added (in green)
    124. 124. Agenda 1.The basics of Site Search Analytics (SSA) 2.Exercise 1 (pattern analysis) 3.Things you can do with SSA 4.Exercise 2 (longitudinal analysis 5.More things you can do with SSA 6.A case study 7.More on metrics 8.Things you can do today 9.Discussion
    125. 125. Comparing referral queries with local queries
    126. 126. Long tail queries: Longer, more complex (fromVanguard) Short head: common queries Long tail: common queries Beneficiary form 401(k) beneficiary career forms amt money market location loans calculator 403(b)(7) account asset transfer authorization automatic investing Wire transfer instructions adoption agreement international wire transfers socially responsible investing Vanguard tax identification number IRA Asset Transfer form fdic insured account early withdrawal penalties
    127. 127. Now on sale Search Analytics forYour Site: Conversations with Your Customers by Louis Rosenfeld (Rosenfeld Media, 2011) www.rosenfeldmedia.com Use code WEBDAGENE2013 for 20% off all Rosenfeld Media books
    128. 128. Louis Rosenfeld lou@louisrosenfeld.com www.louisrosenfeld.com www.rosenfeldmedia.com @louisrosenfeld @rosenfeldmedia This presentation available on SlideShare: http://slidesha.re/otzE2t Say hello
    1. ¿Le ha llamado la atención una diapositiva en particular?

      Recortar diapositivas es una manera útil de recopilar información importante para consultarla más tarde.

    ×