Site search analytics workshop presentation

Workshop:Search Analyticsfor Your SiteLouis Rosenfeldlou@louisrosenfeld.com • @louisrosenfeldUX Hong Kong • 17 February, 2012

Hello, my name is Lou www.louisrosenfeld.com | www.rosenfeldmedia.com

Agenda

Agenda1.The basics of Site Search Analytics (SSA)

Agenda1.The basics of Site Search Analytics (SSA)2.Pattern analysis exercise

Agenda1.The basics of Site Search Analytics (SSA)2.Pattern analysis exercise3.Eight things you can do with SSA

Agenda1.The basics of Site Search Analytics (SSA)2.Pattern analysis exercise3.Eight things you can do with SSA4.A case study

Agenda1.The basics of Site Search Analytics (SSA)2.Pattern analysis exercise3.Eight things you can do with SSA4.A case study5.Some things you can do today

Let’s look at the data

No, let’s look at the real dataCritical elements in bold: IP address, time/date stamp, query, and # of results:XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?access=p&entqr=0 &output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8 &client=www&oe=UTF-8&proxystylesheet=www& q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /searchaccess=p&entqr=0 &output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ie=UTF-8&client=www& q=license+plate&ud=1&site=AllSites &spell=1&oe=UTF-8&proxystylesheet=www& ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16

No, let’s look at the real dataCritical elements in bold: IP address, time/date stamp, query, and # of results: What are usersXXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?access=p&entqr=0 searching? &output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8 &client=www&oe=UTF-8&proxystylesheet=www& q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /searchaccess=p&entqr=0 &output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ie=UTF-8&client=www& q=license+plate&ud=1&site=AllSites &spell=1&oe=UTF-8&proxystylesheet=www& ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16

No, let’s look at the real dataCritical elements in bold: IP address, time/date stamp, query, and # of results: What are usersXXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?access=p&entqr=0 searching? &output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8 &client=www&oe=UTF-8&proxystylesheet=www& q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /searchaccess=p&entqr=0 &output=xml_no_dtd&sort=date%3AD%3AL How often are %3Ad1&ie=UTF-8&client=www& users failing? q=license+plate&ud=1&site=AllSites &spell=1&oe=UTF-8&proxystylesheet=www& ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16

SSA is semantically rich data, and...

SSA is semantically rich data, and... Queries sorted by frequency

...what users want--in their own words

A handful of queries/tasks/ways to navigate/features/ documents A little goes a long waymeet the needs of your most important audiences

A handful of queries/tasks/ways to navigate/features/ documents A little goes a long waymeet the needs of your most important audiences Not all queries are distributed equally

A handful of queries/tasks/ways to navigate/features/ documents A little goes a long waymeet the needs of your most important audiences Nor do they diminish gradually

A handful of queries/tasks/ways to navigate/features/ documents A little goes a long waymeet the needs of your most important audiences 80/20 rule isn’t quite accurate

(and the tail is quite long)

The Long Tail is(and the tail is quite long) much longer than you’d suspect

The Zipf Distribution, textually

Insert Long Tail here

Agenda

Pattern analysis exerciseWork in pairs • Each pair should have a laptop with Excel • Laptop platform (Mac, PC) doesn’t matterDownload data ﬁlesRefer to exercise sheetNo right answersHave fun!

Agenda

Eight things you can do with SSA 1.Make it harder to get lost in deep content 2.Make search smarter 3.Reduce jargon 4.Learn how your audiences differ 5.Know when to publish what 6.Own and enjoy your failures 7.Learn from the next search 8.Predict the future

#1Make it harder to get lost

Start with basic SSA data:queries and query frequency Percent: volume of search activity for a unique query during a particular time period Cumulative Percent: running sum of percentages

Tease out common content types

Tease out common content types Took an hour to... • Analyze top 50 queries (20% of all search activity) • Ask and iterate: “what kind of content would users be looking for when they searched these terms?” • Add cumulative percentages Result: prioritized list of potential content types #1) application: 11.77% #2) reference: 10.5% #3) instructions: 8.6% #4) main/navigation pages: 5.91% #5) contact info: 5.79% #6) news/announcements: 4.27%

Clear content types lead tobetter contextual navigation album pages artist descriptions TV listingsalbum reviews discography artist bios

Clear content types improvesearch performance

Clear content types improvesearch performance Content objects related to products

Clear content types improvesearch performance Content objects related to products Raw search results

#2Make search smarter

Contextualizing “advanced” features

Session data suggestprogression and context

Session data suggestprogression and context search session patterns 1. solar energy 2. how solar energy works

Session data suggestprogression and context search session patterns 1. solar energy 2. how solar energy works search session patterns 1. solar energy 2. energy

Session data suggestprogression and context search session patterns search session patterns 1. solar energy 1. solar energy 2. solar energy charts 2. how solar energy works search session patterns 1. solar energy 2. energy

Session data suggestprogression and context search session patterns search session patterns 1. solar energy 1. solar energy 2. solar energy charts 2. how solar energy works search session patterns search session patterns 1. solar energy 1. solar energy 2. explain solar energy 2. energy

Session data suggestprogression and context search session patterns search session patterns 1. solar energy 1. solar energy 2. solar energy charts 2. how solar energy works search session patterns search session patterns 1. solar energy 1. solar energy 2. explain solar energy 2. energy search session patterns 1. solar energy 2. solar energy news

Recognizingproper nouns,dates, andunique ID#s

#3Reduce jargon

Saving the brand by killing jargonat a community collegeJargon related to online education: FlexEd, COD, College on DemandMarketing’s solution: expensive campaign to educate public (via posters, brochures)The Numbers query rank query (from SSA): #22 online* #101 COD #259 College on Demand #389 FlexTrack * “online” part of 213 queriesResult: content relabeled, money saved

#4Learn how your audiences differ

Who cares about what? (AIGA.org)

Who cares about what? (Open U)

Why analyze queries by audience?Fortify your personas with dataLearn about differences between audiences • Open University “Enquirers”: 16 of 25 queries are for subjects not taught at OU • Open University Students: search for course codes, topics dealing with completing programDetermine what’s commonly important to all audiences (these queries better work well)

#5Know when to publish what

Interest in the football team: going...

Interest in the football team: going... ...going...

Interest in the football team: going... ...going... gone

Time toInterest in the study! football team: going... ...going... gone

BeforeTax Day

AfterTax Day

#6Own and enjoy your failures

Failed navigation?Examining unexpected searching Look for places searches happen beyond main page What’s going on? • Navigational failure? • Content failure? • Something else?

Where navigation is failing (“Professional Resources” page)Do users andAIGA meandifferentthings by“ProfessionalResources”?

Comparing what users ﬁndand what they want

Failed business goals?Developing custom metrics Netﬂix asks 1. Which movies most frequently searched? (query count) 2. Which of them most frequently clicked through? (MDP views) 3. Which of them least frequently added to queue? (queue adds)

#7Learn from the next search

Learn from search sessions(when you can)

Session analysis from theTeach for America intranetThese queries co-occur within sessions: why?      

TFAnet session analysis results• Searches for “delta ICEG” performpoorly (way below the fold)• Users then try an (incorrect)alternative (“delta learning team”) 43

#8Predict the future

Shaping theFinancial Times’ editorial agendaFT compares these • Spiking queries for proper nouns (i.e., people and companies) • Recent editorial coverage of people and companiesDiscrepancy? • Breaking story?! • Let the editors know!

Agenda

Avoiding a disaster at VanguardVanguard used SSA to help benchmark existing search engine’s performance and help select new engineNew search engine “performed” poorlyBut IT needed Information Architect & Dev Team Meeting Where’s the proof? convincing You can’t tell to delay for sure. Search seems to have a few problems… Nah launch .

What to do?Test performance of most frequent queriesMeasure using original two sets of metrics 1.relevance: how reliably the search engine returns the best matches ﬁrst 2.precision: proportion of relevant and irrelevant results clustered at the top of the list

Relevance: 5 metrics(queries tested have “best” result) Mean: Average distance from the topMedian: Less sensitive to outliers, but not useful once at least half are ranked #1Count - Below 1st: How often is the best target something other than 1st?Count – Below 5th: How often is the best target outside the critical area?Count – Below 10th: How often is the best target beyond the ﬁrst page?

Relevance: 5 metrics(queries tested have “best” result) Mean: Average distance from the topMedian: Less sensitive to outliers, but not useful once at least half are ranked #1Count - Below 1st: How often is the best target something other than 1st?Count – Below 5th: How often is the best target outside the critical area?Count – Below 10th: How OK! often is the best target beyond the ﬁrst page?

Relevance: 5 metrics(queries tested have “best” result) Mean: Average distance from the topMedian: Less sensitive to outliers, but not useful once at least half are ranked #1Count - Below 1st: How often is the best target something other than 1st?Count – Below 5th: How often is the best target outside the critical area? Hmmm...Count – Below 10th: How OK! often is the best target beyond the ﬁrst page?

Relevance: 5 metrics(queries tested have “best” result) Mean: Average distance from the topMedian: Less sensitive to outliers, but not useful once at least half are ranked #1Count - Below 1st: How often is the best target something other than 1st? Uh ohCount – Below 5th: How often is the best target outside the critical area? Hmmm...Count – Below 10th: How OK! often is the best target beyond the ﬁrst page?

Precision:rating scale Evaluate frequent queries’ top search results on this scale • r / Relevant: Based on the information the user provided, the pages ranking is completely relevant • n / Near: The page is not a perfect match, but it’s clearly reasonable for it to be ranked highly • m / Misplaced: You can see why the search engine returned it, but it should not be ranked highly • i / Irrelevant: The result has no apparent relationship to the user’s search

Precision:three metrics Metrics based on degrees of permissiveness 1. strict: only counts completely relevant results 2. loose: counts relevant and near results 3. permissive: counts relevant, near, and misplaced results

Putting it all together:old engine (target) and new Note: low relevance and high precision scores are optimal More on Vanguard case study: http://bit.ly/D3B8c

Agenda

Some things you can do today

Some things you can do today1.Set up SSA in Google Analytics

Some things you can do today1.Set up SSA in Google Analytics2.Query your queries

Some things you can do today1.Set up SSA in Google Analytics2.Query your queries3.Start developing a site report card

Turn on SSA in Google AnalyticsSet up GA for your site if you haven’t alreadyThen teach it to parse and capture your search engine’s queries (not set by default)References • http://is.gd/cR0qr • http://is.gd/cR0qP

Seed your analysis byquerying your queriesStarter questions 1. What are the most frequent unique queries? 2. Are frequent queries retrieving quality results? 3. Click-through rates per frequent query? 4. Most frequently clicked result per query? 5. Which frequent queries retrieve zero results? 6. What are the referrer pages for frequent queries? 7. Which queries retrieve popular documents? 8. What interesting patterns emerge in general?

Use SSA to start workon a site report card

Use SSA to start work SSA helps determine commonon a site report card information needs

Agenda

Agenda1.The basics of Site Search Analytics (SSA)2.Pattern analysis exercise3.Eight things you can do with SSA4.A case study5.Some things you can do today Finis!

Now on sale Search Analytics for Your Site: Conversations with Your Customers by Louis Rosenfeld (Rosenfeld Media, 2011) www.rosenfeldmedia.com Use code UXHK12 for 25% off all Rosenfeld Media products

Say helloLouis Rosenfeldlou@louisrosenfeld.comwww.louisrosenfeld.comwww.rosenfeldmedia.com@louisrosenfeld@rosenfeldmediaThis presentation available on SlideShare:http://slidesha.re/otzE2t

http://uxlabs.pl	318
http://johnnyholland.org	253
http://lanyrd.com	17
http://www.dt.ntust.edu.tw	15
http://sprmario.posterous.com	5
http://paper.li	5
http://rosenfeldmedia.com	4
http://sprmario.tumblr.com	2
http://webcache.googleusercontent.com	1
http://nicholasthomas.wordpress.com	1
http://twitter.com	1
http://www.netvibes.com	1

Site search analytics workshop presentation

by Louis Rosenfeld on May 09, 2011

Accessibility

Categories

Upload Details

Usage Rights

12 Embeds 623

Statistics

Site search analytics workshop presentation Presentation Transcript