Roland Dunn - What DO Searchbots Actually Do? - ionSearch 2012
Upcoming SlideShare
Loading in...5
×
 

Roland Dunn - What DO Searchbots Actually Do? - ionSearch 2012

on

  • 736 views

 

Statistics

Views

Total Views
736
Views on SlideShare
736
Embed Views
0

Actions

Likes
1
Downloads
1
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Roland Dunn - What DO Searchbots Actually Do? - ionSearch 2012 Roland Dunn - What DO Searchbots Actually Do? - ionSearch 2012 Presentation Transcript

  • ionSearch 2012Roland DunnPartner – Refined Practice
  • Searchbots:Lost Children ... or ... Hungry Psychopaths?What Do Searchbots Actually Do? (and why it matters) © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • Straightforward for humans and searchbots to find your content? © 2007-2012 Roland Dunn
  • Do you provide what your potential customers want? Straightforward for humans and searchbots to find your content? © 2007-2012 Roland Dunn
  • Do you stand out? Are you compelling, engaging, useful, unique, the best?Do you provide what your potential customers want? Straightforward for humans and searchbots to find your content? © 2007-2012 Roland Dunn
  • Do you stand out? Are you compelling, engaging, useful, unique, the best?Do you provide what your potential customers want? Straightforward for humans and searchbots to find your content? © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • “Googlebot ... may be unable to completely index all the content on your site” © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • Doesn’t tell us what searchbots index © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • Webserver Logfiles .... © 2007-2012 Roland Dunn
  • Webserver Logfiles .... © 2007-2012 Roland Dunn
  • Webserver Logfiles .... © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • Over 2 months, 40% of all Googlebot’s requestswere to …. just 2 URL’s (on-site search URLs) © 2007-2012 Roland Dunn
  • Over 2 months, 40% of all Googlebot’s requestswere to …. just 2 URL’s (on-site search URLs)On a website with approximately 40,000 URLs inGoogle’s index (allegedly, according to site:…) © 2007-2012 Roland Dunn
  • Over 2 months, 40% of all Googlebot’s requestswere to …. just 2 URL’s (on-site search URLs)On a website with approximately 40,000 URLs inGoogle’s index (allegedly, according to site:…)On a website serving approx. 150-200K uniquenatural search visits/month © 2007-2012 Roland Dunn
  • Over 2 months, 40% of all Googlebot’s requestswere to …. 2 URL’s.On a website with approximately 40,000 URLs inGoogle’s index.On a website serving approx. 150-200K uniquenatural search visits/month © 2007-2012 Roland Dunn
  • Over 2 months, 40% of all Googlebot’s requestswere to …. 2 URL’s.On a website with approximately 40,000 URLs inGoogle’s index.On a website serving approx. 150-200K uniquenatural search visits/month © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • Categorise URLs by “top level section” – perhaps more useful than just URLs ... © 2007-2012 Roland Dunn
  • Table of “top level sections” requested by Googlebot © 2007-2012 Roland Dunn
  • Chart of “top level sections” requested by Gbot – 6 months © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • Does Googlebot request all URLs served by a website? © 2007-2012 Roland Dunn
  • Does Googlebot request all URLs served by a website? No! © 2007-2012 Roland Dunn
  • How Does Googlebot Spend Its Time? What Does it Request? © 2007-2012 Roland Dunn
  • Does Googlebot spend its time cost-effectively? © 2007-2012 Roland Dunn
  • Does Googlebot spend its time cost-effectively? Not always. © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • Googlebot does not always request all content © 2007-2012 Roland Dunn
  • Googlebot can become distracted, obsessed, or even lost: © 2007-2012 Roland Dunn
  • Googlebot can become distracted, obsessed, or even lost:• On-site search• Additive filters/faceted URLs e.g. “shoes?size=3&colour=green&price=10&brand=smith”• Sections with thin and/or very similar content © 2007-2012 Roland Dunn
  • Googlebot may not spend its time efficientlyIt needs help focusing on what we value © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • SMX West, March 2011, Matt Cutts: (http://goo.gl/ZZz7E):“... if Google determines a site isn’t as useful to users, they may not crawl it as frequently" © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • Suggestions:• Embrace logfiles – full of useful information © 2007-2012 Roland Dunn
  • Suggestions:• Embrace logfiles – full of useful information• Periodically check your searchbot behaviour © 2007-2012 Roland Dunn
  • Suggestions:• Embrace logfiles – full of useful information• Periodically check your searchbot behaviour• Especially if the “Googlebot found a high number” message keeps appearing © 2007-2012 Roland Dunn
  • If searchbots appear distracted, unfocused, lost: © 2007-2012 Roland Dunn
  • If searchbots appear distracted, unfocused, lost:• Alter internal navigation & linking (e.g. flatten hierarchy) © 2007-2012 Roland Dunn
  • If searchbots appear distracted, unfocused, lost:• Alter internal navigation & linking (e.g. flatten hierarchy)• Robots.txt out (blunt approach) © 2007-2012 Roland Dunn
  • If searchbots appear distracted, unfocused, lost:• Alter internal navigation & linking (e.g. flatten hierarchy)• Robots.txt out (blunt approach)• Alter URL construction (expensive!) © 2007-2012 Roland Dunn
  • If searchbots appear distracted, unfocused, lost:• Alter internal navigation & linking (e.g. flatten hierarchy)• Robots.txt out (blunt approach)• Alter URL construction (expensive!)• Check “sitemaps” – all URLs? What priorities expressed? © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • Lost Children ... or ... Hungry Psychopaths? © 2007-2012 Roland Dunn
  • Or .... Distracted Teenager? © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • Sonification: What do Search Engines Requests Sound Like? © 2007-2012 Roland Dunn
  • Sonification: What do Search Engines Requests Sound Like? © 2007-2012 Roland Dunn
  • © 2007-2012 Roland Dunn
  • Online Branding: Natural Search:http://www.refinedpractice.com/ http://www.cloudshapes.co.uk/T: @RefinedPractice T: @roland_dunn Slides Available At: http://www.cloudshapes.co.uk/talks/ © 2007-2012 Roland Dunn