BrightonSEO: SearchBots: Lost Children or Hungry Psychopaths? What Do Searchbots Actually Do?

  • 3,936 views
Uploaded on

We think we know how searchbots interact with our websites, but do we really? What data is there that tell us where they visited, what they requested, how long they spent? …

We think we know how searchbots interact with our websites, but do we really? What data is there that tell us where they visited, what they requested, how long they spent?

This talk examines various data sources, visualises searchbots visits (animating them over time), explores how we can help searchbots - and even attempts to turn their visits into sound.

More in: Technology , Design
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,936
On Slideshare
0
From Embeds
0
Number of Embeds
9

Actions

Shares
Downloads
0
Comments
0
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Searchbots:Lost Children ... or ... Hungry Psychopaths?What Do Searchbots Actually Do? (and why it matters) © 2007-2012 Roland Dunn
  • 2. © 2007-2012 Roland Dunn
  • 3. “Googlebot ... may be unable to completely index all the content on your site” © 2007-2012 Roland Dunn
  • 4. © 2007-2012 Roland Dunn
  • 5. © 2007-2012 Roland Dunn
  • 6. © 2007-2012 Roland Dunn
  • 7. © 2007-2012 Roland Dunn
  • 8. Doesn’t tell us what searchbots index © 2007-2012 Roland Dunn
  • 9. © 2007-2012 Roland Dunn
  • 10. © 2007-2012 Roland Dunn
  • 11. © 2007-2012 Roland Dunn
  • 12. © 2007-2012 Roland Dunn
  • 13. Webserver Logfiles .... © 2007-2012 Roland Dunn
  • 14. Webserver Logfiles .... © 2007-2012 Roland Dunn
  • 15. Webserver Logfiles .... © 2007-2012 Roland Dunn
  • 16. © 2007-2012 Roland Dunn
  • 17. Over 2 months, 40% of all Googlebot’s requestswere to …. just 2 URL’s (on-site search URLs) © 2007-2012 Roland Dunn
  • 18. Over 2 months, 40% of all Googlebot’s requestswere to …. just 2 URL’s (on-site search URLs)On a website with approximately 40,000 URLs inGoogle’s index (allegedly, according to site:…) © 2007-2012 Roland Dunn
  • 19. Over 2 months, 40% of all Googlebot’s requestswere to …. just 2 URL’s (on-site search URLs)On a website with approximately 40,000 URLs inGoogle’s index (allegedly, according to site:…)On a website serving approx. 150-200K uniquenatural search visits/month © 2007-2012 Roland Dunn
  • 20. Over 2 months, 40% of all Googlebot’s requestswere to …. 2 URL’s.On a website with approximately 40,000 URLs inGoogle’s index.On a website serving approx. 150-200K uniquenatural search visits/month © 2007-2012 Roland Dunn
  • 21. © 2007-2012 Roland Dunn
  • 22. Categorise URLs by “top level section” – perhaps more useful than just URLs ... © 2007-2012 Roland Dunn
  • 23. Table of “top level sections” requested by Googlebot © 2007-2012 Roland Dunn
  • 24. Chart of “top level sections” requested by Gbot – 6 months © 2007-2012 Roland Dunn
  • 25. © 2007-2012 Roland Dunn
  • 26. Does Googlebot request all URLs served by a website? © 2007-2012 Roland Dunn
  • 27. Does Googlebot request all URLs served by a website? No! © 2007-2012 Roland Dunn
  • 28. How Does Googlebot Spend Its Time? What Does it Request? © 2007-2012 Roland Dunn
  • 29. Does Googlebot spend its time cost-effectively? © 2007-2012 Roland Dunn
  • 30. Does Googlebot spend its time cost-effectively? Not always. © 2007-2012 Roland Dunn
  • 31. Googlebot does not always request all content © 2007-2012 Roland Dunn
  • 32. Googlebot can become distracted, obsessed, or even lost: © 2007-2012 Roland Dunn
  • 33. Googlebot can become distracted, obsessed, or even lost:• On-site search• Additive filters/faceted URLs e.g. “shoes?size=3&colour=green&price=10&brand=smith”• Sections with thin and/or very similar content © 2007-2012 Roland Dunn
  • 34. Googlebot may not spend its time efficientlyIt needs help focusing on what we value © 2007-2012 Roland Dunn
  • 35. © 2007-2012 Roland Dunn
  • 36. SMX West, March 2011, Matt Cutts: (http://goo.gl/ZZz7E):“... if Google determines a site isn’t as useful to users, they may not crawl it as frequently" © 2007-2012 Roland Dunn
  • 37. © 2007-2012 Roland Dunn
  • 38. Conclusions:• Searchbots do not always request all content• Searchbots can become distracted• Searchbots may use their time inefficiently• Their visits are precious – we need to treasure them• We may need to help them focus © 2007-2012 Roland Dunn
  • 39. Suggestions:• Embrace logfiles – full of useful information © 2007-2012 Roland Dunn
  • 40. Suggestions:• Embrace logfiles – full of useful information• Check searchbot behaviour © 2007-2012 Roland Dunn
  • 41. Suggestions:• Embrace logfiles – full of useful information• Check searchbot behaviour• If distracted, lost, inefficient, poor experience: • Alter internal navigation & linking (e.g. flatten hierarchy) • Robots.txt out (blunt approach) • Alter URL construction (expensive!) © 2007-2012 Roland Dunn
  • 42. © 2007-2012 Roland Dunn
  • 43. Lost Children ... or ... Hungry Psychopaths? © 2007-2012 Roland Dunn
  • 44. Or .... Distracted Teenager? © 2007-2012 Roland Dunn
  • 45. © 2007-2012 Roland Dunn
  • 46. Sonification: What do Search Engines Requests Sound Like? © 2007-2012 Roland Dunn
  • 47. Sonification: What do Search Engines Requests Sound Like? © 2007-2012 Roland Dunn
  • 48. © 2007-2012 Roland Dunn
  • 49. Online Branding: Natural Search:http://www.refinedpractice.com/ http://www.cloudshapes.co.uk/T: @RefinedPractice T: @roland_dunn Slides Available At: http://www.cloudshapes.co.uk/talks/ © 2007-2012 Roland Dunn