BrightonSEO: SearchBots: Lost Children or Hungry Psychopaths? What Do Searchbots Actually Do?

4,709 views
4,681 views

Published on

We think we know how searchbots interact with our websites, but do we really? What data is there that tell us where they visited, what they requested, how long they spent?

This talk examines various data sources, visualises searchbots visits (animating them over time), explores how we can help searchbots - and even attempts to turn their visits into sound.

Published in: Technology, Design

BrightonSEO: SearchBots: Lost Children or Hungry Psychopaths? What Do Searchbots Actually Do?

  1. 1. Searchbots:Lost Children ... or ... Hungry Psychopaths?What Do Searchbots Actually Do? (and why it matters) © 2007-2012 Roland Dunn
  2. 2. © 2007-2012 Roland Dunn
  3. 3. “Googlebot ... may be unable to completely index all the content on your site” © 2007-2012 Roland Dunn
  4. 4. © 2007-2012 Roland Dunn
  5. 5. © 2007-2012 Roland Dunn
  6. 6. © 2007-2012 Roland Dunn
  7. 7. © 2007-2012 Roland Dunn
  8. 8. Doesn’t tell us what searchbots index © 2007-2012 Roland Dunn
  9. 9. © 2007-2012 Roland Dunn
  10. 10. © 2007-2012 Roland Dunn
  11. 11. © 2007-2012 Roland Dunn
  12. 12. © 2007-2012 Roland Dunn
  13. 13. Webserver Logfiles .... © 2007-2012 Roland Dunn
  14. 14. Webserver Logfiles .... © 2007-2012 Roland Dunn
  15. 15. Webserver Logfiles .... © 2007-2012 Roland Dunn
  16. 16. © 2007-2012 Roland Dunn
  17. 17. Over 2 months, 40% of all Googlebot’s requestswere to …. just 2 URL’s (on-site search URLs) © 2007-2012 Roland Dunn
  18. 18. Over 2 months, 40% of all Googlebot’s requestswere to …. just 2 URL’s (on-site search URLs)On a website with approximately 40,000 URLs inGoogle’s index (allegedly, according to site:…) © 2007-2012 Roland Dunn
  19. 19. Over 2 months, 40% of all Googlebot’s requestswere to …. just 2 URL’s (on-site search URLs)On a website with approximately 40,000 URLs inGoogle’s index (allegedly, according to site:…)On a website serving approx. 150-200K uniquenatural search visits/month © 2007-2012 Roland Dunn
  20. 20. Over 2 months, 40% of all Googlebot’s requestswere to …. 2 URL’s.On a website with approximately 40,000 URLs inGoogle’s index.On a website serving approx. 150-200K uniquenatural search visits/month © 2007-2012 Roland Dunn
  21. 21. © 2007-2012 Roland Dunn
  22. 22. Categorise URLs by “top level section” – perhaps more useful than just URLs ... © 2007-2012 Roland Dunn
  23. 23. Table of “top level sections” requested by Googlebot © 2007-2012 Roland Dunn
  24. 24. Chart of “top level sections” requested by Gbot – 6 months © 2007-2012 Roland Dunn
  25. 25. © 2007-2012 Roland Dunn
  26. 26. Does Googlebot request all URLs served by a website? © 2007-2012 Roland Dunn
  27. 27. Does Googlebot request all URLs served by a website? No! © 2007-2012 Roland Dunn
  28. 28. How Does Googlebot Spend Its Time? What Does it Request? © 2007-2012 Roland Dunn
  29. 29. Does Googlebot spend its time cost-effectively? © 2007-2012 Roland Dunn
  30. 30. Does Googlebot spend its time cost-effectively? Not always. © 2007-2012 Roland Dunn
  31. 31. Googlebot does not always request all content © 2007-2012 Roland Dunn
  32. 32. Googlebot can become distracted, obsessed, or even lost: © 2007-2012 Roland Dunn
  33. 33. Googlebot can become distracted, obsessed, or even lost:• On-site search• Additive filters/faceted URLs e.g. “shoes?size=3&colour=green&price=10&brand=smith”• Sections with thin and/or very similar content © 2007-2012 Roland Dunn
  34. 34. Googlebot may not spend its time efficientlyIt needs help focusing on what we value © 2007-2012 Roland Dunn
  35. 35. © 2007-2012 Roland Dunn
  36. 36. SMX West, March 2011, Matt Cutts: (http://goo.gl/ZZz7E):“... if Google determines a site isn’t as useful to users, they may not crawl it as frequently" © 2007-2012 Roland Dunn
  37. 37. © 2007-2012 Roland Dunn
  38. 38. Conclusions:• Searchbots do not always request all content• Searchbots can become distracted• Searchbots may use their time inefficiently• Their visits are precious – we need to treasure them• We may need to help them focus © 2007-2012 Roland Dunn
  39. 39. Suggestions:• Embrace logfiles – full of useful information © 2007-2012 Roland Dunn
  40. 40. Suggestions:• Embrace logfiles – full of useful information• Check searchbot behaviour © 2007-2012 Roland Dunn
  41. 41. Suggestions:• Embrace logfiles – full of useful information• Check searchbot behaviour• If distracted, lost, inefficient, poor experience: • Alter internal navigation & linking (e.g. flatten hierarchy) • Robots.txt out (blunt approach) • Alter URL construction (expensive!) © 2007-2012 Roland Dunn
  42. 42. © 2007-2012 Roland Dunn
  43. 43. Lost Children ... or ... Hungry Psychopaths? © 2007-2012 Roland Dunn
  44. 44. Or .... Distracted Teenager? © 2007-2012 Roland Dunn
  45. 45. © 2007-2012 Roland Dunn
  46. 46. Sonification: What do Search Engines Requests Sound Like? © 2007-2012 Roland Dunn
  47. 47. Sonification: What do Search Engines Requests Sound Like? © 2007-2012 Roland Dunn
  48. 48. © 2007-2012 Roland Dunn
  49. 49. Online Branding: Natural Search:http://www.refinedpractice.com/ http://www.cloudshapes.co.uk/T: @RefinedPractice T: @roland_dunn Slides Available At: http://www.cloudshapes.co.uk/talks/ © 2007-2012 Roland Dunn

×