ionSearch 2012Roland DunnPartner – Refined Practice
Searchbots:Lost Children ... or ... Hungry Psychopaths?What Do Searchbots Actually Do? (and why it matters)               ...
© 2007-2012 Roland Dunn
Straightforward for humans and searchbots to find your content?                   © 2007-2012 Roland Dunn
Do you provide what your potential customers want?                          Straightforward for humans and searchbots to f...
Do you stand out? Are you compelling, engaging, useful,                                       unique, the best?Do you prov...
Do you stand out? Are you compelling, engaging, useful,                                       unique, the best?Do you prov...
© 2007-2012 Roland Dunn
“Googlebot ... may be unable to completely index  all the content on your site”                      © 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
Doesn’t tell us what searchbots index                      © 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
Webserver Logfiles ....                    © 2007-2012 Roland Dunn
Webserver Logfiles ....                    © 2007-2012 Roland Dunn
Webserver Logfiles ....                    © 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
Over 2 months, 40% of all Googlebot’s requestswere to …. just 2 URL’s (on-site search URLs)                     © 2007-201...
Over 2 months, 40% of all Googlebot’s requestswere to …. just 2 URL’s (on-site search URLs)On a website with approximately...
Over 2 months, 40% of all Googlebot’s requestswere to …. just 2 URL’s (on-site search URLs)On a website with approximately...
Over 2 months, 40% of all Googlebot’s requestswere to …. 2 URL’s.On a website with approximately 40,000 URLs inGoogle’s in...
Over 2 months, 40% of all Googlebot’s requestswere to …. 2 URL’s.On a website with approximately 40,000 URLs inGoogle’s in...
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
Categorise URLs by “top level section” – perhaps more  useful than just URLs ...                         © 2007-2012 Rolan...
Table of “top level sections” requested by Googlebot                        © 2007-2012 Roland Dunn
Chart of “top level sections” requested by Gbot – 6 months                         © 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
Does Googlebot request all URLs served by a website?                           © 2007-2012 Roland Dunn
Does Googlebot request all URLs served by a website?                                                     No!              ...
How Does Googlebot Spend Its Time? What Does it Request?                           © 2007-2012 Roland Dunn
Does Googlebot spend its time cost-effectively?                           © 2007-2012 Roland Dunn
Does Googlebot spend its time cost-effectively?                                                     Not always.           ...
© 2007-2012 Roland Dunn
Googlebot does not always request all content                      © 2007-2012 Roland Dunn
Googlebot can become distracted, obsessed, or even lost:                          © 2007-2012 Roland Dunn
Googlebot can become distracted, obsessed, or even lost:• On-site search• Additive filters/faceted URLs e.g.  “shoes?size=...
Googlebot may not spend its time efficientlyIt needs help focusing on what we value                      © 2007-2012 Rolan...
© 2007-2012 Roland Dunn
SMX West, March 2011, Matt Cutts:   (http://goo.gl/ZZz7E):“... if Google determines a site isn’t as useful to   users, the...
© 2007-2012 Roland Dunn
Suggestions:• Embrace logfiles – full of useful information                               © 2007-2012 Roland Dunn
Suggestions:• Embrace logfiles – full of useful information• Periodically check your searchbot behaviour                  ...
Suggestions:• Embrace logfiles – full of useful information• Periodically check your searchbot behaviour• Especially if th...
If searchbots appear distracted, unfocused, lost:                               © 2007-2012 Roland Dunn
If searchbots appear distracted, unfocused, lost:• Alter internal navigation & linking (e.g. flatten hierarchy)           ...
If searchbots appear distracted, unfocused, lost:• Alter internal navigation & linking (e.g. flatten hierarchy)• Robots.tx...
If searchbots appear distracted, unfocused, lost:• Alter internal navigation & linking (e.g. flatten hierarchy)• Robots.tx...
If searchbots appear distracted, unfocused, lost:• Alter internal navigation & linking (e.g. flatten hierarchy)• Robots.tx...
© 2007-2012 Roland Dunn
Lost Children ... or ... Hungry Psychopaths?                        © 2007-2012 Roland Dunn
Or .... Distracted Teenager?             © 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
Sonification: What do Search Engines Requests Sound Like?                          © 2007-2012 Roland Dunn
Sonification: What do Search Engines Requests Sound Like?                          © 2007-2012 Roland Dunn
© 2007-2012 Roland Dunn
Online Branding:                               Natural Search:http://www.refinedpractice.com/                http://www.cl...
Roland Dunn - What DO Searchbots Actually Do? - ionSearch 2012
Upcoming SlideShare
Loading in...5
×

Roland Dunn - What DO Searchbots Actually Do? - ionSearch 2012

534

Published on

Published in: Technology, Design
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
534
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Roland Dunn - What DO Searchbots Actually Do? - ionSearch 2012

  1. 1. ionSearch 2012Roland DunnPartner – Refined Practice
  2. 2. Searchbots:Lost Children ... or ... Hungry Psychopaths?What Do Searchbots Actually Do? (and why it matters) © 2007-2012 Roland Dunn
  3. 3. © 2007-2012 Roland Dunn
  4. 4. Straightforward for humans and searchbots to find your content? © 2007-2012 Roland Dunn
  5. 5. Do you provide what your potential customers want? Straightforward for humans and searchbots to find your content? © 2007-2012 Roland Dunn
  6. 6. Do you stand out? Are you compelling, engaging, useful, unique, the best?Do you provide what your potential customers want? Straightforward for humans and searchbots to find your content? © 2007-2012 Roland Dunn
  7. 7. Do you stand out? Are you compelling, engaging, useful, unique, the best?Do you provide what your potential customers want? Straightforward for humans and searchbots to find your content? © 2007-2012 Roland Dunn
  8. 8. © 2007-2012 Roland Dunn
  9. 9. “Googlebot ... may be unable to completely index all the content on your site” © 2007-2012 Roland Dunn
  10. 10. © 2007-2012 Roland Dunn
  11. 11. © 2007-2012 Roland Dunn
  12. 12. © 2007-2012 Roland Dunn
  13. 13. © 2007-2012 Roland Dunn
  14. 14. Doesn’t tell us what searchbots index © 2007-2012 Roland Dunn
  15. 15. © 2007-2012 Roland Dunn
  16. 16. © 2007-2012 Roland Dunn
  17. 17. © 2007-2012 Roland Dunn
  18. 18. © 2007-2012 Roland Dunn
  19. 19. Webserver Logfiles .... © 2007-2012 Roland Dunn
  20. 20. Webserver Logfiles .... © 2007-2012 Roland Dunn
  21. 21. Webserver Logfiles .... © 2007-2012 Roland Dunn
  22. 22. © 2007-2012 Roland Dunn
  23. 23. Over 2 months, 40% of all Googlebot’s requestswere to …. just 2 URL’s (on-site search URLs) © 2007-2012 Roland Dunn
  24. 24. Over 2 months, 40% of all Googlebot’s requestswere to …. just 2 URL’s (on-site search URLs)On a website with approximately 40,000 URLs inGoogle’s index (allegedly, according to site:…) © 2007-2012 Roland Dunn
  25. 25. Over 2 months, 40% of all Googlebot’s requestswere to …. just 2 URL’s (on-site search URLs)On a website with approximately 40,000 URLs inGoogle’s index (allegedly, according to site:…)On a website serving approx. 150-200K uniquenatural search visits/month © 2007-2012 Roland Dunn
  26. 26. Over 2 months, 40% of all Googlebot’s requestswere to …. 2 URL’s.On a website with approximately 40,000 URLs inGoogle’s index.On a website serving approx. 150-200K uniquenatural search visits/month © 2007-2012 Roland Dunn
  27. 27. Over 2 months, 40% of all Googlebot’s requestswere to …. 2 URL’s.On a website with approximately 40,000 URLs inGoogle’s index.On a website serving approx. 150-200K uniquenatural search visits/month © 2007-2012 Roland Dunn
  28. 28. © 2007-2012 Roland Dunn
  29. 29. © 2007-2012 Roland Dunn
  30. 30. © 2007-2012 Roland Dunn
  31. 31. Categorise URLs by “top level section” – perhaps more useful than just URLs ... © 2007-2012 Roland Dunn
  32. 32. Table of “top level sections” requested by Googlebot © 2007-2012 Roland Dunn
  33. 33. Chart of “top level sections” requested by Gbot – 6 months © 2007-2012 Roland Dunn
  34. 34. © 2007-2012 Roland Dunn
  35. 35. Does Googlebot request all URLs served by a website? © 2007-2012 Roland Dunn
  36. 36. Does Googlebot request all URLs served by a website? No! © 2007-2012 Roland Dunn
  37. 37. How Does Googlebot Spend Its Time? What Does it Request? © 2007-2012 Roland Dunn
  38. 38. Does Googlebot spend its time cost-effectively? © 2007-2012 Roland Dunn
  39. 39. Does Googlebot spend its time cost-effectively? Not always. © 2007-2012 Roland Dunn
  40. 40. © 2007-2012 Roland Dunn
  41. 41. Googlebot does not always request all content © 2007-2012 Roland Dunn
  42. 42. Googlebot can become distracted, obsessed, or even lost: © 2007-2012 Roland Dunn
  43. 43. Googlebot can become distracted, obsessed, or even lost:• On-site search• Additive filters/faceted URLs e.g. “shoes?size=3&colour=green&price=10&brand=smith”• Sections with thin and/or very similar content © 2007-2012 Roland Dunn
  44. 44. Googlebot may not spend its time efficientlyIt needs help focusing on what we value © 2007-2012 Roland Dunn
  45. 45. © 2007-2012 Roland Dunn
  46. 46. SMX West, March 2011, Matt Cutts: (http://goo.gl/ZZz7E):“... if Google determines a site isn’t as useful to users, they may not crawl it as frequently" © 2007-2012 Roland Dunn
  47. 47. © 2007-2012 Roland Dunn
  48. 48. Suggestions:• Embrace logfiles – full of useful information © 2007-2012 Roland Dunn
  49. 49. Suggestions:• Embrace logfiles – full of useful information• Periodically check your searchbot behaviour © 2007-2012 Roland Dunn
  50. 50. Suggestions:• Embrace logfiles – full of useful information• Periodically check your searchbot behaviour• Especially if the “Googlebot found a high number” message keeps appearing © 2007-2012 Roland Dunn
  51. 51. If searchbots appear distracted, unfocused, lost: © 2007-2012 Roland Dunn
  52. 52. If searchbots appear distracted, unfocused, lost:• Alter internal navigation & linking (e.g. flatten hierarchy) © 2007-2012 Roland Dunn
  53. 53. If searchbots appear distracted, unfocused, lost:• Alter internal navigation & linking (e.g. flatten hierarchy)• Robots.txt out (blunt approach) © 2007-2012 Roland Dunn
  54. 54. If searchbots appear distracted, unfocused, lost:• Alter internal navigation & linking (e.g. flatten hierarchy)• Robots.txt out (blunt approach)• Alter URL construction (expensive!) © 2007-2012 Roland Dunn
  55. 55. If searchbots appear distracted, unfocused, lost:• Alter internal navigation & linking (e.g. flatten hierarchy)• Robots.txt out (blunt approach)• Alter URL construction (expensive!)• Check “sitemaps” – all URLs? What priorities expressed? © 2007-2012 Roland Dunn
  56. 56. © 2007-2012 Roland Dunn
  57. 57. Lost Children ... or ... Hungry Psychopaths? © 2007-2012 Roland Dunn
  58. 58. Or .... Distracted Teenager? © 2007-2012 Roland Dunn
  59. 59. © 2007-2012 Roland Dunn
  60. 60. Sonification: What do Search Engines Requests Sound Like? © 2007-2012 Roland Dunn
  61. 61. Sonification: What do Search Engines Requests Sound Like? © 2007-2012 Roland Dunn
  62. 62. © 2007-2012 Roland Dunn
  63. 63. Online Branding: Natural Search:http://www.refinedpractice.com/ http://www.cloudshapes.co.uk/T: @RefinedPractice T: @roland_dunn Slides Available At: http://www.cloudshapes.co.uk/talks/ © 2007-2012 Roland Dunn
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×