Tools and Tactics for Diagnosing
Technical Search Issues




                           Vanessa Fox
Diagnostic Checklists and
        Resources




• Search Accessibility Checklist
• Search Discoverability Checklist
• Diag...
Search Engine Tools
 Google Webmaster Central
 http://www.google.com/webmasters

 Microsoft Live Search Webmaster Center
 ...
Ranking and Diagnostic Tools
 SEOBook Rank Checker
 http://tools.seobook.com/firefox/rank-checker/

 Firefox Web Developer...
How Search
Engines Work
Crawling                   Indexing             Ranking

Discover links             Canonicalization     Relevance
Check r...
Search Engine Crawlers Haven’t Quite Grown Up Yet
Crawling                Indexing            Ranking

Lack of discovery       Duplication         Display issues
Crawl inef...
Step 1: Get the Data



Pages crawled
Pages indexed
Web traffic
Key ranking metrics
Crawling   Which pages have the search engines crawled?

           What kind of pages are they?




             Has the ...
Benchmarking

    Top ten queries that bring
    search traffic
    Search results position
    URL that ranks
Crawl Issues
Crawl Log Example: Apache Log Analyzer 2 Feed
All Pages Google’s Crawled
1 /**
2 * @see ApacheLogAnalyzer2Feed          Al...
Communicating with Search Robots
Extractable Link Issues: Flash
Extractable Link Issues: Images
Extractable Link Issues: AJAX
Extractable Link Issues: URL Errors
Extractable Link Issues: URLs That Expire
URL Discovery Checklist

 Comprehensive external links
 At least one internal link to every
  page
 XML Sitemap referen...
URL Structure Checklist

Keep number of parameters in
dynamic URLs short
Don’t use temporary URLs that
expire
 Ensure r...
Canonicalization Checklist
 Have only URL for each page
 Put all unneeded details in cookies, rather than URLs
  (sessio...
Crawl Efficiency Checklist

Ensure page load times aren’t slow as to reduce number of
pages crawled
Ensure server is res...
Indexing Issues
Indexing Example: XML Sitemaps
XML Sitemap
                                             Pages Indexed From Sitemap
<?xml v...
Duplicate Content Issues
Partner Content




http://www.google.co.uk/search?q=%22The+Radisson+Edwardian+Vanderbilt+Hotel+stands+am
Indexing Diagnostic Checklist




Have the pages ever been indexed?

If deindexed, are you sure they are no longer in th...
Indexing Checklist: Content Extraction
 Ensure content is in text wherever possible
 Ensure text isn’t hidden in:
   Ja...
Indexing Checklist: Semantic Markup
 Use keywords in title tag
 Ensure each page has a unique meta description tag
 Use...
Optimizing Images
 Don’t put text in images
 Use descriptive ALT text
 Use descriptive filenames
 Provide caption and ...
Ranking
Issues
How’s the Search Engine Traffic?
 Overall Percentage   Percentage Non-Branded
Do You Rank For the Right Things?
              arbor   snowboards   snowboard
Google           1            49        500...
If ranking loss…


Drop For All Keywords              Drop For Only Some Keywords

Does the site rank for different   Do d...
Ranking Checklist

 Relevance
    What is the page about?
    Are the pages ranking for the desired query more relevant...
The Webmaster Guidelines
                     Common Definition of Spam

         On page schemes            Keyword stuff...
Getting Out of the Penalty Box
1. Check if you’ve been penalized
–   Live Search: http://webmaster.live.com
–   Google: ht...
Traffic Issues
Traffic Drop
Display Issues
Would you click this link?
Does the Result Inspire Clicks?
First step in diagnosis: find the root
Jane and Robot Developer Summit
June 12th, 2009 – San Francisco
FREE for SMX attendees!
janeandrobot.com
Twitter.com/janea...
Diagnosing Technical Issues With Search Engine Optimization
Diagnosing Technical Issues With Search Engine Optimization
Upcoming SlideShare
Loading in...5
×

Diagnosing Technical Issues With Search Engine Optimization

21,949

Published on

If your site is having trouble ranking well in search engines such as Google, you've lost ranking, or you've having trouble with a site move or migration, the trouble could be with the site's technical architecture.

View checklists to help diagnose issues with crawling, indexing, and ranking your site's content.

Published in: Technology, Design
3 Comments
45 Likes
Statistics
Notes
No Downloads
Views
Total Views
21,949
On Slideshare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
584
Comments
3
Likes
45
Embeds 0
No embeds

No notes for slide
  • who in audience is: seo Dev Product Know a lot about seo, etc.
  • Diagnosing Technical Issues With Search Engine Optimization

    1. 1. Tools and Tactics for Diagnosing Technical Search Issues Vanessa Fox
    2. 2. Diagnostic Checklists and Resources • Search Accessibility Checklist • Search Discoverability Checklist • Diagnostic Tools janeandrobot.com
    3. 3. Search Engine Tools Google Webmaster Central http://www.google.com/webmasters Microsoft Live Search Webmaster Center http://webmaster.live.com Yahoo! Site Explorer http://siteexplorer.search.yahoo.com Google Analytics http://www.google.com/analytics Google Search http://www.google.com Created by NineByBlue.com
    4. 4. Ranking and Diagnostic Tools SEOBook Rank Checker http://tools.seobook.com/firefox/rank-checker/ Firefox Web Developer Toolbar https://addons.mozilla.org/en-US/firefox/addon/60 Firefox Firebug http://getfirebug.com/ Firefox Live HTTP Headers https://addons.mozilla.org/en-US/firefox/addon/3829 Google Search http://adlab.msn.com/Keyword-Forecast/default.aspx http://janeandrobot.com/resources Created by NineByBlue.com
    5. 5. How Search Engines Work
    6. 6. Crawling Indexing Ranking Discover links Canonicalization Relevance Check robots rules Context extraction Value Bandwidth considerations Topic association Uniqueness URLs Web-wide value Display
    7. 7. Search Engine Crawlers Haven’t Quite Grown Up Yet
    8. 8. Crawling Indexing Ranking Lack of discovery Duplication Display issues Crawl inefficiency Extraction issues Lack of quality links URL issues (infinite, Lack of exposed Guidelines violations redirects, dynamic) content Non-focused content Inaccessible links Non-optimized media
    9. 9. Step 1: Get the Data Pages crawled Pages indexed Web traffic Key ranking metrics
    10. 10. Crawling Which pages have the search engines crawled? What kind of pages are they? Has the search engine Indexing indexed all of the crawled pages? How’s the search engine Ranking traffic?
    11. 11. Benchmarking Top ten queries that bring search traffic Search results position URL that ranks
    12. 12. Crawl Issues
    13. 13. Crawl Log Example: Apache Log Analyzer 2 Feed All Pages Google’s Crawled 1 /** 2 * @see ApacheLogAnalyzer2Feed All Profile Pages Google’s Crawled 3 */ 1 /** 4 require_once 'ApacheLogAnalyzer2Feed.php'; 5 2 * @see ApacheLogAnalyzer2Feed 6 //3 create a new instance, parse access.log and */ 7 //4 write test.xml'ApacheLogAnalyzer2Feed.php'; require_once 5 8 $tool = new ApacheLogAnalyzer2Feed('access.log', 9 'test.xml'); a new instance, parse access.log and write test.xml 6 // create 10 // $tool = entries matching Googlebot useragent 7 select new ApacheLogAnalyzer2Feed('access.log', 'test.xml'); 11 $tool->addFilter('User-Agent',Googlebot useragent with a regular 8 // select entries matching 'Mozilla/5.0 9 expression pattern 12 (compatible; Googlebot/2.1; 13 +http://www.google.com/bot.html)');'regexp:Googlebot'); 10 $tool->addFilter('User-Agent', 14 // run select entries with Request matching a regular expression 11 // 12 // pattern 15 $tool->run(); 12 13 $tool->addFilter('Request', 'regexp:/site/profile.php'); 14 // run 15 $tool->run(); 16 http://code.simonecarletti.com/wiki/apachelog2feed
    14. 14. Communicating with Search Robots
    15. 15. Extractable Link Issues: Flash
    16. 16. Extractable Link Issues: Images
    17. 17. Extractable Link Issues: AJAX
    18. 18. Extractable Link Issues: URL Errors
    19. 19. Extractable Link Issues: URLs That Expire
    20. 20. URL Discovery Checklist  Comprehensive external links  At least one internal link to every page  XML Sitemap referenced in robots.txt with the comprehensive list of canonical URLs  Comprehensive HTML sitemap  Ensure links load without JavaScript, images, or other rich media  Ensure robots.txt and meta robots tag is used correctly http://janeandrobot.com/library/managing-robots-access-to-your-website
    21. 21. URL Structure Checklist Keep number of parameters in dynamic URLs short Don’t use temporary URLs that expire  Ensure redirects are 301 and are short Use dashes rather than underscores when separating words Use keywords in URLs for higher click through and better anchor text
    22. 22. Canonicalization Checklist  Have only URL for each page  Put all unneeded details in cookies, rather than URLs (session IDs, tracking parameters)  Don’t allow infinite parameters  Use 301 redirects for any URL changes  301 redirect www/non-www  Use absolute URLs for internal links  Ensure canonical version is in XML Sitemap  Use rel=canonical attribute for optional parameters  Block print and other versions with robots.txt http://searchengineland.com/canonical-tag-16537 http://janeandrobot.com/library/url-referrer-tracking
    23. 23. Crawl Efficiency Checklist Ensure page load times aren’t slow as to reduce number of pages crawled Ensure server is responsive Return a 304 for unchanged content Use compression Return a 404 for not found content Ensure each page has at least one link Avoid infinite redirects and redirect loops Ensure most important pages are linked from home page No JavaScript redirects or meta refresh redirects (if possible) Reasonable crawl-delay setting (if used at all) Reasonable use of Google Webmaster Tools crawl setting
    24. 24. Indexing Issues
    25. 25. Indexing Example: XML Sitemaps XML Sitemap Pages Indexed From Sitemap <?xml version=quot;1.0quot; encoding=quot;UTF-8quot;?> <urlset xmlns=quot;http://www.sitemaps.org/schemas/sitemap /0.9quot;> <url> <loc>http://www.example.com/</loc> </url> <url> <loc>http://www.example.com/page1.php</loc> </url> <url> <loc>http://www.example.com/page2.php</loc> </url> </urlset> http://sitemaps.org http://www.google.com/webmasters
    26. 26. Duplicate Content Issues
    27. 27. Partner Content http://www.google.co.uk/search?q=%22The+Radisson+Edwardian+Vanderbilt+Hotel+stands+am
    28. 28. Indexing Diagnostic Checklist Have the pages ever been indexed? If deindexed, are you sure they are no longer in the index? Is the indexing loss across all engines? What was percentage of loss? Is there a pattern? Check Google Webmaster Tools for errors/blocking Did you change infrastructure/CMS/implement redirects? What’s the linking pattern?
    29. 29. Indexing Checklist: Content Extraction  Ensure content is in text wherever possible  Ensure text isn’t hidden in: JavaScript/AJAX Flash Video Images  Avoid multiple URLs for the same page and very similar pages
    30. 30. Indexing Checklist: Semantic Markup  Use keywords in title tag  Ensure each page has a unique meta description tag  Use keywords in (single) H1 tag  Appropriate use of H2 – H6 tag  Relevant anchor text in a href tags  Put Javascript in .js file (except onclick event functions) and style details in .css  Validate HTML to ensure it renders  Provide focus for each page  Ensure pages provide unique and valuable content beyond boilerplate template and reused content
    31. 31. Optimizing Images  Don’t put text in images  Use descriptive ALT text  Use descriptive filenames  Provide caption and surrounding text  Be cautious about logo images  Consider blocking non-useful images with robots.txt  Don’t provide alternate text using CSS that styles the text off the page (such as -9999) http://janeandrobot.com/post/Effectively-Using-Images.aspx
    32. 32. Ranking Issues
    33. 33. How’s the Search Engine Traffic? Overall Percentage Percentage Non-Branded
    34. 34. Do You Rank For the Right Things? arbor snowboards snowboard Google 1 49 500+ Yahoo 1 80 500+ Live Search 3 128 500+
    35. 35. If ranking loss… Drop For All Keywords Drop For Only Some Keywords Does the site rank for different Do different pages rank highest than queries than before? used to rank before? Did you substantially change the Are the pages that used to rank still site content? indexed? Did you change the underlying site infrastructure? Was there a large change in linking behavior? Could there be a penalty?
    36. 36. Ranking Checklist  Relevance  What is the page about?  Are the pages ranking for the desired query more relevant?  Do the pages use the language of the searcher?  Value  How many relevant links (and how authoritative are they?)  What’s the value of the page? (do more useful pages rank above yours?)  SERP display  Are the title and snippet compelling?  Do Sitelinks appear for navigational queries?  What universal elements appear on the page?  Does the site rank for non-branded queries?
    37. 37. The Webmaster Guidelines Common Definition of Spam On page schemes Keyword stuffing Fake/ stolen content Hidden text Hidden links Cloaking Linking schemes Paid Links Link exchanges Doorway pages Deceptive redirects http://google.com/support/webmasters/bin/answer.py?answer=35769
    38. 38. Getting Out of the Penalty Box 1. Check if you’ve been penalized – Live Search: http://webmaster.live.com – Google: http://google.com/webmasters ** maybe ** 2. Review the webmaster guidelines – Google, Live Search, Yahoo 3. Identify the issue 4. Fix it! 5. Request re-evaluation – Google: http://google.com/webmasters – Live Search: http://webmaster.live.com
    39. 39. Traffic Issues
    40. 40. Traffic Drop
    41. 41. Display Issues
    42. 42. Would you click this link?
    43. 43. Does the Result Inspire Clicks?
    44. 44. First step in diagnosis: find the root
    45. 45. Jane and Robot Developer Summit June 12th, 2009 – San Francisco FREE for SMX attendees! janeandrobot.com Twitter.com/janeandrobot Ninebyblue.com Twitter.com/vanessafox
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×