Tools and Tactics for Diagnosing
Technical Search Issues
Vanessa Fox

Diagnostic Checklists and
Resources
• Search Accessibility Checklist
• Search Discoverability Checklist
• Diagnostic Tools
janeandrobot.com

Search Engine Tools
Google Webmaster Central
http://www.google.com/webmasters
Microsoft Live Search Webmaster Center
http://webmaster.live.com
Yahoo! Site Explorer
http://siteexplorer.search.yahoo.com
Google Analytics
http://www.google.com/analytics
Google Search
http://www.google.com
Created by NineByBlue.com

Ranking and Diagnostic Tools
SEOBook Rank Checker
http://tools.seobook.com/firefox/rank-checker/
Firefox Web Developer Toolbar
https://addons.mozilla.org/en-US/firefox/addon/60
Firefox Firebug
http://getfirebug.com/
Firefox Live HTTP Headers
https://addons.mozilla.org/en-US/firefox/addon/3829
Google Search
http://adlab.msn.com/Keyword-Forecast/default.aspx
http://janeandrobot.com/resources Created by NineByBlue.com

How Search
Engines Work

Crawling Indexing Ranking
Discover links Canonicalization Relevance
Check robots rules Context extraction Value
Bandwidth considerations Topic association Uniqueness
URLs Web-wide value Display

Search Engine Crawlers Haven’t Quite Grown Up Yet

Crawling Indexing Ranking
Lack of discovery Duplication Display issues
Crawl inefficiency Extraction issues Lack of quality links
URL issues (infinite, Lack of exposed Guidelines violations
redirects, dynamic) content Non-focused content
Inaccessible links Non-optimized
media

Step 1: Get the Data
Pages crawled
Pages indexed
Web traffic
Key ranking metrics

Crawling Which pages have the search engines crawled?
What kind of pages are they?
Has the search engine
Indexing indexed all of the
crawled pages?
How’s the search engine
Ranking traffic?

Benchmarking
Top ten queries that bring
search traffic
Search results position
URL that ranks

Crawl Issues

Crawl Log Example: Apache Log Analyzer 2 Feed
All Pages Google’s Crawled
1 /**
2 * @see ApacheLogAnalyzer2Feed All Profile Pages Google’s Crawled
3 */
1 /**
4 require_once 'ApacheLogAnalyzer2Feed.php';
5 2 * @see ApacheLogAnalyzer2Feed
6 //3 create a new instance, parse access.log and
*/
7 //4 write test.xml'ApacheLogAnalyzer2Feed.php';
require_once
5
8 $tool = new ApacheLogAnalyzer2Feed('access.log',
9 'test.xml'); a new instance, parse access.log and write test.xml
6 // create
10 // $tool = entries matching Googlebot useragent
7 select new ApacheLogAnalyzer2Feed('access.log', 'test.xml');
11 $tool->addFilter('User-Agent',Googlebot useragent with a regular
8 // select entries matching 'Mozilla/5.0
9 expression pattern
12 (compatible; Googlebot/2.1;
13 +http://www.google.com/bot.html)');'regexp:Googlebot');
10 $tool->addFilter('User-Agent',
14 // run select entries with Request matching a regular expression
11 //
12 // pattern
15 $tool->run(); 12
13 $tool->addFilter('Request', 'regexp:/site/profile\\.php');
14 // run
15 $tool->run();
16
http://code.simonecarletti.com/wiki/apachelog2feed

Communicating with Search Robots

Extractable Link Issues: Flash

Extractable Link Issues: Images

Extractable Link Issues: AJAX

Extractable Link Issues: URL Errors

Extractable Link Issues: URLs That Expire

URL Discovery Checklist
 Comprehensive external links
 At least one internal link to every
page
 XML Sitemap referenced in
robots.txt with the
comprehensive list of canonical
URLs
 Comprehensive HTML sitemap
 Ensure links load without
JavaScript, images, or other rich
media
 Ensure robots.txt and meta
robots tag is used correctly
http://janeandrobot.com/library/managing-robots-access-to-your-website

URL Structure Checklist
Keep number of parameters in
dynamic URLs short
Don’t use temporary URLs that
expire
 Ensure redirects are 301 and are
short
Use dashes rather than
underscores when separating
words
Use keywords in URLs for higher
click through and better anchor text

Canonicalization Checklist
 Have only URL for each page
 Put all unneeded details in cookies, rather than URLs
(session IDs, tracking parameters)
 Don’t allow infinite parameters
 Use 301 redirects for any URL changes
 301 redirect www/non-www
 Use absolute URLs for internal links
 Ensure canonical version is in XML Sitemap
 Use rel=canonical attribute for optional parameters
 Block print and other versions with robots.txt
http://searchengineland.com/canonical-tag-16537
http://janeandrobot.com/library/url-referrer-tracking

Crawl Efficiency Checklist
Ensure page load times aren’t slow as to reduce number of
pages crawled
Ensure server is responsive
Return a 304 for unchanged content
Use compression
Return a 404 for not found content
Ensure each page has at least one link
Avoid infinite redirects and redirect loops
Ensure most important pages are linked from home page
No JavaScript redirects or meta refresh redirects (if possible)
Reasonable crawl-delay setting (if used at all)
Reasonable use of Google Webmaster Tools crawl setting

Indexing Issues

Indexing Example: XML Sitemaps
XML Sitemap
Pages Indexed From Sitemap
<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<urlset
xmlns=\"http://www.sitemaps.org/schemas/sitemap
/0.9\">
<url>
<loc>http://www.example.com/</loc>
</url>
<url>
<loc>http://www.example.com/page1.php</loc>
</url>
<url>
<loc>http://www.example.com/page2.php</loc>
</url>
</urlset>
http://sitemaps.org http://www.google.com/webmasters

Duplicate Content Issues

Partner Content
http://www.google.co.uk/search?q=%22The+Radisson+Edwardian+Vanderbilt+Hotel+stands+am

Indexing Diagnostic Checklist
Have the pages ever been indexed?
If deindexed, are you sure they are no longer in the index?
Is the indexing loss across all engines?
What was percentage of loss?
Is there a pattern?
Check Google Webmaster Tools for errors/blocking
Did you change infrastructure/CMS/implement redirects?
What’s the linking pattern?

Indexing Checklist: Content Extraction
 Ensure content is in text wherever possible
 Ensure text isn’t hidden in:
JavaScript/AJAX
Flash
Video
Images
 Avoid multiple URLs for the same page and very similar pages

Indexing Checklist: Semantic Markup
 Use keywords in title tag
 Ensure each page has a unique meta description tag
 Use keywords in (single) H1 tag
 Appropriate use of H2 – H6 tag
 Relevant anchor text in a href tags
 Put Javascript in .js file (except onclick event functions)
and style details in .css
 Validate HTML to ensure it renders
 Provide focus for each page
 Ensure pages provide unique and valuable content
beyond boilerplate template and reused content

Optimizing Images
 Don’t put text in images
 Use descriptive ALT text
 Use descriptive filenames
 Provide caption and surrounding text
 Be cautious about logo images
 Consider blocking non-useful images with robots.txt
 Don’t provide alternate text using CSS that styles the text off
the page (such as -9999)
http://janeandrobot.com/post/Effectively-Using-Images.aspx

Ranking
Issues

How’s the Search Engine Traffic?
Overall Percentage Percentage Non-Branded

Do You Rank For the Right Things?
arbor snowboards snowboard
Google 1 49 500+
Yahoo 1 80 500+
Live Search 3 128 500+

If ranking loss…
Drop For All Keywords Drop For Only Some Keywords
Does the site rank for different Do different pages rank highest than
queries than before? used to rank before?
Did you substantially change the Are the pages that used to rank still
site content? indexed?
Did you change the underlying
site infrastructure?
Was there a large change in
linking behavior?
Could there be a penalty?

Ranking Checklist
 Relevance
 What is the page about?
 Are the pages ranking for the desired query more relevant?
 Do the pages use the language of the searcher?
 Value
 How many relevant links (and how authoritative are they?)
 What’s the value of the page? (do more useful pages rank above
yours?)
 SERP display
 Are the title and snippet compelling?
 Do Sitelinks appear for navigational queries?
 What universal elements appear on the page?
 Does the site rank for non-branded queries?

The Webmaster Guidelines
Common Definition of Spam
On page schemes Keyword stuffing
Fake/ stolen content
Hidden text
Hidden links
Cloaking
Linking schemes Paid Links
Link exchanges
Doorway pages
Deceptive redirects
http://google.com/support/webmasters/bin/answer.py?answer=35769

Getting Out of the Penalty Box
1. Check if you’ve been penalized
– Live Search: http://webmaster.live.com
– Google: http://google.com/webmasters ** maybe **
2. Review the webmaster guidelines
– Google, Live Search, Yahoo
3. Identify the issue
4. Fix it!
5. Request re-evaluation
– Google: http://google.com/webmasters
– Live Search: http://webmaster.live.com

Traffic Issues

Traffic Drop

Display Issues

Would you click this link?

Does the Result Inspire Clicks?

First step in diagnosis: find the root

Jane and Robot Developer Summit
June 12th, 2009 – San Francisco
FREE for SMX attendees!
janeandrobot.com
Twitter.com/janeandrobot
Ninebyblue.com
Twitter.com/vanessafox