URLs and Domains Nathan Buggia, Live Search Webmaster Center Oct 7 th , 2008

What’s a URL (and where search engines get stuck) http://auto.msn.co.uk/autos/default.aspx?id=AA#found Protocol Hostname Path Query Fragment auto.msn.co.uk Subdomain TLD (ccTLD)

http://auto.msn.co.uk/autos/default.aspx?id=AA#found

HTTP Status Codes 200 – Everything’s okay (make sure you don’t return this code on a “Page not Found”!) 404 – File not found 301 – File has been moved 302 – File is temporarily somewhere else

200 – Everything’s okay (make sure you don’t return this code on a “Page not Found”!)

404 – File not found

301 – File has been moved

302 – File is temporarily somewhere else

Robots Exclusion Protocol – Common Mistakes microsoft.com/robots.txt does not apply to technet.microsoft.com

microsoft.com/robots.txt does not apply to technet.microsoft.com

Robots Exclusion Protocol http://janeandrobot.com/post/Managing-Robots-Access-To-Your-Website.aspx

Parameter Tracking http://mysite.com/?from=PROMO_1 Trap the request Create a cookie with from=PROMO_1 Set Cache-Control:no-cache content header Do a 301 redirect to http://mysite.com

http://mysite.com/?from=PROMO_1

Trap the request

Create a cookie with from=PROMO_1

Set Cache-Control:no-cache content header

Do a 301 redirect to http://mysite.com

Duplicate Content When there is more than one URL for the same content http://oreilly.com http://oreilly.com/index.csp http://www.oreilly.com http://www.oreilly.com/index.csp https://oreilly.com https://oreilly.com/index.csp Create a few simple rules that will remove duplicate URLs by 301 redirecting all variations to the shortest, most authoritative URL. Often called “Domain Canonicalization” http://janeandrobot.com/post/canonical-url-canonicalization-domain.aspx

When there is more than one URL for the same content

http://oreilly.com

http://oreilly.com/index.csp

http://www.oreilly.com

http://www.oreilly.com/index.csp

https://oreilly.com

https://oreilly.com/index.csp

Duplicate Content – Gateway Pages When a log-in page, or region select page is placed on each URL unless you already have a cookie Because search engines don’t support cookies, they may see every URL on your site having the same content

When a log-in page, or region select page is placed on each URL unless you already have a cookie

Sitemaps

Great Tools From Search Engines Live Search ( webmaster.live.com ) Crawl Issues (404s, Too many parameters, REP, Bad ContentType) Rank Info (PageRank, DomainRank) Backlinks/Outbound links Google ( google.com/webmaster ) Crawl Issues (404s, REP, Timeout, Unreachable) Comprehensive Link Explorer (outbound, inbound links) Set WWW vs. Non-WWW Yahoo ( SiteExplorer.search.yahoo.com ) Feedback on URL Parameters Backlinks/ Outbound links

Live Search ( webmaster.live.com )

Crawl Issues (404s, Too many parameters, REP, Bad ContentType)

Rank Info (PageRank, DomainRank)

Backlinks/Outbound links

Google ( google.com/webmaster )

Crawl Issues (404s, REP, Timeout, Unreachable)

Comprehensive Link Explorer (outbound, inbound links)

Set WWW vs. Non-WWW

Yahoo ( SiteExplorer.search.yahoo.com )

Feedback on URL Parameters

Backlinks/ Outbound links

Issues Encountered by URL

webmaster.live.com